Installing PFS on Minikube

This topic describes how to install Pivotal Function Service (PFS) on a development machine using minikube for macOS or Windows or Linux.

Requirements

  • The Minikube CLI has been installed at version v1.3.1 or later.
  • A Minikube VM driver for your OS has been installed:
  • Docker Desktop for Mac or Windows has been installed at version 2.1.0.1 or later.
  • The kubectl CLI has been installed at version 1.14 or later. The kubectl CLI is included with Docker Desktop for Mac or Windows.
  • The pfs CLI has been downloaded and installed.
  • The PFS thick bundle has been downloaded.

Install a local Registry

Since PFS installations require a container registry, these instructions include running a local registry accessible from Kubernetes as well as from the host development machine at registry.pfs.svc.cluster.local:5000.

  1. Use the docker CLI to run the registry:2 container from Docker, listening on port 5000, and persisting images in the ~/.registry/storage directory.

    docker run -d -p 5000:5000 --restart=always --volume ~/.registry/storage:/var/lib/registry registry:2
    

    In Windows PowerShell (after enabling Shared Drives for drive C: in your Docker settings)

    docker run -d -p 5000:5000 --restart=always --volume c:/Users/$env:UserName/.registry/storage:/var/lib/registry registry:2
    
  2. Edit the /etc/hosts file on your development machine, adding the name registry.pfs.svc.cluster.local on the same line as the entry for localhost. On macOS the file should look something like this.

    ##
    127.0.0.1       localhost registry.pfs.svc.cluster.local
    255.255.255.255 broadcasthost
    ::1             localhost
    

    In Windows PowerShell running as Administrator

    Add-Content `
      -Path c:\Windows\System32\Drivers\etc\hosts `
      -Value '127.0.0.1 registry.pfs.svc.cluster.local'
    
  3. Validate that the registry is running.

    docker ps
    
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
    02ea46d51f58        registry:2          "/entrypoint.sh /etc…"   About an hour ago   Up About a minute   0.0.0.0:5000->5000/tcp   sharp_pike
    
  4. Validate that the registry at registry.pfs.svc.cluster.local:5000 is reachable from your development machine.

    curl registry.pfs.svc.cluster.local:5000/v2/_catalog
    
    {"repositories":[]}
    
  5. Configure the docker daemon with an insecure registy at registry.pfs.svc.cluster.local:5000. E.g. in ~/.docker/daemon.json

    {
    "debug": true,
    "experimental": false,
    "insecure-registries": [
        "registry.pfs.svc.cluster.local:5000"
    ]
    }
    
  6. Follow the instructions to Push relocated images to the local registry, making sure that the images are relocated to registry.pfs.svc.cluster.local:5000.

Configure Minikube Default Settings

Replace ??? below with hyperkit for macOS, hyperv for Windows, and kvm2 for Linux.

minikube config set cpus 4
minikube config set memory 4096
minikube config set vm-driver ???

On Windows use a terminal with Administrator privileges for all `minikube` commands.

Start Minikube

This command will use your minikube config settings, and enable insecure access to the local registry.

minikube start --insecure-registry registry.pfs.svc.cluster.local:5000

Use kubectl config current-context to verify that the context is set to minikube. If necessary set the current context using kubectl config use-context minikube.

In a separate terminal window, watch the pods in the cluster.

watch kubectl get pod --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c98db65d4-mj6tb           1/1     Running   0          78s
kube-system   coredns-5c98db65d4-wl9r4           1/1     Running   0          78s
kube-system   etcd-minikube                      1/1     Running   0          28s
kube-system   kube-addon-manager-minikube        1/1     Running   0          26s
kube-system   kube-apiserver-minikube            1/1     Running   0          29s
kube-system   kube-controller-manager-minikube   1/1     Running   0          5s
kube-system   kube-proxy-g7sb5                   1/1     Running   0          78s
kube-system   kube-scheduler-minikube            1/1     Running   0          21s
kube-system   storage-provisioner                1/1     Running   0          77s

Configure a fixed IP address

This IP address will allow processes in Minikube to reach the registry running on your host. Configuring a fixed IP address avoids the problem of the IP address changing whenever you connect your machine to a different network. If your machine already uses the 172.16.x.x range for other purposes, choose an address in a different range e.g. 172.31.x.x..

export DEV_IP=172.16.1.1

Create an alias on MacOS:

sudo ifconfig lo0 alias $DEV_IP

Create an alias on Linux:

sudo ifconfig lo:0 $DEV_IP

Note that the alias will need to be reestablished when you restart your machine. This can be avoided by using say a launchdeamon on MacOS or by editing /etc/network/interfaces on Linux.

To create a loopback with a fixed IP address on Windows, see here.

Minikube /etc/hosts

Add an entry to /etc/hosts inside the minikube VM, pointing the registry to the IP address of the host. This will result in registry.pfs.svc.cluster.local resolving to the host machine allowing the docker daemon in minikube to pull images from the local registry. This uses the DEV_IP environment variable from the previous step.

export DEV_IP=172.16.1.1
minikube ssh "echo \"$DEV_IP       registry.pfs.svc.cluster.local\" | sudo tee -a  /etc/hosts"

In Windows PowerShell

$ENV:DEV_IP = '172.16.1.1'
minikube ssh "echo ""$Env:DEV_IP       registry.pfs.svc.cluster.local"" | sudo tee -a  /etc/hosts"

Kubernetes Service and Endpoint

Create a kubernetes service without selectors called registry in the pfs namespace and a kubernetes endpoint with the same name pointing to the static IP address of your development machine. This will result in registry.pfs.svc.cluster.local resolving to the host machine, allowing container builds running in the cluster, to work with the local registry.

kubectl create namespace pfs

cat <<EOF | kubectl apply -n pfs -f -
---
kind: Service
apiVersion: v1
metadata:
  name: registry
spec:
  ports:
  - protocol: TCP
    port: 5000
    targetPort: 5000
---
kind: Endpoints
apiVersion: v1
metadata:
  name: registry
subsets:
  - addresses:
      - ip: $DEV_IP
    ports:
      - port: 5000
EOF

In Windows PowerShell, create a registry.yaml file with the Service and Endpoints yaml from above, replacing $DEV_IP with the IP address. Then apply that using kubectl apply -n pfs -f registry.yaml.

Configure duffle

Set the environment variables required by the duffle Kubernetes driver, create a namespace for duffle, create a service account for duffle and give it cluster-admin permissions.

export SERVICE_ACCOUNT=duffle-runtime
export KUBE_NAMESPACE=duffle
kubectl create namespace $KUBE_NAMESPACE
kubectl create serviceaccount "${SERVICE_ACCOUNT}" -n "${KUBE_NAMESPACE}"
kubectl create clusterrolebinding "${SERVICE_ACCOUNT}-cluster-admin" --clusterrole cluster-admin --serviceaccount "${KUBE_NAMESPACE}:${SERVICE_ACCOUNT}"

See Using PFS with Windows for setting environment variables and using multi-line commands in Windows.

Install PFS

Change to the directory with the downloaded PFS thick bundle and run duffle install with the relocation mapping file created during image relocation.

duffle install my-pfs pfs-bundle-thick.tgz --bundle-is-file \
--relocation-mapping pfs-relmap.json \
--driver k8s \
--set node_port=true

The --node-port option is required for access to Kubernetes services via NodePort rather than LoadBalancer.

Watch for all the pods to start running.

NAMESPACE         NAME                                      READY   STATUS    RESTARTS   AGE
docker            compose-6c67d745f6-rdgnn                  1/1     Running   1          29m
docker            compose-api-57ff65b8c7-ptx45              1/1     Running   1          29m
istio-system      cluster-local-gateway-79488cf4d4-cxtcn    1/1     Running   0          80s
istio-system      istio-ingressgateway-7dd7c78b6b-ddznz     2/2     Running   0          80s
istio-system      istio-pilot-864f8bb559-qzz72              1/1     Running   0          80s
knative-build     build-controller-7cb488654c-97ps8         1/1     Running   0          80s
knative-build     build-webhook-6d44665d7c-qdmcf            1/1     Running   0          80s
knative-serving   activator-574f6c964d-nfcpc                1/1     Running   0          77s
knative-serving   autoscaler-5f4fc6ffb9-kh4tx               1/1     Running   0          77s
knative-serving   controller-6c96bf4dfd-jcjc4               1/1     Running   0          77s
knative-serving   networking-certmanager-65db98c9f6-p8hdx   1/1     Running   0          77s
knative-serving   networking-istio-7c596dff5f-99zwx         1/1     Running   0          77s
knative-serving   webhook-78f56b7695-jnwtn                  1/1     Running   0          77s
kube-system       coredns-584795fc57-pp44c                  1/1     Running   1          30m
kube-system       coredns-584795fc57-zdmb5                  1/1     Running   1          30m
kube-system       etcd-docker-desktop                       1/1     Running   1          29m
kube-system       kube-apiserver-docker-desktop             1/1     Running   1          29m
kube-system       kube-controller-manager-docker-desktop    1/1     Running   1          29m
kube-system       kube-proxy-4b68z                          1/1     Running   1          30m
kube-system       kube-scheduler-docker-desktop             1/1     Running   1          29m
riff-system       controller-79d7dfb4df-sv4qh               1/1     Running   0          57s
riff-system       webhook-59bb6b499-82bjk                   1/1     Running   0          56s

PFS is now installed.

Configure PFS to use your registry.

The pfs credential apply command will prompt for a password for the registry-user assuming basic authentication. In this case you can simply hit return, since the local registry is unauthenticated.

export REGISTRY=registry.pfs.svc.cluster.local:5000
export REGISTRY_USER=testuser

pfs credential apply my-pfs \
  --registry $REGISTRY \
  --registry-user $REGISTRY_USER \
  --default-image-prefix $REGISTRY/$REGISTRY_USER

You can now create your first function.