Routing to Workloads

Page last updated:

This topic explains how Pivotal Ingress Router routes traffic to the worker nodes of a Kubernetes cluster.

For information about how to enable this feature, see the Configure User Provided Values section of Installing Ingress Router.

For information about how to use this feature, see Creating Workload Routes.

Overview

Pivotal Ingress Router is capable of routing TCP traffic to workloads running on your clusters through the “Routing to Workloads” feature. Similar to routing to Kubernetes clusters, this feature uses Istio to inspect network traffic. However, for “Routing to Workloads”, Istio forwards traffic to the correct worker VM(s) where the workload is running. NodePort is used to expose the pod(s) running your workload so that traffic from IngressGateway can reach the worker VM(s).

Prerequisites

• Pivotal Ingress Router is installed (see Installing Pivotal Ingress Routing).

Enable Routing to Workloads

1. Authenticate kubectl with the system cluster. See Installation for more info.
2. To enable the feature, set ingressRouter.experimental.workloadRouting.enabled in user-provided-values.yaml to true.

3. Run helm template and kubectl apply to deploy Ingress Router with the new feature enabled.

   $ helm template helm/pivotal-service-mesh \
     --namespace ingress-router-system \
     -f helm/pivotal-service-mesh/user-provided-values.yaml \
     > /tmp/ingressrouter-installation.yaml
   $ kubectl apply -f /tmp/ingressrouter-installation.yaml
   

4. Run kubectl --namespace ingress-router-system get pods to confirm all pods are in a running state.

   $ kubectl --namespace ingress-router-system get pods
   NAME                                                            READY   STATUS      RESTARTS   AGE
   cluster-registrar-b85b464b4-k4rdx                               1/1     Running     0          176m
   ...
   master-routing-controller-6777cb67c8-mhnc9                      1/1     Running     0          176m 
   workload-routing-controller-manager-88bfd7bbc-dvcp4             1/1     Running     0          176m
   wrc-system-cluster-0051-58df467ddb-bmfhp                        1/1     Running     0          176m
   wrc-test-cluster-e59d-58d6857bc6-t7mg5                          1/1     Running     1          176m
   

   The pods that starting with wrc are called workload-routing-controllers. When the routing to workloads feature is enabled, Pivotal Ingress Router deploys an additional workload-routing-controller pod per cluster.

Configuring Port Range

To use workload routing, a port range, or set of port ranges must be configured. The WorkloadRoutingControllers choose ports from the port range, or port ranges, when a VirtualService is created. These ports will opened on the IngressGateway pod, allowing connections originating from outside the network to reach the workload.

For example, given the port range is 31000-31010, when a VirtualService is created for a workload, clients from outside the network can make a request to the Load Balancer IP or System Cluster Kubernetes Worker IP(s) on a port in the range to connect to the workload. If this is the first VirtualService, that workload will be reachable at <Load Balancer IP or System Cluster Kubernetes Worker IP>:31000 (31000 since its the first number in the range).

To create a port range, create the following custom resource and apply it with kubectl -n ingress-router-system apply -f <File name>:

---
apiVersion: ingressrouter.pivotal.io/v1
kind: PortAllocation
metadata:
  name: workload-allocation
  namespace: ingress-router-system
spec:
  portRanges:
  - start: 31000
    end: 31010

This create a range of 31000 to 31010 for workloads.

Creating service account for cluster

For workload routing, the Ingress Router must be able to watch external clusters for new VirtualService resources. When the Ingress Router sees a new VirtualService resource, it will then enable routing to the workloads on that cluster. To configure routing to the workload the Ingress Router needs permissions to get the node IPs. The Ingress Router needs a service account for that cluster to watch the VirtualService resources.

Additionally, the Ingress Router will install the VirtualService CRD onto the workload cluster with the same service account.

To create the service account, kubectl apply this yaml to your workload cluster.

1. Save this to a file called: ingress-router-bootstrap-service-account.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-router-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pivotal-ingress-router
  namespace: ingress-router-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pivotal-ingress-router
  namespace: ingress-router-system
rules:
- apiGroups: ["networking.istio.io"]
  resources: ["virtualservices"]
  verbs: ["get", "watch", "list", "update"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "create", "patch"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: pivotal-ingress-router
  namespace: ingress-router-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: pivotal-ingress-router
subjects:
- kind: ServiceAccount
  name: pivotal-ingress-router
  namespace: ingress-router-system

1. Apply the ingress-router-bootstrap-service-account.yaml file. kubectl --context <workload-cluster-name> apply -f ingress-router-bootstrap-service-account.yaml

Now that you have a service account, you need to create a kubeconfig and provide this kubeconfig to the Ingress Router by creating a secret in the cluster that the Ingress Router is deployed.

1. Create a kubeconfig using the service account token, ca-cert, and the Kubernetes cluster api url.

workload_cluster=<workload_cluster>
secret_name="$(kubectl --context $workload_cluster -n ingress-router-system get serviceaccounts pivotal-ingress-router -o jsonpath='{.secrets[0].name}')"
token="$(kubectl --context $workload_cluster -n ingress-router-system get secrets $secret_name -o jsonpath='{.data.token}' | base64 -D)"
ca_cert="$(kubectl --context $workload_cluster -n ingress-router-system get secrets $secret_name -o jsonpath='{.data.ca\.crt}')"
api_server="$(pks cluster $workload_cluster --json | jq -r .parameters.kubernetes_master_host)"

cat << EOF > kubeconfig-workload-cluster.yaml

apiVersion: v1
kind: Config
clusters:
- name: cluster
  cluster:
    server: https://$api_server:8443
    certificate-authority-data: $ca_cert
contexts:
- name: cluster
  context:
    cluster: cluster
    user: pivotal-ingress-router
current-context: cluster
users:
- name: pivotal-ingress-router
  user:
    token: $token
EOF

1. Create a secret for the system cluster. It must be named kubeconfig-<workload-cluster-name>-<last 4 digits of cluster UUID>.

For example, if you see this as your output from running pks clusters: PKS Version Name k8s Version Plan Name UUID Status Action 1.5.1-build.8 my-system-cluster 1.14.6 privileged 9a4fbb09-3efe-4780-8479-f58e8e09c1b0 succeeded UPGRADE 1.5.1-build.8 workload-cluster 1.14.6 small d357ddb5-c24c-4556-bb29-c0805c03d2c2 succeeded UPGRADE

Then the name of your secret will be: kubeconfig-workload-cluster-d2c2.

Run this kubectl command to create the secret. kubectl --context <system-cluster-name> -n ingress-router-system create secret generic <secret-name> --from-file kubeconfig=kubeconfig-workload-cluster.yaml