Scaling Ingress Resources (NSX-T only)

Page last updated:

Warning: VMware Enterprise PKS v1.6 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to monitor the health status of the NSX-T load balancer and scale ingress resources.

Note: This feature requires NCP v2.5.1 or later.

About Load Balancer Monitoring and Ingress Scaling

For each Kubernetes cluster provisioned by Enterprise PKS with NSX-T, there are 2 layer 7 virtual servers (HTTP and HTTPS) attached to the cluster load balancer service that perform ingress routing.

Using custom resources, Enterprise PKS v1.6 gives you the ability to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

Monitoring the NSX-T Load Balancer Service

You can use the NSXLoadBalancerMonitor CRD to monitor the NSX-T load balancer service, including traffic, usage and health score information.

For each load balancer service created by the NSX-T load balancer, NCP (by way of the CRD) creates a corresponding NSXLoadBalancerMonitor object:

  • 1 layer 4 load balancer created by default for the Kubernetes master nodes
  • 1 layer 4 auto-scaled load balancer created for each service resource
  • 2 layer 7 ingress load balancers which can be interactively scaled

For each type of load balancer, the NSXLoadBalancerMonitor returns statistics showing the number of connections and throughput of the virtual servers. In addition, a health score is calculated that reflects the current performance of that load balancer.

If the health score is poor for one of the layer 4 load balancers, you can use a network profile to increase the size of the NSX-T load balancer service.

If the health score is poor for the layer 7 ingress load balancers, you can use the LoadBalancer CRD to manually scale ingress.

Refer to the NSXLoadBalancerMonitor Actions table for details you can take based on the results of the NSXLoadBalancerMonitor.

Using the NSXLoadBalancerMonitor CRD

To run the NSXLoadBalancerMonitor CRD, issue the following set of kubectl commands.

Issue the following command to view the NSXLoadBalancerMonitor CRD:

kubectl get crd

Issue the following command to get the UUID of the NSX-T load balancer deployed for the cluster:

kubectl get nsxlbmonitors

Issue the following command to view statistics, throughput, and health score for all virtual servers deployed by a specific load balancer service:

kubectl describe nsxlbmonitors UUID-OF-LOAD-BALANCER

Example Results Returned by the NSXLoadBalancerMonitor CRD

The following examples shows the results returned by the NSXLoadBalancerMonitor CRD for an example NSX-T load balancer service.

$ kubectl describe nsxlbmonitor f61a8cec-28eb-4b0c-bf4a-906f3ce2d8e6
Name:         f61a8cec-28eb-4b0c-bf4a-906f3ce2d8e6
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  vmware.com/v1alpha1
Health:
  Metrics:
    Cpu Usage Percentage:         0
    Poolmember Usage Percentage:  1
  Service Pressure Index:         0,LOW
  Infra Pressure Index:           0,LOW
  Metrics:
    Cpu Usage Percentage:         0
    Lb Service Usage Percentage:  0
    Memory Usage Percentage:      0
    Poolmember Usage Percentage:  0
Kind:                             NSXLoadBalancerMonitor
Metadata:
  Creation Timestamp:  2019-11-13T19:37:10Z
  Generation:          914
  Resource Version:    17139
  Self Link:           /apis/vmware.com/v1alpha1/nsxlbmonitors/f61a8cec-28eb-4b0c-bf4a-906f3ce2d8e6
  UID:                 f56d3cf5-748d-44c3-8026-c6c569fde954
Traffic:
  Bytes In Rate:         0
  Bytes Out Rate:        0
  Current Session Rate:  0
  Ip Address:            192.168.160.102
  Max Sessions:          0
  Packets In Rate:       0
  Packets Out Rate:      0
  Protocol:              TCP
  Total Sessions:        0
  Virtual Server Name:   pks-042bccde-2197-4e06-863e-55129bf2e195-http
  Bytes In Rate:         0
  Bytes Out Rate:        0
  Current Session Rate:  0
  Ip Address:            192.168.160.102
  Max Sessions:          0
  Packets In Rate:       0
  Packets Out Rate:      0
  Protocol:              TCP
  Total Sessions:        0
  Virtual Server Name:   pks-042bccde-2197-4e06-863e-55129bf2e195-https_terminated
Usage:
  Current Server Pool Count:     1
  Current Virtual Server Count:  3
Events:                          <none>

NSXLoadBalancerMonitor Actions

The NSXLoadBalancerMonitor CRD returns two health scores:

  • servicePressureIndex which represents an overall health score for the NSX-T load balancer service
  • infraPressureIndex which represents the heath score of the NSX-T Edge Node that is running the load balancer and associated virtual servers

Based on the health score the user can decide what action to take. The table below summarizes the actions that you can take based on the health scores.

servicePressureIndex infraPressureIndex Cluster Manager Infrastructure Admin
LOW or WARM LOW or WARM NONE NONE
LOW or WARM HIGH Alert infra admin Move the LBS from the CRITICAL Edge Node to another Edge Node.
HIGH LOW or WARM Resolve the LBS health score by scaling the ingress LB and, if necessary, by increasing the size of the LBS using network profile. NONE
HIGH HIGH Alert infra admin; Resolve the LBS health score by scaling the ingress LB and, if necessary, by increasing the size of the LBS using network profile. Move the LBS from the CRITICAL Edge Node to another Edge Node.

Scaling Ingress Using the LoadBalancer CRD

The LoadBalancer CRD provides you with an interactive method to scale the load balancer for ingress routing. After you have used the NSXLoadBalancerMonitor CRD to check the health status of the load balancer service, you can use the LoadBalancer CRD to create a new ingress load balancer. Then you annotate the Kubernetes K8S ingress resource with the newly created ingress load balancer. NCP will attach the ingress rules to the scaled out load balancer.

LoadBalancer CRD Example

The following example creates a new ingress load balancer.

# kubectl apply –f lb.yaml
apiVersion: vmware.com/v1alpha1
kind: LoadBalancer
metadata:
  name: cluster1-lbs0                     # display name of the loadBalancer
spec:
  httpConfig:                             # optional config to support http/https route on the loadBalancer. Set as httpConfig: {} to apply default settings.
    virtualIP:                            # optional, defaults to auto_allocate     
    port: 233                             # optional, defaults to 80
    tls:
      port: 2333                          # optional, defaults to 443
      secretName: default_secret          # optional, defaults to nil
      secretNamespace: default            # optional, defaults to nil
    xForwardedFor: INSERT                 # optional, available values are INSERT, REPLACE, defaults to nil
    affinity:
      type: source_ip                     # optional, available values are sourceIP, cookie 
      timeout: 100                        # optional, defaults to 10800
  size: MEDIUM                            # optional, defaults to SMALL
  virtualNetwork: virtualnetwork1         # optional, defaults to nil
status:
  httpVirtualIP: <realized external ip>   # auto-allocated or user desired external ip for http/https virtual server

Note: You must deploy the new ingress load balancer in the same namespace where you deploy the ingress resource.

The following example annotates the Kubernetes ingress resource with the newly created ingress load balancer.

# kubectl apply –f ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: svc-ingress1
  annotations:
    nsx/loadbalancer: cluster1-lbs0       # loadbalancer-name-that-you-gave-in-crd
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /testpath
          backend:
            serviceName: svc1
            servicePort: 80

Please send any feedback you have to pks-feedback@pivotal.io.