Configure the TCP Layer 4 Load Balancer

Note: As of v1.8, Enterprise PKS has been renamed to VMware Tanzu Kubernetes Grid Integrated Edition. Some screenshots in this documentation do not yet reflect the change.

Page last updated:

This topic describes how to define network profile to configure the NSX-T Load Balancer for VMware Tanzu Kubernetes Grid Integrated Edition.

Overview

The NSX-T Load Balancer is a logical load balancer that handles a number of functions using virtual servers and pools.

The NSX-T load balancer creates a load balancer service for each Kubernetes cluster provisioned by Tanzu Kubernetes Grid Integrated Edition with NSX-T. For each load balancer service, NCP, by way of the CRD, creates corresponding NSXLoadBalancerMonitor objects.

By default Tanzu Kubernetes Grid Integrated Edition deploys the following NSX-T virtual servers for each Kubernetes cluster:

  • One TCP layer 4 load balancer virtual server for the Kubernetes API server.
  • One TCP layer 4 auto-scaled load balancer virtual server for each Kubernetes service resource of type: LoadBalancer.
  • Two HTTP/HTTPS layer 7 ingress routing virtual servers. These virtual server are attached to the Kubernetes Ingress Controller cluster load balancer service and can be manually scaled. Tanzu Kubernetes Grid Integrated Edition uses Kubernetes custom resources to monitor the state of the NSX-T load balancer service and scale the virtual servers created for ingress.

For information about configuring TCP layer 4 ingress controller see Configure the TCP Ingress Controller Network Profile, below.

For information about configuring layer 7 ingress routing load balancers see Scaling the HTTP/S Layer 7 Ingress Load Balancers Using the LoadBalancer CRD. For information about configuring the layer 7 ingress controller see Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller.

For more information about the NSX-T Load Balancer, see NSX-T Load Balancer in the VMware documentation.

For more information about Kubernetes custom resources, see Custom resources in the Kubernetes documentation.

Configure the TCP Ingress Controller Network Profile

The TCP layer 4 virtual server provisioned for each Kubernetes service is controlled by the parameters exposed in a network profile.

Note: The TCP layer 4 virtual server that fronts the Kubernetes API server is always created, and it is not controlled by the parameters exposed in the network profile.

NSX-T TCP Ingress Controller Network Profile Configuration

The NSX Ingress Controller is configured using the ncp.ini network profile configuration file.

The TCP Ingress Controller network profile has the following format:

{
  "name": "network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": NSX-LB,        
           "x_forwarded_for": "FORWARD-TYPE",
           "max_l4_lb_service": MAX-SERVERS,
           "l4_persistence_type": "source_ip",
           "l4_algorithm": "LB-BEHAVE"           
           }
       }
    }
}

Where:

For example:

{
  "name": "network_profile",
  "description": "DESCRIP",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "nsx_lb": true,        
           "x_forwarded_for": "replace",
           "max_l4_lb_service": 10,
           "l4_persistence_type": "source_ip",
           "l4_algorithm": "weighted_round_robin"           
           }
       }
    }
}

The following table describes the Ingress Controller configuration parameters:

Parameter Type Description
name String User-defined name of the network profile.
description String User-defined description for the network profile.
parameters Map One or more name-value pairs.
cni_configurations Map Map containing two key-value pairs: type and parameters.
type Constant String Only nsxt is accepted.
parameters Map Map containing one or more key-value pairs for NCP settings.
nsx_lb Boolean Flag to control if the NSX-T Load Balancer is used for the Kubernetes cluster.
x_forwarded_for String Sets the original client source IP in the request header. Default is none. Accepts "insert" and "replace".
max_l4_lb_service Integer Limit the maximum number of layer 4 virtual servers per cluster. Minimum is 1.
l4_persistence_type String Connection stickiness based on source_ip (only accepted value).
l4_lb_algorithm String Specify the layer 4 load balancer behavior. Accepts "round_robin" (default), "least_connection", "ip_hash", "weighted_round_robin".

The nsx_lb parameter is used to control the TCP layer 4 virtual server that is provisioned for each Kubernetes service of type: LoadBalancer.

The nsx_ingress_controller parameter is used to control if NCP is used as the Ingress Controller for the Kubernetes cluster. By default when you define an ingress resource for a Kubernetes cluster, NCP instructs the NSX-T load balancer to provision 2 layer 7 virtual services (HTTP and HTTPS) as the Ingress Controller.

The nsx_ingress_controller parameter is subject to the nsx_lb parameter as described in the following table:

nsx_lb setting nsx_ingress_controller: true nsx_ingress_controller: false
nsx_lb: true Use the NSX-T Layer 4 LoadBalancer and the NCP-provisioned Layer 7 Ingress Controller. Use the NSX-T Layer 4 LoadBalancer and a third-party Ingress Controller, such as NGINX.
nsx_lb: false Invalid configuration. You cannot disable the NSX-T Load Balancer and use NCP as the Ingress Controller. The network profile will fail validation. Use a third-party load balancer and a third-party ingress controller.

Configure Which NSX Load Balancer to Use

The nsx_lb flag controls whether to deploy either the NSX-T Load Balancer or a third-party load balancer, such as Nginx.
The nsx_lb parameter accepts true or false. The default setting is true and the NSX-T Load Balancer is deployed. To use a third party load balancer, set this parameter to false.

For example:

{
  "name": "example_network_profile",
  "description": "nsx_lb and nsx_ingress_controller are enabled",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
         "nsx_lb": false,
         "nsx_ingress_controller": false
      }
    }
  }
}

Note: The nsx_lb parameter maps to the use_native_loadbalancer parameter in NCP.ini.

Since the nsx_ingress_controller is a component of the NSX-T Load Balancer, if you choose to use a third-party load balancer (nsx_lb: false), you must also explicitly disable the nsx_ingress_controller. See Defining Network Profiles for the HTTP/S Layer 7 Ingress Controller.

Configure the Client Source IP Address

The X-Forwarded-For HTTP header field is used to identify the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer.

In network profile, the x_forwarded_for parameter can be enabled to ensure that the client IP address will be set in the HTTP request header. The x_forwarded_for parameter is useful in situations where it is important to know the source IP address of the client request, such as for auditing purposes.

The x_forwarded_for parameter type is String that accepts "insert" and "replace". Any other type will be rejected. Missing entry is accepted.

If set to “insert”, the client source IP will be appended (comma separated) to the existing set of client source IP addresses. If set to “replace”, any existing client source IP address will be replaced with the current client source IP address.

For example, with the following network profile the source IP address of the client will be appended to the existing set of client source IP addresses:

{
  "name": "example-network-profile",
  "description": "x_forwarded_for insert",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "x_forwarded_for": "insert"
      }
    }
  }
}

For example, with the following network profile the existing source IP address of the client will replace all other client source IP entries:

{
  "name": "example-network-profile",
  "description": "x_forwarded_for replace",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "x_forwarded_for": "replace"
      }
    }
  }
}

Configure the Maximum Number of Layer 4 Load Balancer Virtual Servers

The default NSX-T Load Balancer behavior is that auto-scaling is unlimited. This means that the number of layer 4 virtual servers that can be deployed is governed only by the capacity of the Edge Cluster where the load balancer service is deployed. As a result, in theory it is possible for a single Kubernetes cluster to use up all of the layer 4 virtual servers that the Edge Cluster can support.

The max_l4_service parameter sets the upper limit for the number of virtual servers that can be used by a Kubernetes cluster. You can use this parameter to limit the number of virtual servers that can be created per Kubernetes cluster.

The max_l4_lb_service data type is an integer. The value must be larger or equal to 1. Missing entry is accepted.

For example, the following network profile uses the max_l4_lb_service parameter to limit the number of layer 4 virtual servers to 100 per cluster:

{
  "name": "example_network_profile",
  "description": "max_l4_lb_service",
  "parameters" : {
     "cni_configurations": {
         "type": "nsxt",
         "parameters": {
           "max_l4_lb_service": 100
         }
     }
  }
}

Configure the Layer 4 Persistence Type

The l4_persistence_type is used to set connection stickiness based on source_ip.

The l4_persistence_type data type is string. The only accepted value is source_ip.

{
  "name": "example_network_profile",
  "description": "l4_persistence_type",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
        "l4_persistence_type": "source_ip"
      }
    }
  }
}

Configure the Layer 4 Load Balancer Algorithm

The l4_lb_algorithm is used to set the algorithm type for the layer 4 NSX-T Load Balancer service.

The l4_lb_algorithm data type is string enumeration that accepts one of the following values:

  • "round_robin" (default)
  • "least_connection"
  • "ip_hash"
  • "weighted_round_robin"

For example, the following network profile specifies the weighted_round_robin as the load balancer algorithm:

{
  "name": "example_network_profile",
  "description": "l4_lb_algorithm",
  "parameters": {
    "cni_configurations": {
      "type": "nsxt",
      "parameters": {
        "l4_lb_algorithm": "weighted_round_robin"
      }
    }
  }
}

Please send any feedback you have to pks-feedback@pivotal.io.