Creating and Deleting Network Profiles (NSX-T Only)

Note: As of v1.8, Enterprise PKS has been renamed to VMware Tanzu Kubernetes Grid Integrated Edition. Some screenshots in this documentation do not yet reflect the change.

Page last updated:

This topic describes how VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) administrators can create and delete network profiles for Kubernetes clusters provisioned by TKGI on vSphere with NSX‑T integration.

This topic also describes the use cases for when a TKGI administrator should use a network profile.

Prerequisite

TKGI supports network profiles on TKGI on vSphere with NSX‑T only.

To manage TKGI network profiles you must be either a cluster manager or cluster administrator:

  • To create or delete a network profile, you must be a cluster administrator, pks.clusters.admin.

  • To use or manage network profiles, you must be a cluster manager, pks.clusters.manage or a cluster administrator, pks.clusters.admin.

Note: If a cluster manager, pks.clusters.manage, attempts to create or delete a network profile, the following error occurs: “You do not have enough privileges to perform this action. Please contact the TKGI administrator.

Overview

You can use network profiles to customize your TKGI Kubernetes clusters on vSphere with NSX‑T.

Cluster administrators can administer network profiles in the following ways:

Cluster administrators can also use and manage network profiles in all the ways that a cluster manager can:

For information on when to use network profiles, see Network Profile Use Cases below.

Create a Network Profile

To create a network profile in TKGI:

  1. Create a network profile configuration JSON file with the following content:

    {
        "name": "PROFILE-NAME",
        "description": "PROFILE-DESCRIP",
        "parameters": {
    
        }
    }
    

    Where:

    • PROFILE-NAME is the internal name for your network profile.
    • PROFILE-DESCRIP is an internal description for your network profile.
  2. Edit the file to specify your network parameters. For information about the available network parameters, see Network Profile Parameters below.

  3. Review your network profile configuration carefully. You can modify only a cluster’s pod_ip_block_ids network profile parameter after applying a network profile to a cluster.

  4. To create a network profile from your network profile configuration, run the following TKGI CLI command:

    tkgi create-network-profile PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION
    

    Where PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION is the path to your network profile configuration file.

    For example:

    $ tkgi create-network-profile np-routable-pods.json
    
    Network profile example-network-profile successfully created
    
  5. Store a copy of your network profile configuration in case you need to modify the network profile in the future.

Cluster managers can create new clusters with your network profile and assign your network profile to existing clusters. For information on managing network profiles, see Using and Managing Network Profiles.

Network Profile Parameters

TKGI supports the following top-level network profile parameters.

Note: Only the pod_ip_block_ids parameter can be updated after applying a network profile to a cluster.

Parameter Type Description
name String Name of the network profile.
description String Description of the network profile.
parameters Map One or more name-value pairs.
lb_size String Size of the NSX-T load balancer service: "small" (default), "medium", and "large".
pod_ip_block_ids String Array of Pod IP Block UUIDs.
pod_subnet_prefix Integer Size of the Pods IP Block subnet.
pod_routable Boolean Make routable the Pods subnet. Default is false.
fip_pool_ids String Array of floating IP pool UUIDs defined in NSX-T.
t0_router_id String Tenant Tier-0 Router UUID defined in NSX-T.
master_vms_nsgroup_id String Namespace Group UUID as defined in NSX-T.
nodes_dns String Array (up to 3) of DNS server IP addresses for lookup of Kubernetes nodes and pods.
dns_lookup_mode String DNS lookup mode: "API" for Kubernetes API load balancer and "API_INGRESS" for ingress controller.
ingress_prefix String Ingress controller hostname prefix for DNS lookup.
single_tier_topology Boolean Use a single Tier-1 Router per cluster (shared). Default is true.
infrastructure_networks String Array of IP addresses and subnets for Node Networks for use with a Shared Tier-1 topology in a Multi-Tier-0 environment.
cni_configurations Map Map containing key-value pairs for configuring NCP (see table below).

cni_configurations Network Profile Parameters

TKGI supports the following cni_configurations parameters:

Parameter Type Description
type String Only the constant "nsxt" is accepted.
parameters Map One or more name-value pairs for NCP settings.
x_forwarded_for String Use the same source IP for calling clients. Accepts "insert" and "replace".
nsx_lb Boolean Use NSX-T layer 4 virtual server for each Kubernetes service of type LoadBalancer. Default is true.
nsx_ingress_controller Boolean Use NSX-T layer 7 virtual server as the ingress controller for the Kubernetes cluster. Default is true.
ingress_ip String IP address to use for the ingress controller.
log_settings Map Parameters for configuring NCP logging.
log_level String Accepts: "INFO", "WARNING", "DEBUG", "ERROR", and "CRITICAL".
log_dropped_traffic Boolean Log dropped firewall traffic. Default is false.
ingress_persistence_settings String Parameters for customizing Layer 7 persistence.
persistence_type String Specify the ingress persistence type: "none", "cookie", or "source_ip".
persistence_timeout Integer Persistence timeout interval in seconds.
max_l4_lb_service Integer Limit the maximum number of layer 4 virtual servers per cluster. Minimum is 1.
l4_persistence_type String Connection stickiness based on source_ip.
l4_lb_algorithm String Layer 4 load balancer behavior: "round_robin" (default), "least_connection", "ip_hash", "weighted_round_robin".
top_firewall_section_marker String UUID of the top section-id for the distributed firewall (DFW) section as defined in NSX-T.
bottom_firewall_section_marker String UUID of the bottom section-id for the distributed firewall (DFW) section as defined in NSX-T.

Network Profile Example

The following is an example of a complete network profile JSON configuration:

{
    "name": "example-network-profile",
    "description": "Example Network Profile with All Available Parameters -- FOR ILLUSTRATION PURPOSES ONLY",
    "parameters": {
        "lb_size": "large",
        "pod_ip_block_ids": [
            "ebe78a74-a5d5-4dde-ba76-9cf4067eee55",
            "ebe78a74-a5d5-4dde-ba76-9cf4067eee56" ],
        "pod_subnet_prefix": 27,
        "pod_routable": true,
        "fip_pool_ids": [
            "e50e8f6e-1a7a-45dc-ad49-3a607baa7fa0",
            "ebe78a74-a5d5-4dde-ba76-9cf4067eee55" ],
        "t0_router_id": "5a7a82b2-37e2-4d73-9cb1-97a8329e1a90",
        "master_vms_nsgroup_id": "9b8d535a-d3b6-4735-9fd0-56305c4a5293",
        "node_ip_block_ids": [
            "2250dc43-63c8-4bb8-b8cf-c6e12ccfb7de", "3d577e5c-dcaf-4921-9458-d12b0e1318e6" ],
        "node_routable": true,
        "node_subnet_prefix": 20,
        "nodes_dns": [
            "8.8.8.8", "192.168.115.1", "192.168.116.1" ],      
        "dns_lookup_mode": "API_INGRESS",
        "ingress_prefix": "ingress",
        "single_tier_topology": true,
        "infrastructure_networks": [
            "30.0.0.0/24",
            "192.168.111.0/24",
            "192.168.115.1" ],
        "cni_configurations": {
            "type": "nsxt",
            "parameters": {
                "nsx_lb": false,
                "nsx_ingress_controller": true,         
                "x_forwarded_for": "insert",
                "ingress_ip": "192.168.160.212",
                "log_settings": {
                    "log_level": "DEBUG",
                "ingress_persistence_settings": {
                    "persistence_type": "cookie",
                    "persistence_timeout": 1 },
                "max_l4_lb_service": 10,
                "l4_persistence_type": "source_ip",
                "l4_lb_algorithm": "weighted_round_robin",
                "top_firewall_section_marker":"section-id",
                "bottom_firewall_section_marker":"section-id"                         
            }
        }
    }
}

Note: This example network profile is for illustration purposes only. It is not intended to be used as a template for a network profile configuration.

Update an Existing Network Profile

To update an existing cluster’s network profile:

  1. Confirm the Network Profile Property Supports Updates
  2. Create a Modified Network Profile Configuration
  3. Create a Modified Network Profile
  4. Update the Cluster With a Modified Network Profile

Confirm the Network Profile Property Supports Updates

If a network profile has been applied to an existing cluster, you can update only the following network profile parameters:

  • pod_ip_block_ids:

    The primary use case for updating the network profile pod_ip_block_ids field is to add additional IP addresses for pods when a cluster is at or near the point exhausting all available public IP addresses for pods.

    You can change the pod_ip_block_ids array parameter as follows:

    • Add more IP Block IDs in the array
    • Reorder the IP Block IDs in the array

    You cannot remove existing pod IP block IDs from a cluster by creating and assigning a new network profile with pod_ip_block_ids array values removed.

    For more information on modifying a network profile pod_ip_block_ids field, see Add Pod IPs in Customizing Pod Networks. For more information on the pod_ip_block_ids field, see Network Profile Parameters above.


You cannot update your existing clusters with any other network profile changes. For more information, see Update-Cluster Network Profile Validation Rules below.

Create a Modified Network Profile Configuration

To create a modified network profile configuration file:

  1. Make a copy of your original network profile configuration file.

    If it is not possible to obtain the original network profile, create a new network profile with the original values in all of the fields.
  2. Change the name field to a unique name.
  3. If updating the pod_ip_block_ids field, reorder the IP Block IDs or add additional IP Block IDs.

    For example, the following network profile has two pod_ip_block_ids, the first is the original IP block used when creating the cluster, and the second is the new IP block to use for pods.

    {
        "description": "Example network profile for adding pod IP addresses to an existing cluster",
        "name": "pod-ips-add",
        "parameters": {
          "pod_ip_block_ids": [
            "ebe78a74-a5d5-4dde-ba76-9cf4067eee55",
            "ebe78a74-a5d5-4dde-ba76-9cf4067eee56"
          ]
        }
    }
    

    Note: Update only network profile properties that support being updated.

    For more information on configuring a network profile, see Network Profile Parameters above.

  4. Review and save the network profile configuration file.

  5. Store a copy of your network profile configuration in case you need to modify the network profile in the future.

Create a Modified Network Profile

To create a network profile from a configuration file:

  1. Run the following TKGI CLI command:

    tkgi create-network-profile PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION
    

    Where PATH-TO-YOUR-NETWORK-PROFILE-CONFIGURATION is the path to your network profile configuration file.

Update the Cluster with a Modified Network Profile

To update a cluster with a modified network profile:

  1. To apply the network profile created above to your cluster, run the following command::

    tkgi update-cluster CLUSTER-NAME --network-profile NETWORK-PROFILE-NAME
    

    Where:

    • CLUSTER-NAME is the unique name of your cluster.
    • NETWORK-PROFILE-NAME is the name of the network profile you want to use for your cluster.

TKGI validates the network profile before updating the cluster with the new network profile. For more information, see Update-Cluster Network Profile Validation Rules below.

Update-Cluster Network Profile Validation Rules

TKGI uses strict validation rules before applying a network profile to a cluster with an existing network profile:

  • If a field in the original network profile is empty, the system ignores the empty field even if the field is included in the new network profile.
  • If a field in the new network profile is empty, the system ignores the field even if the field is not empty in the original network profile.
  • If the pod_ip_block_ids field in the new network profile contains the same entries as the existing network profile, the entry passes validation.
  • If a field in the new network profile conflicts with the field in the existing network profile, the system reports the conflict and fails the validation.

Delete a Network Profile

TKGI administrators can delete a network profile that is not in use.

To delete a network profile:

  1. Run the following TKGI CLI command:

    tkgi delete-network-profile NETWORK-PROFILE-NAME
    

    Where NETWORK-PROFILE-NAME is the name of the network profile you want to delete.

Note: You cannot delete a network profile that is in use.

Network Profile Use Cases

Network profiles let you customize configuration parameters for Kubernetes clusters provisioned by TKGI on vSphere with NSX‑T.

You can apply a network profile to a Kubernetes cluster for the following scenarios:

Topic Description
Size a Load Balancer Customize the size of the NSX-T load balancer service that is created when a Kubernetes cluster is provisioned.
Customizing Pod Networks Customize Kubernetes Pod Networks, including adding pod IP addresses, subnet size, and routability.
Customize Node Networks Customize Kubernetes Node Networks, including the IP addresses, subnet size, and routability.
Customize Floating IP Pools Specify a custom floating IP pool.
Configure Bootstrap NSGroups Specify an NSX-T Namespace Group where Kubernetes master nodes will be added to during cluster creation.
Configure Edge Router Selection Specify the NSX-T Tier-0 router where Kubernetes node and Pod networks will be connected to.
Specify Nodes DNS Servers Specify one or more DNS servers for Kubernetes clusters.
Configure DNS for Pre-Provisioned IPs Configure DNS lookup of the Kubernetes API load balancer or ingress controller.
Configure the TCP Layer 4 Load Balancer Configure layer 4 TCP load balancer settings; use third-party load balancer.
Configure the HTTP/S Layer 7 Ingress Controller Configure layer 7 HTTP/S ingress controller settings; use third-party ingress controller.
Define DFW Section Markers Configure top or bottom section markers for explicit DFW rule placement.
Configure NCP Logging Configure NCP logging.
Dedicated Tier-1 Topology Use dedicated Tier-1 routers, rather than a shared router, for each cluster’s Kube node, Namespace, and NSX-T load balancer.

Please send any feedback you have to pks-feedback@pivotal.io.