Upgrading PKS with NSX-T to NSX-T v2.4.0.1

Page last updated:

Warning: Pivotal Container Service (PKS) v1.3 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy.
To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to upgrade your PKS with NSX-T environment from NSX-T v2.3 to v2.4.

Step 0: Prepare to Upgrade

Review related documentation in preparation for the upgrade of PKS:

  1. Review the PKS Release Notes for the supported upgrade path and known issues.
  2. Review the VMware Product Interoperability Matrix for PKS in the VMware documentation.
  3. Review the NSX-T 2.4 release notes.

Step 1: Upgrade to PKS v1.3.6

Upgrade the PKS tile from a supported version to to PKS v1.3.6. When you upgrade the PKS tile, the target version of NCP is installed (v2.4.0 in this case). This must be done before you upgrade to NSX-T v2.4.x.

If you are performing the upgrade during a maintenance window, it is not necessary to upgrade the Kubernetes clusters at this time, so you can deselect the upgrade all clusters errand for PKS. However, if you want your Kubernetes clusters to be upgraded immediately, ensure that the upgrade all clusters errand is enabled.

To upgrade the PKS tile to v1.3.6:

  1. Download the PKS v1.3.6 tile from the Pivotal Network.
  2. Upload the PKS v1.3.6 tile to Ops Manager.
  3. Stage the 1.3.6 tile for deployment.
  4. Review pending changes.
  5. Apply changes.

Step 2: Verify Supported vSphere Versions and Required ESXi Patches

NSX-T v2.4.x supports the following vSphere versions with patches:

  • VMware vSphere 6.7 EP06 (Release name: ESXi670-201901001) is the minimum supported version with NSX-T 2.4.0 (KB 2143832)
  • VMware vSphere 6.5 P03 (Release Name: ESXi650-201811002) is the minimum supported version with NSX-T 2.4.0 (KB 2143832)

To verify the installation of supported vSphere versions and ESXi patches, perform the following steps:

  1. Refer to the VMware Product Interoperability Matrices. Supported vSphere versions for NSX-T 2.4.0

  2. Hover over the Information icon for vSphere 6.7 U1 and NSX-T 2.4: version-specific compatibility information is displayed. For example, see the message “VMware vSphere 6.7 EP06 (Release name: ESXi670-201901001) is the minimum supported version with NSX-T 2.4.0 (KB 2143832)” below:

    Supported vSphere versions for NSX-T 2.4.0

    For details on the ESXi v6.7 U1 EP06 patch, refer to the VMware KB article Build numbers and versions of VMware ESXi/ESX.

  3. Apply required ESXi patch upgrades:

Step 3: Upgrade from NSX-T v2.3.1 to NSX-T v2.4

Upgrade NSX-T from v2.3.1 to v2.4.0.1.

  1. To upgrade NSX-T from v2.3.1 to v2.4.0.1, refer to Upgrading NSX-T Data Center in the VMware documentation.

Note: You must use at least version v2.4.0.1 due to the following known issue in v2.4.0: Important information before upgrading to NSX-T Data Center 2.4.0 (67449). See the Upgrade Path section of the Release Notes for information on obtaining the hot-patch.

Note: When upgrading NSX-T, at the stage that the ESXi Transport Nodes are upgraded (“Hosts”), you may want to create a different host group for each ESXi host in the correct order so that hosts in maintenance mode only get upgraded. In vCenter, put each EXSi Transport Node (TN) host into maintenance mode, 1 at a time. Create the host group for that ESXi host and upgrade only it, then remove it from maintenance mode. Repeat this process for all ESXi TN hosts.

Note: Once you upgrade to NSX-T 2.4, the T0 router(s) and all other management plane objects can be seen only from the Advanced Networking Configuration tab. They will not be migrated to the new Policy UI.

Note: There are architectural changes in NSX 2.4. The NSX Controller is now a component of the NSX Manager. Once the NSX-T upgrade is complete, you will have a single NSX-T Manager node. Power off the NSX Controllers. At the end of the upgrade, you can delete the NSX Controller VMs. For more information, see Delete NSX Controllers in the NSX-T documentation.

Note: Once the upgrade to NSX 2.4 is complete, you may want to verify that your PKS environment is functioning properly by logging in to PKS and creating a small test cluster. If you cannot do this, troubleshoot the upgrade before proceeding. For more information, see Troubleshooting Upgrade Failures in the NSX-T documentation.

Step 4: Deploy Two Additional NSX Managers

With NSX-T v2.4, the NSX Controller component is now part of the NSX Manager. Previously the NSX Manager was a singleton, and HA was achieved using multiple NSX Controllers. With NSX-T v2.4, since the standalone NSX Controller component is no longer used, to achieve HA you need to deploy multiple (three) NSX Managers.

  1. To deploy additional NSX Managers, refer to the Upgrading NSX-T Data Center documentation for guidance.

Note: When you add additional NSX Managers, the system prompts you to enter a Compute Manager, which is a vCenter Server. For more information, see Add a Compute Manager in the NSX-T documentation.

Step 5: Configure the NSX Manager VIP

Since you have deployed two additional NSX Managers (for a total of three), you need create a virtual IP address that can be used as a single endpoint to access the NSX Management cluster.

To create a VIP for the NSX Management cluster:

  1. Log in to the NSX Manager interface.
  2. Go to System > Overview.
  3. Select Virtual IP > Edit.
  4. Enter a publicly routable IP address, such as
  5. Click Save.

    At this point in time, you can connect to any NSX-T manager using its own IP address, or use the VIP to connect to NSX-T Manager. Both methods work. However, note that the VIP is associated with a single NSX Manager.

  6. To determine which NSX Manager the VIP is associated with, select the Virtual IP.

VIP Association

Step 6: Generate and Register a New NSX Manager CA Cert with the Cluster API

Both the BOSH Director tile and the PKS tile expect the NSX Manager CA certificate. However, the current NSX Manager CA certificate is associated with the original NSX Manager IP address. You need to generate a new NSX Manager CA cert using the VIP address, then register this certificate with NSX-T using the Cluster Certificate API:

  1. To generate a new NSX Manage CA certificate and private key using the VIP address, follow the instructions in the Generate NSX CA Cert PKS documentation. Make sure you use the VIP address, such as in our example above.

  2. Import the new CA certificate to the NSX Manager. Refer to Import the Certificate to NSX Manager for instructions on doing this.

  3. Register this certificate with the NSX Management cluster using a cURL command against the Cluster Certificate API.

    Note: In general, you can follow the instructions provided in Register the Certificate with NSX Manager, with the exception that API endpoint is changed to the Cluster Certificate API.

  4. Create environment variables for the VIP address and the certificate ID:

    export CERTIFICATE_ID="63bb6646-052c-49df-b603-64d7e5bdb5bf"
  5. To register the new NSX-T Manager CA cert with the NSX Manager, run the following Cluster Certificate API command:

    curl --insecure -u admin:'PASSWORD' -X POST "https://$NSX_MANAGER_IP_ADDRESS/api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=$CERTIFICATE_ID"

    Where PASSWORD is your NSX Manager admin account password.

    For example:

    export CERTIFICATE_ID="73bb66t6-0523-51df-k603-64g9g5bdb5rl"
    curl --insecure -u admin:'PASSWORD' -X POST "https://$NSX_MANAGER_IP_ADDRESS/api/v1/cluster/api-certificate? \

  6. To register the new certificate with your other NSX-T Manager appliances, repeat the process for each appliance, using each NSX Manager’s IP address as $NSX_MANAGER_IP_ADDRESS.

  7. To verify, using a browser go to the VIP address of the NSX Manager. Login and check that the new cert is used by the site (accessed using the VIP address).

  8. To further verify, SSH to each NSX Manager host and run the following two commands. All certificates returned should be the same.

    get certificate api
    get certificate cluster

Step 7: Update PKS and BOSH with New NSX Manager Cert and VIP

The last procedure in the upgrade process is to modify the BOSH Tile and the PKS Tile with the new VIP address for the NSX Manager and the new NSX-T Manager CA cert (using VIP info). Apply the changes and ensure that the Upgrade all clusters errand is selected, then deploy PKS.

To update the BOSH tile:

  1. Log into Ops Manager.
  2. In the BOSH Director tile, select the vCenter Configuration tab.
  3. In the NSX Address field, enter the VIP address for the NSX Management Cluster.
  4. In the NSX CA Cert field, enter the new CA certificate for the NSX Management Cluster that uses the VIP address.
  5. Save the BOSH tile changes. Update BOSH with VIP and Cert

To update the PKS tile:

  1. Log into Ops Manager.
  2. In the PKS tile, select the Networking tab.
  3. In the NSX Manager hostname field, enter the VIP address for the NSX Management Cluster.
  4. In the NSX Manager CA Cert field, enter the new CA certificate for the NSX Management Cluster (that uses the VIP address).
  5. Save the PKS tile changes. Update PKS with VIP and Cert

Step 8: Upgrade all Kubernetes Clusters

Once you have updated the PKS and BOSH tiles, apply the changes. Be sure to run the “Upgrade all [Kubernetes] clusters errand”. Doing so will allow NCP configurations on all Kubernetes clusters to be updated with the new NSX-T Management Cluster VIP and CA certificate.

To complete the upgrade:

  1. Go to the Installation Dashboard in Ops Manager.
  2. Click Review Pending Changes.
  3. Expand the Errands list for PKS.
  4. Ensure that the Upgrade all clusters errand is selected.
  5. Click Apply Changes. Upgrade all Kubernetes clusters

Step 9: Verify PKS Upgrade

Once the upgrade is complete, verify that NCP configuration is automatically updated with the new VIP (instead of individual NSX-T Manager node IP address).

  1. To verify the NCP configuration has been updated with the new VIP, run a command for each Kubernetes cluster (service-instance_UUID):

    bosh ssh master/0 -d SERVICE-INSTANCE-UUID

    Where SERVICE-INSTANCE-UUID is the Kubernetes cluster UUID.

    For example:

    $ bosh ssh master/0 -d service-instance_d9b662d0-23e1-4239-b641-ed20ee62e692

    The returned “nsx_api_managers” address should be the new VIP address.

Step 10: Update PKS and Kubernetes CLIs

Update the PKS and Kubernetes CLIs on any local machine where you run commands that interact with your upgraded version of PKS.

To update your CLIs, download and re-install the PKS and Kubernetes CLI distributions that are provided with PKS on Pivotal Network.

For more information about installing the CLIs, see the following topics:

Please send any feedback you have to pks-feedback@pivotal.io.