LATEST VERSION: v1.3 - RELEASE NOTES
Pivotal Container Service v1.3

Upgrading PKS with NSX-T to NSX-T v2.4.0.1

Page last updated:

This topic describes how to upgrade your PKS with NSX-T environment from NSX-T v2.3 to v2.4.

Step 0: Prepare to Upgrade

Review related documentation in preparation for the upgrade of PKS:

  1. Review the PKS Release Notes for the supported upgrade path and known issues.
  2. Review the VMware Product Interoperability Matrix for PKS in the VMware documentation.
  3. Review the NSX-T 2.4 release notes.

Step 1: Upgrade to PKS v1.3.6

Upgrade the PKS tile from a supported version to to PKS v1.3.6. When you upgrade the PKS tile, the target version of NCP is installed (v2.4.0 in this case). This must be done before you upgrade to NSX-T v2.4.x.

If you are performing the upgrade during a maintenance window, it is not necessary to upgrade the Kubernetes clusters at this time, so you can deselect the upgrade all clusters errand for PKS. However, if you want your Kubernetes clusters to be upgraded immediately, ensure that the upgrade all clusters errand is enabled.

To upgrade the PKS tile to v1.3.6:

  1. Download the PKS v1.3.6 tile from the Pivotal Network.
  2. Upload the PKS v1.3.6 tile to Ops Manager.
  3. Stage the 1.3.6 tile for deployment.
  4. Review pending changes.
  5. Apply changes.

Step 2: Verify Supported vSphere Versions and Required ESXi Patches

NSX-T v2.4.x supports the following vSphere versions with patches: - VMware vSphere 6.7 EP06 (Release name: ESXi670-201901001) is the minimum supported version with NSX-T 2.4.0 (KB 2143832) - VMware vSphere 6.5 P03 (Release Name: ESXi650-201811002) is the minimum supported version with NSX-T 2.4.0 (KB 2143832)

Supported vSphere versions for NSX-T 2.4.0

Refer to the VMware Product Interoperability Matrices. For example, hover over the Information icon for vSphere 6.7 U1 and NSX-T 2.4 and you will see the following message: “VMware vSphere 6.7 EP06 (Release name: ESXi670-201901001) is the minimum supported version with NSX-T 2.4.0 (KB 2143832).”

Supported vSphere versions for NSX-T 2.4.0

For details on the ESXi v6.7 U1 EP06 patch, refer to the VMware KB article Build numbers and versions of VMware ESXi/ESX.

To perform the patch upgrade using vCenter, refer to the vSphere Upgrade Manager documentation for guidance on applying the patch. See also the VMware ESXi Upgrade documentation for additional details. To patch ESXi hosts in an air-gapped environment, use Zip files as described in the VMware ESXi documenation

Step 3: Upgrade from NSX-T v2.3.1 to NSX-T v2.4

Upgrade NSX-T from v2.3.1 to v2.4.0.1. You must use at least version v2.4.0.1 due to the following known issue in v2.4.0: https://kb.vmware.com/s/article/67449. See the Upgrade Path section of the Release Notes for information on obtaining the hot-patch.

To perform the upgrade, refer to Upgrading NSX-T Data Center in the VMware documentation.

Note: When upgrading NSX-T, at the stage that the ESXi Transport Nodes are upgraded (“Hosts”), you may want to create a different host group for each ESXi host in the correct order so that hosts in maintenance mode only get upgraded. In vCenter, put each EXSi Transport Node (TN) host into maintenance mode, 1 at a time. Create the host group for that ESXi host and upgrade only it, then remove it from maintenance mode. Repeat this process for all ESXi TN hosts.

Note: Once you upgrade to NSX-T 2.4, the T0 router(s) and all other management plane objects can be seen only from the Advanced Networking Configuration tab. They will not be migrated to the new Policy UI.

Note: There are architectural changes in NSX 2.4. The NSX Controller is now a component of the NSX Manager. Once the NSX-T upgrade is complete, you will have a single NSX-T Manager node. Power off the NSX Controllers. At the end of the upgrade, you can delete the NSX Controller VMs. For more information, see Delete NSX Controllers in the NSX-T documentation.

Note: Once the upgrade to NSX 2.4 is complete, you may want to verify that your PKS environment is functioning properly by logging in to PKS and creating a small test cluster. If you cannot do this, troubleshoot the upgrade before proceeding. For more information, see Troubleshooting Upgrade Failures in the NSX-T documentation.

Step 4: Deploy Two Additional NSX Managers

With NSX-T v2.4, the NSX Controller component is now part of the NSX Manager. Previously the NSX Manager was a singleton, and HA was achieved using multiple NSX Controllers. With NSX-T v2.4, since the standalone NSX Controller component is no longer used, to achieve HA you need to deploy multiple (three) NSX Managers. Refer to the Upgrading NSX-T Data Center documentation for guidance on adding additional NSX Managers.

Note: When you add additional NSX Managers, the system prompts you to enter a Compute Manager, which is a vCenter Server. For more information, see Add a Compute Manager in the NSX-T documentation.

Step 5: Configure the NSX Manager VIP

Since you have deployed two additional NSX Managers (for a total of three), you need create a virtual IP address that can be used as a single endpoint to access the NSX Management cluster.

To create a VIP for the NSX Management cluster:

  • Log in to the NSX Manager interface.
  • Go to System > Overview.
  • Select Virtual IP > Edit.
  • Enter a publicly routable IP address, such as 10.40.206.5.
  • Click Save.

At this point in time, you can connect to any NSX-T manager using its own IP address, or use the VIP to connect to NSX-T Manager. Both methods work. However, note that the VIP is associated with a single NSX Manager. To determine which NSX Manager the VIP is associated with, select the Virtual IP.

VIP Association

Step 6: Generate and Register a New NSX Manager CA Cert with the Cluster API

Both the BOSH Director tile and the PKS tile expect the NSX Manager CA certificate. However, the current NSX Manager CA certificate is associated with the original NSX Manager IP address. You need to generate a new NSX Manager CA cert using the VIP address, then register this certificate with NSX-T using the Cluster Certificate API.

To generate a new NSX Manage CA certificate and private key using the VIP address, follow the instructions in the Generate NSX CA Cert PKS documentation. Make sure you use the VIP address, such as 10.40.206.5 in our example above.

Once you have created the new CA certificate using the VIP address, import the new CA certificate to the NSX Manager. Refer to Import the Certificate to NSX Manager for instructions on doing this.

Once you have imported the NSX Manager certificate, register this certificate with the NSX Management cluster using a cURL command against the Cluster Certificate API.

Note: In general the instructions provided in the Register the Certificate with NSX Manager documentation can be followed, with the exception that API endpoint is changed to the Cluster Certificate API.

First, create environment variables for the VIP address and the certificate ID:

export NSX_MANAGER_IP_ADDRESS=10.40.206.5

export CERTIFICATE_ID="63bb6646-052c-49df-b603-64d7e5bdb5bf"

Next, register the new NSX-T Manager CA cert using a cURL request to the Cluster Certificate API:

curl --insecure -u admin:'PASSWORD' -X POST "https://$NSX_MANAGER_IP_ADDRESS/api/v1/cluster/api-certificate?action=set_cluster_certificate&certificate_id=$CERTIFICATE_ID"

The certificate will be registered with the NSX Manager that the VIP address is associated with.

To verify, using a browser go to the VIP address of the NSX Manager. Login and check that the new cert is used by the site (accessed using the VIP address).

To further verify, SSH to each NSX Manager host and run the following two commands. All certificates returned should be the same.

get certificate api
get certificate cluster

Step 7: Update PKS and BOSH with New NSX Manager Cert and VIP

The last procedure in the upgrade process is to modify the BOSH Tile and the PKS Tile with the new VIP address for the NSX Manager and the new NSX-T Manager CA cert (using VIP info). Apply the changes and ensure that the Upgrade all clusters errand is selected, then deploy PKS.

To update the BOSH tile:

  1. Log into Ops Manager.
  2. In the BOSH Director tile, select the vCenter Configuration tab.
  3. In the NSX Address field, enter the VIP address for the NSX Management Cluster.
  4. In the NSX CA Cert field, enter the new CA certificate for the NSX Management Cluster that uses the VIP address.
  5. Save the BOSH tile changes. Update BOSH with VIP and Cert

To update the PKS tile:

  1. Log into Ops Manager.
  2. In the PKS tile, select the Networking tab.
  3. In the NSX Manager hostname field, enter the VIP address for the NSX Management Cluster.
  4. In the NSX Manager CA Cert field, enter the new CA certificate for the NSX Management Cluster (that uses the VIP address).
  5. Save the PKS tile changes. Update PKS with VIP and Cert

Step 8: Upgrade all Kubernetes Clusters

Once you have updated the PKS and BOSH tiles, apply the changes. Be sure to run the “Upgrade all [Kubernetes] clusters errand”. Doing so will allow NCP configurations on all Kubernetes clusters to be updated with the new NSX-T Management Cluster VIP and CA certificate.

To complete the upgrade:

  1. Go to the Installation Dashboard in Ops Manager.
  2. Click Review Pending Changes.
  3. Expand the Errands list for PKS.
  4. Ensure that the Upgrade all clusters errand is selected.
  5. Click Apply Changes. Upgrade all Kubernetes clusters

Step 9: Verify PKS Upgrade

Once the upgrade is complete, verify that NCP configuration is automatically updated with the new VIP (instead of individual NSX-T Manager node IP address).

To do this, run a command similar to the following for each Kubernetes cluster (service-instance_UUID):

bosh ssh master/0 -d service-instance_d9b662d0-23e1-4239-b641-ed20ee62e692

Note the “nsx_api_managers” address. It should be the VIP.

Step 10: Update PKS and Kubernetes CLIs

Update the PKS and Kubernetes CLIs on any local machine where you run commands that interact with your upgraded version of PKS.

To update your CLIs, download and re-install the PKS and Kubernetes CLI distributions that are provided with PKS on Pivotal Network.

For more information about installing the CLIs, see the following topics:


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub