LATEST VERSION: v1.2 - RELEASE NOTES
Pivotal Container Service v1.2

Upgrading PKS with NSX-T

Page last updated:

This topic explains how to upgrade the Pivotal Container Service (PKS) tile for environments using vSphere with NSX-T.

PKS v1.2 supports NSX-T v2.3, which is the recommended NSX-T version. To upgrade to PKS v1.2, you must first upgrade to PKS v1.1.5 or later. For details, see the VMware Product Interoperability Matrix for PKS in the VMware documentation.

Note: When you upgrade PKS on vSphere with NSX-T, workloads in your Kubernetes cluster are unavailable while the NSX Edge nodes run the upgrade. Configure NSX Edge for high availability using Active/Standby mode to avoid workload downtime. For more information, see the Configure NSX Edge for High Availability (HA) section of Preparing NSX-T Before Deploying PKS.

Prepare to Upgrade

Before you begin upgrading the PKS tile, follow the steps below:

  1. Review the Release Notes for the version you are upgrading to.
  2. Verify the health of your Kubernetes environment by following the steps below:
    1. To verify that all nodes are in a ready state, run kubectl get nodes for all Kubernetes contexts.
    2. To verify that all pods are running, run kubectl get pods --all-namespaces for all Kubernetes contexts.
    3. To verify that all the processes are in a running state, run bosh -d MY-DEPLOYMENT instances --ps for each deployment, replacing MY-DEPLOYMENT with the deployment name. Your PKS deployment name begins with pivotal-container-service and includes a BOSH-generated hash.
  3. Make sure there are no issues with vSphere by following the steps below:
    1. Verify that datastores have enough space.
    2. Verify that hosts have enough memory.
    3. Verify that there are no alarms.
    4. Verify that hosts are in a good state.
  4. (Optional) Back up the environment using the procedures in the following topics:

Upgrade to Ops Manager v2.2.2+ or v2.3.1+ and PKS v1.1.5

Before you upgrade to PKS v1.2, you must upgrade PKS to v1.1.5+, upgrade NSX-T to v2.2, and upgrade Ops Manager to v2.2.2+ or v2.3.1+.

For more information, refer to the compatibility chart below:

PKS Version Compatible NSX-T Versions Compatible Ops Manager Versions
1.2 2.2, 2.3 2.2.2+, 2.3.1+
1.1.6 2.1, 2.2 2.1.x, 2.2.x
1.1.5 2.1, 2.2 2.1.x, 2.2.x
1.1.4 2.1 2.1.x, 2.2.x
1.1.3 2.1 2.1.0 - 2.1.6
1.1.2 2.1 2.1.x, 2.2.x
1.1.1 2.1 - Advanced Edition 2.1.0 - 2.1.6

Upgrade Ops Manager on PKS v1.1.5 and Later

If you already use PKS v1.1.5+, perform the steps below to upgrade NSX-T and Ops Manager:

  1. To upgrade to NSX-T v2.2, follow the procedures detailed in Upgrading NSX-T in the VMware documentation.

  2. To upgrade to Ops Manager v2.2.2+ or v2.3.1+, follow the procedures detailed in Upgrade Ops Manager and Installed Products to v2.2 or Upgrade Ops Manager and Installed Products to v2.3.

  3. Operators should add additional workloads and create an additional cluster to ensure that the PKS control plane remains functional. For more information about performing those actions, see About Workload Upgrades in Maintaining Workload Uptime and Creating Clusters.

You can monitor the PKS control plane VM by clicking the Pivotal Container Service tile, selecting Status tab, and reviewing the Pivotal Container Service VM’s data points. If any data points are at capacity, scale your deployment accordingly.

Upgrade Ops Manager on PKS v1.1.4 and Earlier

If you use PKS v1.1.4 or earlier, perform the steps below to upgrade PKS, NSX-T, and Ops Manager:

  1. To upgrade PKS v1.1.4 or earlier, follow the procedures detailed in Upgrade PKS in the PKS v1.1 documentation.

  2. To upgrade to NSX-T v2.2, follow the procedures detailed in Upgrading NSX-T in the VMware documentation.

  3. To upgrade to Ops Manager v2.2.2+ or v2.3.1+, follow the procedures detailed in Upgrade Ops Manager and Installed Products to v2.2 or Upgrade Ops Manager and Installed Products to v2.3.

  4. Operators should add additional workloads and create an additional cluster to ensure that the PKS control plane remains functional. For more information about performing those actions, see About Workload Upgrades in Maintaining Workload Uptime and Creating Clusters.

You can monitor the PKS control plane VM by clicking the Pivotal Container Service tile, selecting Status tab, and reviewing the Pivotal Container Service VM’s data points. If any data points are at capacity, scale your deployment accordingly.

Upgrade the PKS Tile

To upgrade PKS, you follow the same Ops Manager process that you use to install the tile for the first time. Your configuration settings migrate to the new version automatically. Follow the steps below to perform an upgrade.

  1. Review the Release Notes for the version you are upgrading to.

  2. Download the desired version of the product from Pivotal Network.

  3. Navigate to the Ops Manager Installation Dashboard and click Import a Product.

  4. Browse to the PKS product file and select it. Uploading the file takes several minutes. Upload PKS product file to Ops Manager

  5. Under the Import a Product button, click + next to Pivotal Container Service. This adds the tile to your staging area.

    Import the PKS product file

  6. Ops Manager adds the PKS tile to your staging area.

  7. (Optional) If the stemcell is not current, click the Missing stemcell link and follow the steps below: Verify successful import of PKS

    1. On the Stemcell Library page, click Import Stemcell. Import stemcell
    2. Select the PKS product and click Apply Stemcell to Products. Apply stemcell to PKS
    3. Verify that Ops Manager successfully applied the stemcell. Verify stemcell assignment
    4. Select the Installation Dashboard link to return to the Installation Dashboard. Verify status is green

Increase the Kubernetes Worker Node VM Size

The default Kubernetes worker node VM size provides insufficient disk space for PKS v1.1.5 or later on vSphere with NSX-T.

Note: If you do not increase the size of the Kubernetes worker node VM before you upgrade, the VM can run out of ephemeral disk space and cause the upgrade to fail.

Follow the steps below to increase the Kubernetes worker node VM size:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the Pivotal Container Service tile.

  3. Click Plan 1.

  4. Under Worker VM Type, select a VM type with a minimum disk size of 16 GB.

  5. (Optional) If you have additional plans configured, repeat this procedure for each plan.

Verify NSX-T Manager CA Certificate Settings

Follow the steps below:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the Pivotal Container Service tile.

  3. Click Networking.

  4. Under NSX Manager CA Cert, choose one of the following options:

    • Confirm that you have a valid NSX-T Manager CA certificate.
    • Select Disable SSL certificate verification.

      Note: You cannot choose both options. If you provide an NSX Manager CA certificate and also disable SSL certificate verification, the PKS upgrade fails.

Apply Changes to the PKS Tile

Return to the Installation Dashboard and do one of the following:

  • Apply Changes: Under Pending Changes, click Apply Changes. Apply changes and upgrade
  • Review Pending Changes [BETA]:
    1. Click Review Pending Changes.
    2. Click Pivotal Container Service.
    3. Click Apply Changes.

Verify the Upgrade

After you apply changes to the PKS tile and the upgrade is complete, verify that your Kubernetes environment is healthy and confirm that NCP is running on the master node VM.

For the BOSH CLI commands in this section, replace the text as follows:

  • MY-ENV: the alias you set for your BOSH Director. For more information, see Managing PKS Deployments with BOSH.
  • MY-DEPLOYMENT: the name of your PKS deployment. PKS deployment names begin with pivotal-container-service and include a unique BOSH-generated hash.
  • VM-NAME: your Kubernetes master node VM name.
  • ID: your Kubernetes master node VM ID. This is a unique BOSH-generated hash.

Verify Kubernetes Environment Health

Verify the health of your Kubernetes environment by following the steps below:

  1. To verify that all nodes are in a ready state, run kubectl get nodes for all Kubernetes contexts.
  2. To verify that all pods are running, run kubectl get pods --all-namespaces for all Kubernetes contexts.
  3. To verify that all the processes are in a running state, run bosh -d MY-DEPLOYMENT instances --ps for each deployment.

Verify NCP Changes

In PKS v1.1.5 and later, NCP runs as a BOSH host process. Each Kubernetes master node VM has one NCP process running. If your cluster has multiple master nodes, one NCP process is active while the others are on standby. For more information, see Architectural Changes.

Verify the NCP changes by following the steps below:

  1. From the Ops Manager VM, run bosh -e MY-ENV log-in.
  2. Run bosh -e MY-ENV -d MY-DEPLOYMENT vms.
  3. In the output of the previous command, find the Kubernetes master node VM name and ID.
  4. Run bosh -e MY-ENV -d MY-DEPLOYMENT ssh VM-NAME/ID.
  5. On the master node VM, run monit summary. Verify that you see Process: 'ncp' is running.
  6. To check if the NCP process is active or on standby, run /var/vcap/jobs/ncp/bin/nsxcli -c get ncp-master status.
  7. To restart the NCP process, run monit restart ncp.
  8. To verify that the NCP process restarts successfully, run monit summary.

(Optional) Upgrade vSphere

Upgrade vSphere from version 6.5 or 6.5 U1 to 6.5 U2.

For more information, see Upgrading vSphere in an NSX Environment in the VMware documentation.


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub