Upgrading Enterprise PKS with NSX-T

Page last updated:

This topic explains how to upgrade Enterprise Pivotal Container Service (Enterprise PKS) for environments using vSphere with NSX-T.

Warning: Each Kubernetes cluster contains a NSX-T Container Plug-in (NCP) which provides integration between NSX-T and Kubernetes. Before upgrading the clusters, the clusters will contains NCP version 2.4.x which is not compatible with NSX-T v2.5, hence, you cannot upgrade NSX-T v2.4.x to NSX-T v2.5 before upgrading all of your Kubernetes clusters. Upgrading all your clusters will result in the NCP component being upgraded to NCP v2.5 which will be compatible with both NSX-T 2.4.x and NSX-T v2.5.x. Once you complete the upgrade of all Kubernetes clusters managed by Enterprise PKS, you may upgrade to NSX-T v2.5. Any cluster that is not on upgraded before upgrading to NSX-T v2.5 will be orphaned and inoperable.

Overview

The supported upgrade paths to Enterprise PKS v1.5.0 are from Enterprise PKS v1.4.1 and later. Enterprise PKS v1.5.0 is compatible with Ops Manager v2.5.12 and later or Ops Manager v2.6.6 and later.

Warning: Do not manually upgrade your Kubernetes version. Enterprise PKS includes the compatible Kubernetes version.

Before you begin the upgrade, follow the procedures in the Prerequisites section to plan and prepare your upgrade.

After you complete the prerequisite steps, continue to the procedures in the During the Upgrade section. These steps guide you through the process of upgrading the Enterprise PKS tile, upgrading Ops Manager, importing an updated stemcell, and applying the changes to your deployment.

After you complete the upgrade, follow the procedures in the After the Upgrade section to verify that your upgraded Enterprise PKS deployment is running properly.

Prerequisites

Before you upgrade Enterprise PKS, you must do the following:

  1. Determine Your Upgrade Path
  2. Prepare to Upgrade

Determine Your Upgrade Path

Use the following table to determine your upgrade path to Enterprise PKS v1.5.

Target Ops Manager version Upgrade path
v2.5
  1. Upgrade to Ops Manager v2.5.12 and later.
  2. Upgrade to Enterprise PKS v1.5.
  3. Upgrade to NSX-T Data Center v2.5.
v2.6
  1. Upgrade to Ops Manager v2.6.6 and later.
  2. Upgrade to Enterprise PKS v1.5.
  3. Upgrade to NSX-T Data Center v2.5.

Prepare to Upgrade

If you have not already, complete all of the steps in the Upgrade Preparation Checklist for Enterprise PKS v1.5.

During the Upgrade

This section describes the steps required to upgrade to Enterprise PKS v1.5:

  1. Upgrade to Ops Manager v2.5.12+ or v2.6.6+
  2. Upgrade to Enterprise PKS v1.5
  3. Download and Import the Stemcell
  4. Verify Errand Configuration
  5. Apply Changes to the Enterprise PKS Tile

Upgrade Ops Manager to v2.5.12+ or v2.6.6+

Before you upgrade to Enterprise PKS v1.5, you must upgrade to Ops Manager v2.5.12 and later or v2.6.6 and later.

To upgrade Ops Manager, do the following:

  1. Complete the procedures in Upgrade Ops Manager and Installed Products to v2.5 or Upgrade Ops Manager and Installed Products to v2.6.

  2. Verify that the PKS control plane remains functional by performing the following steps:

    1. Add more workloads and create an additional cluster. For more information, see About Workload Upgrades in Maintaining Workload Uptime and Creating Clusters.
    2. Monitor the PKS control plane VM by clicking the Enterprise PKS tile, selecting the Status tab, and reviewing the Pivotal Container Service VM’s data points. If any data points are at capacity, scale your deployment accordingly.

Upgrade Enterprise PKS to v1.5

To upgrade to Enterprise PKS v1.5, follow the same Ops Manager process that you use to install the tile for the first time.

Your configuration settings migrate to the new version automatically. Follow the steps below to perform an upgrade.

  1. Review the Release Notes for the version you are upgrading to.

  2. Download the desired version of the product from Pivotal Network.

  3. Navigate to the Ops Manager Installation Dashboard and click Import a Product to upload the product file.

  4. Under the Import a Product button, click + next to Enterprise PKS. This adds the tile to your staging area.

Download and Import the Stemcell

Enterprise PKS v1.5 requires a Xenial stemcell.

If Ops Manager does not have the Xenial stemcell required for Enterprise PKS, the Enterprise PKS tile displays the message Missing stemcell.

Note: If the Stemcell Library in Ops Manager already has a compatible Xenial stemcell, the Missing stemcell link does not appear. You do not need to download or import a new stemcell and can skip this step.

To download and import a new Xenial stemcell, follow the steps below:

  1. On the Enterprise PKS tile, click the Missing stemcell link.

    Verify stemcell assignment

  2. In the Stemcell Library, locate the Enterprise PKS tile and note the required stemcell version.

  3. Navigate to the Stemcells for PCF (Ubuntu Xenial) page on Pivotal Network and download the required stemcell version for your IaaS.

  4. Return to the Installation Dashboard in Ops Manager and click Stemcell Library.

  5. On the Stemcell Library page, click Import Stemcell and select the stemcell file you downloaded from Pivotal Network.

  6. Select the Enterprise PKS tile and click Apply Stemcell to Products.

  7. Verify that Ops Manager successfully applied the stemcell. The stemcell version you imported and applied appears in the Staged column for Enterprise PKS.

  8. Select the Installation Dashboard link to return to the Installation Dashboard.

Verify Errand Configuration

To verify that errands are configured correctly in the Enterprise PKS tile, do the following:

  1. Click the newly-added Enterprise PKS tile.

  2. Click Errands.

  3. Under Post-Deploy Errands, configure Upgrade all clusters errand:

    • If you want to upgrade the Enterprise PKS tile and all your existing Kubernetes clusters simultaneously, verify that Upgrade all clusters errand is set to Default (On). The errand upgrades a single Kubernetes cluster at a time. Upgrading Enterprise PKS-deployed Kubernetes clusters can temporarily interrupt the service as described in Service Interruptions.
    • If you want to upgrade the Enterprise PKS tile only and then upgrade your existing Kubernetes clusters separately, disable Upgrade all clusters errand. For more information, see Upgrading Clusters.

      Warning: If you upgrade the Enterprise PKS tile or update your tile configuration, with Upgrade all clusters errand disabled, you must upgrade all your Kubernetes clusters before the next upgrade or configuration update.

  4. Under Post-Deploy Errands, set the Run smoke tests errand to On. The errand uses the Enterprise PKS Command Line Interface (PKS CLI) to create a Kubernetes cluster and then delete it. If the creation or deletion fails, the errand fails and the installation of the Enterprise PKS tile is aborted.

  5. Click Save.

  6. Review the other configuration panes. Click Save on any panes where you make changes.

    Note: When you upgrade Enterprise PKS, you must place singleton jobs in the AZ you selected when you first installed the Enterprise PKS tile. You cannot move singleton jobs to another AZ.

Apply Changes to the Enterprise PKS Tile

Perform the following steps to complete the upgrade of the Enterprise PKS tile:

  1. Return to the Installation Dashboard in Ops Manager.

  2. Click Review Pending Changes. For more information about this Ops Manager page, see Reviewing Pending Product Changes.

  3. Click Apply Changes.

  4. (Optional) To monitor the progress of the Upgrade all clusters errand using the BOSH CLI, do the following:

    1. Log in to the BOSH Director by running bosh -e MY-ENVIRONMENT log-in from a VM that can access your Enterprise PKS deployment. For more information, see Managing Enterprise PKS Deployments with BOSH.
    2. Run bosh -e MY-ENVIRONMENT tasks.
    3. Locate the task number for the errand in the # column of the BOSH output.
    4. Run bosh task TASK-NUMBER, replacing TASK-NUMBER with the task number you located in the previous step.

After the Upgrade

After you complete the upgrade to Enterprise PKS v1.5, complete the following verifications and upgrades:

Update PKS and Kubernetes CLIs

Update the PKS and Kubernetes CLIs on any local machine where you run commands that interact with your upgraded version of Enterprise PKS.

To update your CLIs, download and re-install the PKS and Kubernetes CLI distributions that are provided with Enterprise PKS on Pivotal Network.

For more information about installing the CLIs, see the following topics:

Verify the Upgrade

After you apply changes to the Enterprise PKS tile and the upgrade is complete, perform the following steps:

  1. Verify that your Kubernetes environment is healthy. To verify the health of your Kubernetes environment, see Verifying Deployment Health.

  2. Verify that the PKS control plane remains functional by performing the following steps:

    1. Add more workloads and create an additional cluster. For more information, see About Workload Upgrades in Maintaining Workload Uptime and Creating Clusters.
    2. Monitor the PKS control plane VM by clicking the Enterprise PKS tile, selecting the Status tab, and reviewing the Pivotal Container Service VM’s data points. If any data points are at capacity, scale your deployment accordingly.

Upgrade NSX-T Data Center

Warning: Each Kubernetes cluster contains a NSX-T Container Plug-in (NCP) which provides integration between NSX-T and Kubernetes. Before upgrading the clusters, the clusters will contains NCP version 2.4.x which is not compatible with NSX-T v2.5, hence, you cannot upgrade NSX-T v2.4.x to NSX-T v2.5 before upgrading all of your Kubernetes clusters. Upgrading all your clusters will result in the NCP component being upgraded to NCP v2.5 which will be compatible with both NSX-T 2.4.x and NSX-T v2.5.x. Once you complete the upgrade of all Kubernetes clusters managed by Enterprise PKS, you may upgrade to NSX-T v2.5. Any cluster that is not on upgraded before upgrading to NSX-T v2.5 will be orphaned and inoperable.

  1. You cannot upgrade to NSX-T v2.5.x until you have upgraded all Kubernetes clusters to NCP 2.5. The workflow is as follows:

    • Upgrade the PKS tile from 1.4.1 or later to 1.5.
    • Upgrade all Kubernetes clusters to NCP 2.5 (using the “Upgrade all clusters errand” or individually)
    • Upgrade to NSX-T 2.5.x (only after all Kubernetes clusters have been upgraded)
  2. Upgrade NSX-T from v2.4 to v2.5. See the Release Notes supported versions.

  3. Confirm that your vSphere v6.5 or v6.7 installation is on the supported version and patch for NSX-T v2.5.

  4. To perform the upgrade, refer to the NSX-T Data Center Upgrade Guide in the VMware documentation.

  5. If you made any architectural changes to your NSX-T environment that affect Enterprise PKS, such as adding a VIP address or load balancer for the NSX-T Management Cluster, update the PKS and BOSH tile with the new information, specifically:

  • In the BOSH Director tile, select the vCenter Configuration tab and update the following:

    • The NSX Address field
    • The NSX CA Cert field
    • Save the BOSH tile changes. Update BOSH with VIP and Cert
  • In the PKS tile, select the Networking tab and update the following:

    • The NSX Manager hostname field
    • The NSX Manager CA Cert field
    • Save the PKS tile changes Update Enterprise PKS with VIP and Cert
  1. If you made changes to the BOSH or PKS tiles:

    • Go to the Installation Dashboard in Ops Manager.
    • Click Review Pending Changes.
    • Expand the Errands list for Enterprise PKS.
    • Ensure that the Upgrade all clusters errand is selected.
    • Click Apply Changes. Upgrade all Kubernetes clusters
  2. Verify the Upgrade of NSX-T Data Center.

After you apply changes to the Enterprise PKS tile and the upgrade is complete, verify that your Kubernetes environment is healthy. To verify the health of your Kubernetes environment, see Verifying Deployment Health.


Please send any feedback you have to pks-feedback@pivotal.io.