About Enterprise PKS Upgrades

Page last updated:

This topic provides conceptual information about Enterprise PKS upgrades, including upgrading the PKS control plane and PKS-provisioned Kubernetes clusters.

For step-by-step instructions on upgrading Enterprise PKS and PKS-provisioned Kubernetes clusters, see:

• Upgrading Enterprise PKS (Flannel Networking)
• Upgrading Enterprise PKS (NSX-T Networking)
• Upgrading Clusters

Overview

An Enterprise PKS upgrade modifies the version of Enterprise PKS, for example, from v1.6.x to v1.7.0 or from v1.7.0 to v1.7.1.

By default, Enterprise PKS is set to perform a full upgrade, which upgrades both the PKS control plane and all PKS-provisioned Kubernetes clusters.

However, you can choose to upgrade Enterprise PKS in two phases by upgrading the PKS control plane first and then upgrading your PKS-provisioned Kubernetes clusters later.

Both the full upgrade and the PKS control plane upgrade are performed through the Enterprise PKS tile only. When upgrading PKS-provisioned Kubernetes clusters, you can use either the Enterprise PKS tile or the PKS CLI. See the table below.

Typically, if you choose to upgrade PKS-provisioned Kubernetes clusters only, you will upgrade them through the PKS CLI.

Deciding Between Full and Two-Phase Upgrade

When deciding whether to perform the default full upgrade or to upgrade the PKS control plane and PKS-provisioned Kubernetes clusters separately, consider your organization needs.

For example, if your organization runs PKS-provisioned Kubernetes clusters in both development and production environments and you want to upgrade only one environment first, you can achieve your goal by upgrading the PKS control plane and PKS-provisioned Kubernetes separately instead of performing a full upgrade.

Examples of other advantages of upgrading Enterprise PKS in two phases include:

• Faster Enterprise PKS tile upgrades. If you have a large number of clusters in your Enterprise PKS deployment, performing a full upgrade can significantly increase the amount of time required to upgrade the Enterprise PKS tile.

• More granular control over cluster upgrades. In addition to enabling you to upgrade subsets of clusters, the PKS CLI supports upgrading each cluster individually.

• Not a monolithic upgrade. This helps isolate the root cause of an error when troubleshooting upgrades. For example, when a cluster-related upgrade error occurs during a full upgrade, the entire Enterprise PKS tile upgrade may fail.

Warning: If you disable the default full upgrade and upgrade only the PKS control plane, you must upgrade all your PKS-provisioned Kubernetes clusters before the next Enterprise PKS tile upgrade. Disabling the default full upgrade and upgrading only the PKS control plane cause the PKS version tagged in your Kubernetes clusters to fall behind the Enterprise PKS tile version. If your PKS-provisioned Kubernetes clusters fall more than one version behind the tile, Enterprise PKS cannot upgrade the clusters.

What Happens During Full PKS and PKS Control Plane Upgrades

You can perform full PKS upgrades and PKS control plane upgrades only through the Enterprise PKS tile.

After you add a new Enterprise PKS tile version to your staging area on the Ops Manager Installation Dashboard, Ops Manager automatically migrates your configuration settings into the new tile version.

For more information, see:

• Full PKS Upgrades
• PKS Control Plane Upgrades

Full PKS Upgrades

During a full PKS upgrade, the Enterprise PKS tile does the following:

1. (Only in v1.7) Creates the PKS Database VM. When you first upgrade from Enterprise PKS v1.6 to v1.7, the upgrade process creates the PKS Database VM, a new dedicated MySQL VM.

   • This MySQL VM resides alongside the PKS API VM on the PKS control plane.
   • The upgrade process then migrates the PKS v1.6 MySQL data from the PKS API VM to the new dedicated MySQL VM.
   • Subsequent Enterprise PKS upgrades, from earlier to later patch versions of PKS v1.7, do not include this step.

2. Upgrades the PKS control plane, which hosts the PKS API and UAA servers. This control plane upgrade causes temporary outages as described in Control Plane Outages below.

3. Upgrades PKS-provisioned Kubernetes clusters.

   • Upgrading PKS-provisioned Kubernetes clusters is controlled by the Upgrade all clusters errand in the Enterprise PKS tile.
   • The cluster upgrade process recreates all clusters, which may cause cluster outages. For more information, see What Happens During Cluster Upgrades below.

PKS Control Plane Upgrades

When upgrading the PKS control plane only, the Enterprise PKS tile follows the process described in Full PKS Upgrades above, step 1 and 2. It does not upgrade PKS-provisioned Kubernetes clusters, step 3.

Control Plane Outages

Upgrading the Enterprise PKS control plane temporarily interrupts the following:

• Logging in to the PKS CLI and using all pks commands
• Using the PKS API to retrieve information about clusters
• Using the PKS API to create and delete clusters
• Using the PKS API to resize clusters

These outages do not affect the Kubernetes clusters themselves. During a PKS control plane upgrade, you can still interact with clusters and their workloads using the Kubernetes Command Line Interface, kubectl.

For more information about the PKS control plane, see PKS Control Plane Overview in Enterprise PKS Architecture.

Canary Instances

The Enterprise PKS tile is a BOSH deployment.

BOSH-deployed products can set a number of canary instances to upgrade first, before the rest of the deployment VMs. BOSH continues the upgrade only if the canary instance upgrade succeeds. If the canary instance encounters an error, the upgrade stops running and other VMs are not affected.

The Enterprise PKS tile uses one canary instance when deploying or upgrading Enterprise PKS.

What Happens During Cluster Upgrades

Upgrading PKS-provisioned Kubernetes clusters updates their Kubernetes version to the version included with the Enterprise PKS tile. It also updates the PKS version tagged in your clusters to the Enterprise PKS tile version.

You can upgrade PKS-provisioned Kubernetes clusters either through the Enterprise PKS tile or the PKS CLI. See the table below.

During an upgrade of PKS-provisioned clusters, Enterprise PKS recreates your clusters. This includes the following stages for each cluster you upgrade:

1. Master nodes are recreated.
2. Worker nodes are recreated.

Depending on your cluster configuration, these recreations may cause Master Nodes Outage or Worker Nodes Outage as described below.

Master Nodes Outage

When Enterprise PKS upgrades a single-master cluster, you cannot interact with your cluster, use kubectl, or push new workloads.

To avoid this loss of functionality, VMware recommends using multi-master clusters.

Worker Nodes Outage

When Enterprise PKS upgrades a worker node, the node stops running containers. If your workloads run on a single node, they will experience downtime.

To avoid downtime for stateless workloads, VMware recommends using at least one worker node per availability zone (AZ). For stateful workloads, VMware recommends using a minimum of two worker nodes per AZ.

Note: When the Upgrade all clusters errand is enabled in the Enterprise PKS tile, updating the tile with a new Linux or Windows stemcell rolls every Linux or Windows VM in each Kubernetes cluster. This automatic rolling ensures that all your VMs are patched. To avoid workload downtime, use the resource configuration recommended in Master Nodes Outage and Worker Nodes Outage above and in Maintaining Workload Uptime.