Upgrading Clusters
Page last updated:
This topic describes how to upgrade Kubernetes clusters provisioned by VMware Enterprise PKS (Enterprise PKS) through the Enterprise PKS Command Line Interface (PKS CLI).
For information about how to upgrade PKS-provisioned clusters
through the Enterprise PKS tile,
see Verify Errand Configuration in one of the following topics:
For conceptual information about Enterprise PKS upgrades, see About Enterprise PKS Upgrades.
Overview
Upgrading a PKS-provisioned Kubernetes cluster updates the Enterprise PKS version and the Kubernetes version of the cluster. PKS-provisioned Kubernetes clusters upgrade when:
- You upgrade Enterprise PKS with the Upgrade all clusters errand enabled in the Enterprise PKS tile > Errands.
- You run
pks upgrade-cluster
orpks upgrade-clusters
as described in Upgrade Clusters below.
For example, running pks upgrade-cluster
upgrades the cluster you specify to your current version of Enterprise PKS
and to the version of Kubernetes
that is included with your current version of Enterprise PKS.
WARNING: Do not change the number of master/etcd nodes for any plan that was used to create currently-running clusters. Enterprise PKS does not support changing the number of master/etcd nodes for plans with existing clusters.
Prerequisites
Before upgrading PKS-provisioned Kubernetes clusters:
- Verify the cluster you are upgrading supports upgrading. For information, see Verify Your Clusters Support Upgrading in the Upgrade Preparation Checklist for Enterprise PKS.
- Verify that your Kubernetes environment is healthy. For information, see Verifying Deployment Health.
- Install the PKS CLI. For information, see Installing the PKS CLI.
- Log in to Enterprise PKS using
pks login
. For more information, see Logging in to Enterprise PKS.
Upgrade Clusters
You can upgrade individual or multiple clusters using the following procedures:
To monitor or stop a cluster upgrade, follow the procedures in Manage Your Kubernetes Cluster Upgrade Job below.
Upgrade a Single Cluster
The Enterprise PKS CLI provides upgrade-cluster
for upgrading an individual Enterprise PKS-provisioned
Kubernetes cluster.
To upgrade an individual Kubernetes cluster, run the following command:
pks upgrade-cluster CLUSTER-NAME
Where CLUSTER-NAME
is the name of the Kubernetes cluster you want to upgrade.
To upgrade multiple clusters, see Upgrade Multiple Kubernetes Clusters below.
For more information about the pks upgrade-cluster
command, see pks upgrade-cluster in the PKS CLI documentation.
Upgrade Multiple Clusters
The Enterprise PKS CLI provides upgrade-clusters
for upgrading multiple Enterprise PKS-provisioned
Kubernetes clusters.
You can upgrade clusters serially, serially with some clusters designated as canary clusters, or entirely in parallel.
To upgrade a single cluster, see Upgrade a Single Kubernetes Cluster above.
For more information about the pks upgrade-clusters
command, see pks upgrade-clusters in the PKS CLI documentation.
Upgrade Clusters in Parallel
To upgrade multiple Kubernetes clusters, run the following command:
pks upgrade-clusters --clusters CLUSTER-NAMES --max-in-flight CLUSTER-COUNT --wait
Where:
CLUSTER-NAMES
is a comma-delimited list of the names of the Kubernetes clusters you want to upgrade.CLUSTER-COUNT
is the maximum number of clusters to upgrade in parallel within an an AZ.- If the
CLUSTER-NAMES
list is longer than theCLUSTER-COUNT
, the first set of clusters are upgraded in parallel and subsequent clusters are queued and then upgraded in parallel as the preceding cluster upgrades complete. - If an upgrade fails for a cluster in the
CLUSTER-NAMES
list, the job continues to a subsequent cluster in the list. - To run the cluster upgrade job as a background task, remove the
--wait
argument.
Note: Run
upgrade-clusters
with a--max-in-flight
argument less than your BOSH Director > Director Config > Director Workers value. For example, if your Director Workers value remains the default of5
, runupgrade-clusters
with a--max-in-flight
argument value less than5
.Note:
max-in-flight
is an optional argument. Ifmax-in-flight
is not set, Enterprise PKS uses the defaultmax-in-flight
value of1
and the clusters are upgraded serially.- If the
For example:
$ pks upgrade-clusters --clusters k8-cluster-000,k8-cluster-001,k8-cluster-002 --max-in-flight 2 --wait
You are about to upgrade k8-cluster-000, k8-cluster-001 and k8-cluster-002. Warning: This operation may be long running and may block further operations on the cluster(s) until complete
Continue? (y/n):y Your taskID for the upgrade task is: d772aba0-2670-4fba-b26c-044b19d6ab60 Started upgrading cluster: k8-cluster-000 Started upgrading cluster: k8-cluster-001 Finished upgrading cluster: k8-cluster-000 Started upgrading cluster: k8-cluster-002 Finished upgrading cluster: k8-cluster-001 Finished upgrading cluster: k8-cluster-002 Upgrade task d772aba0-2670-4fba-b26c-044b19d6ab60 is done.
Upgrade Clusters With Canaries
To upgrade multiple clusters and automatically stop upgrading clusters if a cluster upgrade fails, specify your cluster list as canary clusters. You can specify one or more clusters as canary clusters.
To upgrade multiple clusters with one or more canary clusters, run the following command:
pks upgrade-clusters --canaries CANARY-CLUSTER-NAMES --clusters CLUSTER-NAMES --wait
Where:
CANARY-CLUSTER-NAMES
is a comma-delimited list of the names of the Kubernetes clusters you want to upgrade as canary clusters.CLUSTER-NAMES
is a comma-delimited list of Kubernetes clusters to upgrade if all canary clusters successfully upgrade.- The specified canary clusters are upgraded prior to upgrading the clusters in your
CLUSTER-NAMES
list. - Canary clusters are always upgraded serially.
- If an upgrade fails for a canary cluster, the entire
upgrade-clusters
job stops. - If an upgrade fails for a cluster in the
CLUSTER-NAMES
list, theupgrade-clusters
job continues to a subsequent cluster in the list. - To run the cluster upgrade job as a background task, remove the
--wait
argument.
Note:
--clusters
is a required argument. To configureupgrade-clusters
to stop for any cluster upgrade failure, specify only one cluster in yourCLUSTER-NAMES
list and the remaining clusters in yourCANARY-CLUSTER-NAMES
list.Note: Canary clusters are always upgraded serially. Only the clusters specified in the
--clusters
list are upgraded in parallel when you runupgrade-clusters
with both--canaries
and--max-in-flight
arguments.- The specified canary clusters are upgraded prior to upgrading the clusters in your
For example:
$ pks upgrade-clusters --canaries k8-cluster-dev,k8-cluster-000,k8-cluster-001 --clusters k8-cluster-002 --wait
You are about to upgrade k8-cluster-dev k8-cluster-000, k8-cluster-001 and k8-cluster-002. Warning: This operation may be long running and may block further operations on the cluster(s) until complete Continue? (y/n):y Your taskID for the upgrade task is: ce31a1bb-380a-453f-afa0-835ffa1ce6ac Started upgrading cluster: k8-cluster-000 Upgrading cluster succeeded: k8-cluster-000 Started upgrading cluster: k8-cluster-001 Upgrading cluster succeeded: k8-cluster-001 Started upgrading cluster: k8-cluster-dev Upgrading cluster failed: k8-cluster-dev Upgrade task ce31a1bb-380a-453f-afa0-835ffa1ce6ac is done.
Manage Your Cluster Upgrade Job
You can use the PKS CLI to monitor and manage your Enterprise PKS-provisioned Kubernetes cluster upgrade jobs.
Monitor Your Clusters
To review the status of the actions being performed on your clusters, run the following command:
pks clusters
For example:
$ pks clusters
Upgrade is available to PKS Version: 1.7.0-build.16
PKS Version Name k8s Version Plan Name UUID Status Action 1.7.0-build.16 k8-cluster-000 1.16.7 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 succeeded UPGRADE 1.7.0-build.16 k8-cluster-001 1.16.7 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 failed UPGRADE 1.7.0-build.16 k8-cluster-002 1.16.7 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 in progress UPGRADE 1.7.0-build.16 k8-cluster-003 1.16.7 small 9527ebaa-e2fa-422f-a52b-de3c3f0e39a4 queued UPGRADE
Monitor Your Cluster Upgrade Job
To review the status of your upgrade-clusters
job, run the following command:
pks task TASKID
Where TASKID
is the ID of the task that was returned when you ran pks upgrade-clusters
.
For example:
$ pks task ce31a1bb-380a-453f-afa0-835ffa1ce6ac
Your upgrade task is: done
Name Status Start time End time isCanary k8-cluster-000 succeeded Mon, 14 Oct 2019 12:00:00 PDT Mon, 14 Oct 2019 12:19:54 PDT true k8-cluster-001 failed Mon, 14 Oct 2019 12:20:00 PDT --- true
Stop Your Cluster Upgrade Job
To cancel a running upgrade-clusters
job, run the following PKS CLI command:
pks cancel-task TASKID
Where TASKID
is the ID of the task that was returned when you ran pks upgrade-clusters
.
Warning: pks cancel-task
does not cancel cluster upgrades currently in progress. This command only cancels a job’s pending cluster upgrades.
After Upgrading Clusters
(Optional) Restore Cluster Sizing
If you scaled your cluster up for the upgrade and you prefer to restore your cluster to its original sizing, you can now scale the cluster back down to its previous configuration. VMware recommends that you not scale down your clusters and continue to run them with recommended configurations, reducing the chance of a future outage.
Please send any feedback you have to pks-feedback@pivotal.io.