Rotating Cluster Certificates
Page last updated:
This topic describes how to rotate certificates for Kubernetes clusters created by Tanzu Kubernetes Grid Integrated Edition (TKGI).
Overview of Rotating Cluster Certificates
Certificate authority (CA) certificates used by TKGI-created Kubernetes clusters and the leaf-level certificates that they issue include the following:
Cluster-specific certificates: Kubernetes cluster certificates that have values and expiration dates unique to each cluster.
The following table lists the cluster-specific certificates and how to rotate them.Certificates When Used How to Rotate kubo_ca_2018
,etcd_ca_2018
, and their leaf certificatesAll clusters. See Rotate Cluster-Specific Certificates below. tls_nsx_t
andtls_nsx_lb
NSX-T only. These certificates must be registered with NSX Manager. See the Knowledge Base article How to rotate TKGi tls-nsx-t cluster certificate. Warning: Rotating NSX-T cluster certificates causes TKGI API downtime.
Shared cluster certificates: Certificates that are common to all TKGI-deployed Kubernetes clusters. Shared cluster certificates do not have unique values for each cluster. Some of them are used by the TKGI control plane for communication between the TKGI control plane and the clusters.
To rotate shared cluster certificates, see Rotate Shared Cluster Certificates below.
For how to rotate certificates used only by the TKGI control plane, and not Kubernetes clusters,
see Rotating TKGI Control Plane Certificates.
Rotate Cluster-Specific Certificates
This procedure rotates the following CA certificates and their leaf certificates:
-
kubo_ca_2018
:tls-metrics-server-2018
tls-kubelet-client-2018
tls-kubelet-2018
tls-kube-controller-manager-2018
etcd_ca_2018
:tls-etcdctl-root-2018-2
tls-etcdctl-flanneld-2018-2
tls-etcdctl-2018-2
tls-etcd-2018-2
All TKGI-deployed Kubernetes clusters use kubo_ca_2018
, etcd_ca_2018
, and their leaf certificates.
Their values are unique to each cluster,
and their expiration dates depend on the cluster creation date.
The procedure below uses the CredHub Maestro command line interface (CLI) to rotate the
kubo_ca_2018
and etcd_ca_2018
CA certificates and their leaf certificates.
For more information about the CredHub Maestro CLI, see Getting Started with CredHub Maestro in the Ops Manager documentation.
Limitations:
- In deployments that use NSX-T networking, clusters also have unique NSX-T certificates that must be registered with the NSX Manager. To rotate those, see Rotate Cluster-Specific NSX-T Certificates, below.
- This procedure differs from the procedures to rotate shared cluster certificates
or TKGI control plane certificates.
Warning: Do not use the instructions in this section to rotate NSX-T certificates or shared Kubernetes cluster certificates.
Prerequisites
To rotate certificates using the CredHub Maestro CLI, you must have the following:
- TKGI v1.9 or later. Earlier versions of TKGI are not compatible with the CredHub Maestro CLI.
- Kubernetes clusters upgraded to TKGI v1.9.
- Ops Manager v2.9 or later.
- The
pks.cluster.admin
UAA scope.
The certificate rotation procedure with the CredHub Maestro CLI provided below has been tested on Ops Manager v2.9 and TKGI v1.9.
Downtime
Depending on cluster topology, rotating kubo_ca_2018
or etcd_ca_2018
may cause cluster downtime while cluster nodes restart:
- Multiple control plane (master) and worker nodes: No downtime
- Single control plane node: Cluster control plane downtime
- Single worker node: Workload downtime
Prepare to Rotate
Before rotating your certificates, complete the following steps:
- Retrieve the Cluster UUID
- Access CredHub Maestro on the Ops Manager VM
- Identify Your Cluster Deployment Names
- Determine Which Certificates Are Expiring
Retrieve the Cluster UUID
This section describes how to retrieve the universally unique identifier (UUID) of a TKGI-provisioned Kubernetes cluster. You will use this UUID in Identify Your Cluster Deployment Names and Rotate the Certificates below.
To retrieve the UUID of a TKGI-provisioned Kubernetes cluster:
- Log in to TKGI. For instructions, see Logging in to Tanzu Kubernetes Grid Integrated Edition.
To view the list of your deployed clusters, run
tkgi clusters
.For example:
$ tkgi clusters Name Plan Name UUID Status Action test multi-master ae681cd1-7ff4-4661-b12c-49a5b543f16f succeeded CREATE
In the output of the
tkgi clusters
command, locate your target cluster and record its UUID. If you want to rotate thekubo_ca_2018
andetcd_ca_2018
CA certificates and their leaf certificates for multiple clusters, locate all your target clusters in the output and record the UUIDs.Proceed to Access CredHub Maestro on the Ops Manager VM below.
Access CredHub Maestro on the Ops Manager VM
To access the CredHub Maestro CLI on the Ops Manager VM:
- SSH into the Ops Manager VM. For instructions, see Log in to the Ops Manager VM with SSH in the Ops Manager documentation.
Set the BOSH command line and CredHub environment variables on the Ops Manager VM.
To set the BOSH environment variables, follow the instructions in Set the BOSH Environment Variables on the Ops Manager VM in the Ops Manager documentation.
For example:
$ export BOSH_CLIENT=ops_manager \ BOSH_CLIENT_SECRET=example_secret \ BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate \ BOSH_ENVIRONMENT=10.0.16.5 bosh
To set the CredHub environment variables, export the following variables:
CREDHUB_SERVER
is the URL of the BOSH Director CredHub server. This should beBOSH_ENVIRONMENT:8844
.CREDHUB_CLIENT
is the name of the CredHub client. This is the same asBOSH_CLIENT
.CREDHUB_SECRET
is the CredHub client secret. This is the same asBOSH_CLIENT_SECRET
.CREDHUB_CA_CERT
is the path or value of the CredHub trusted CA certificate. This is the same asBOSH_CA_CERT
.
For example:
$ export CREDHUB_SERVER=10.0.16.5:8844 \ CREDHUB_CLIENT=ops_manager \ CREDHUB_SECRET=example_secret \ CREDHUB_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
Identify Your Cluster Deployment Names
To identify your Kubernetes cluster deployment names, run:
bosh deployments
Kubernetes cluster deployment names begin with
service-instance_
. For example,service-instance_ae681cd1-7ff4-4661-b12c-49a5b543f16f
, whereae681cd1-7ff4-4661-b12c-49a5b543f16f
is the cluster UUID you retrieved in Retrieve the Cluster UUID above.
Determine Which Certificates Are Expiring
To review the expiration dates of the
kubo_ca_2018
andetcd_ca_2018
CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster:Run the following command:
maestro list --expires-within TIME-PERIOD --deployment-name service-instance_CLUSTER-UUID
Where:
TIME-PERIOD
is the expiry window you want to filter. Valid units ared
for days,w
for weeks,m
for months, andy
for years. For example,1y
lists the certificates expiring within one year.CLUSTER-UUID
is the cluster UUID you retrieved in Retrieve the Cluster UUID above.
For more information about how to check certificate expiration dates, see maestro list in the Ops Manager documentation.
Locate the
kubo_ca_2018
andetcd_ca_2018
CA certificates and their leaf certificates in the output of themaestro list
command.
Rotate the Certificates
This section describes how to rotate the
kubo_ca_2018
and etcd_ca_2018
CA certificates and their leaf certificates
for a TKGI-provisioned Kubernetes cluster.
To rotate these certificates for multiple clusters,
repeat the procedure below for each cluster.
You can modify this procedure to rotate kubo_ca_2018
and etcd_ca_2018
separately, rather than at the same time.
To rotate the kubo_ca_2018
and etcd_ca_2018
CA certificates and their leaf
certificates for a TKGI-provisioned
Kubernetes cluster:
Regenerate the
kubo_ca_2018
andetcd_ca_2018
CA certificates:maestro regenerate ca --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018 maestro regenerate ca --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
Where
CLUSTER-UUID
is the cluster UUID. You retrieved the UUID in Retrieve the Cluster UUID above.
This step creates a new version of the CA certificates.If you are running Ops Manager v2.10 or later, skip to the next step. If you are running Ops Manager v2.9, mark the latest version of each CA certificate as transitional:
maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018 maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
Redeploy the cluster:
Download the latest cluster deployment manifest:
bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST
Where
PATH-TO-DEPLOYMENT-MANIFEST
is the location where you want to save the cluster deployment manifest. For example,/tmp/manifest.yml
.Deploy the cluster:
bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST
After the cluster redeployment completes successfully, mark the signing version of each CA certificate as transitional:
maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018 maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
This command also removes the transitional flag from the latest version of the CA certificates.
Regenerate all leaf certificates signed by
kubo_ca_2018
andetcd_ca_2018
:maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018 maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
Redeploy the cluster:
bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST
After the cluster redeployment completes successfully, remove the transitional flag:
maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018 maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
Redeploy the cluster:
bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST
Rotate Cluster-Specific NSX-T Certificates
To rotate the unique NSX-T certificates used by a TKGI cluster and register them with the NSX Manager, see the Knowledge Base article How to rotate TKGi tls-nsx-t cluster certificate.
Warning: Rotating NSX-T cluster certificates causes TKGI API downtime.
Rotate Shared Cluster Certificates
TKGI rotates shared cluster certificates automatically with selected tile upgrades. Most of these certificates have four- or five-year expiry periods, so users do not ordinarily need to rotate them.
After a shared certificate is rotated, all clusters must be updated. If the certificate is also used by the TKGI control plane, the TKGI control plane must be redeployed as well.
Certificate-specific notes:
kubo_odb_ca_2018
: Shared by Kubernetes clusters and the TKGI control plane. If rotation is needed, contact Support.
Please send any feedback you have to pks-feedback@pivotal.io.