Rotate Kubernetes Cluster Certificates

Page last updated:

This topic describes how to rotate certificates for Kubernetes clusters created by Tanzu Kubernetes Grid Integrated Edition (TKGI).

Overview of TKGI-Provisioned Kubernetes Cluster Certificates

When TKGI provisions a Kubernetes cluster, the system generates certificate authority (CA) certificates and leaf certificates that have values and expiration dates unique to that cluster.


The following table lists the TKGI-provisioned Kubernetes cluster certificates and how to rotate them.

Certificates When Used How to Rotate
kubo_master_ca_2021, kubo_ca_2018, etcd_ca_2018, and their leaf certificates All clusters. See Rotate Kubernetes Cluster Certificates below.
tls_nsx_t and tls_nsx_lb NSX-T only. These certificates must be registered with NSX Manager. See Rotate NSX-T Certificates for Kubernetes Clusters.



To rotate the certificates used by the TKGI control plane, see Rotating TKGI Control Plane Certificates.

Warning: Certificate rotation is an advanced operation. If rotation is needed, it is recommended that you contact Support before proceeding.

Rotate Kubernetes Cluster Certificates

This procedure rotates the following CA certificates and their leaf certificates:

  • kubo_master_ca_2021

    • tls-kubernetes-2018
    • tls-ncp-2018 (with NSX-T)
    • tls-nsx-kube-proxy-2018 (with NSX-T)
  • kubo_ca_2018:

    • tls-metrics-server-2018
    • tls-kubelet-client-2018
    • tls-kubelet-2018
    • tls-kube-controller-manager-2018
  • etcd_ca_2018:

    • tls-etcdctl-root-2018-2
    • tls-etcdctl-flanneld-2018-2
    • tls-etcdctl-2018-2
    • tls-etcd-2018-2

All TKGI-deployed Kubernetes clusters use kubo_master_ca_2021, kubo_ca_2018, etcd_ca_2018, and their leaf certificates. Their values are unique to each cluster, and their expiration dates depend on the cluster creation date.

The procedure below uses the CredHub Maestro command line interface (CLI) to rotate the kubo_master_ca_2021, kubo_ca_2018, and etcd_ca_2018 CA certificates and their leaf certificates.

For more information about the CredHub Maestro CLI, see Getting Started with CredHub Maestro in the Ops Manager documentation.

Limitations:

  • In deployments that use NSX-T networking, clusters also have unique NSX-T certificates that must be registered with the NSX Manager. To rotate those, see Rotate Cluster-Specific NSX-T Certificates, below.
  • This procedure differs from the procedures to rotate shared cluster certificates or TKGI control plane certificates.

    Warning: Do not use the instructions in this section to rotate NSX-T certificates or shared Kubernetes cluster certificates.

Prerequisites

To rotate certificates using the CredHub Maestro CLI, you must have the following:

  • TKGI v1.9 or later. Earlier versions of TKGI are not compatible with the CredHub Maestro CLI.
  • Kubernetes clusters upgraded to TKGI v1.9.
  • Ops Manager v2.9 or later.
  • The pks.cluster.admin UAA scope.

The certificate rotation procedure with the CredHub Maestro CLI provided below has been tested on Ops Manager v2.9 and TKGI v1.9.

Downtime

Depending on cluster topology, rotating kubo_master_ca_2021, kubo_ca_2018, or etcd_ca_2018 may cause cluster downtime while cluster nodes restart:

  • Multiple control plane (master) and worker nodes: No downtime
  • Single control plane node: Cluster control plane downtime
  • Single worker node: Workload downtime

Prepare to Rotate

Before rotating your certificates, complete the following steps:

  1. Retrieve the Cluster UUID
  2. Access CredHub Maestro on the Ops Manager VM
  3. Identify Your Cluster Deployment Names
  4. Determine Which Certificates Are Expiring

Retrieve the Cluster UUID

This section describes how to retrieve the universally unique identifier (UUID) of a TKGI-provisioned Kubernetes cluster. You will use this UUID in Identify Your Cluster Deployment Names and Rotate the Certificates below.

To retrieve the UUID of a TKGI-provisioned Kubernetes cluster:

  1. Log in to TKGI. For instructions, see Logging in to Tanzu Kubernetes Grid Integrated Edition.
  2. To view the list of your deployed clusters, run tkgi clusters.

    For example:

    $ tkgi clusters
    
    Name    Plan Name       UUID                                    Status       Action  
    test    multi-master    ae681cd1-7ff4-4661-b12c-49a5b543f16f    succeeded    CREATE
    
  3. In the output of the tkgi clusters command, locate your target cluster and record its UUID. If you want to rotate the kubo_master_ca_2021, kubo_ca_2018, and etcd_ca_2018 CA certificates and their leaf certificates for multiple clusters, locate all your target clusters in the output and record the UUIDs.

  4. Proceed to Access CredHub Maestro on the Ops Manager VM below.

Access CredHub Maestro on the Ops Manager VM

To access the CredHub Maestro CLI on the Ops Manager VM:

  1. SSH into the Ops Manager VM. For instructions, see Log in to the Ops Manager VM with SSH in the Ops Manager documentation.
  2. Set the BOSH command line and CredHub environment variables on the Ops Manager VM.

    1. To set the BOSH environment variables, follow the instructions in Set the BOSH Environment Variables on the Ops Manager VM in the Ops Manager documentation.

      For example:

      $ export BOSH_CLIENT=ops_manager \
      BOSH_CLIENT_SECRET=example_secret \
      BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate \
      BOSH_ENVIRONMENT=10.0.16.5 bosh
      
    2. To set the CredHub environment variables, export the following variables:

      • CREDHUB_SERVER is the URL of the BOSH Director CredHub server. This should be BOSH_ENVIRONMENT:8844.
      • CREDHUB_CLIENT is the name of the CredHub client. This is the same as BOSH_CLIENT.
      • CREDHUB_SECRET is the CredHub client secret. This is the same as BOSH_CLIENT_SECRET.
      • CREDHUB_CA_CERT is the path or value of the CredHub trusted CA certificate. This is the same as BOSH_CA_CERT.

      For example:

      $ export CREDHUB_SERVER=10.0.16.5:8844 \
      CREDHUB_CLIENT=ops_manager \
      CREDHUB_SECRET=example_secret \
      CREDHUB_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
      

Identify Your Cluster Deployment Names

  1. To identify your Kubernetes cluster deployment names, run:

    bosh deployments
    

    Kubernetes cluster deployment names begin with service-instance_. For example, service-instance_ae681cd1-7ff4-4661-b12c-49a5b543f16f, where ae681cd1-7ff4-4661-b12c-49a5b543f16f is the cluster UUID you retrieved in Retrieve the Cluster UUID above.

Determine Which Certificates Are Expiring

  1. To review the expiration dates of the kubo_master_ca_2021, kubo_ca_2018, and etcd_ca_2018 CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster:

    1. Run the following command:

      maestro list --expires-within TIME-PERIOD --deployment-name service-instance_CLUSTER-UUID
      

      Where:

      • TIME-PERIOD is the expiry window you want to filter. Valid units are d for days, w for weeks, m for months, and y for years. For example, 1y lists the certificates expiring within one year.
      • CLUSTER-UUID is the cluster UUID you retrieved in Retrieve the Cluster UUID above.

      For more information about how to check certificate expiration dates, see maestro list in the Ops Manager documentation.

    2. Locate the kubo_master_ca_2021, kubo_ca_2018, and etcd_ca_2018 CA certificates and their leaf certificates in the output of the maestro list command.

Rotate the Certificates

This section describes how to rotate the kubo_master_ca_2021, kubo_ca_2018, and etcd_ca_2018 CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster. To rotate these certificates for multiple clusters, repeat the procedure below for each cluster.

You can modify this procedure to rotate kubo_ca_2018 and etcd_ca_2018 separately, rather than at the same time.

To rotate the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster:

  1. Regenerate the kubo_master_ca_2021, kubo_ca_2018 and etcd_ca_2018 CA certificates:

    maestro regenerate ca --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
    maestro regenerate ca --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
    

    Where CLUSTER-UUID is the cluster UUID. You retrieved the UUID in Retrieve the Cluster UUID above.

    This step creates a new version of the CA certificates.

  2. If you are running Ops Manager v2.10 or later, skip to the next step. If you are running Ops Manager v2.9, mark the latest version of each CA certificate as transitional:

    maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/kubo_master_ca_2021
    maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
    maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
    
  3. Redeploy the cluster:

    1. Download the latest cluster deployment manifest:

      bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST
      

      Where PATH-TO-DEPLOYMENT-MANIFEST is the location where you want to save the cluster deployment manifest. For example, /tmp/manifest.yml.

    2. Deploy the cluster:

      bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST
      
  4. After the cluster redeployment completes successfully, mark the signing version of each CA certificate as transitional:

    maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/kubo_master_ca_2021    
    maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
    maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
    

    This command also removes the transitional flag from the latest version of the CA certificates.

  5. Regenerate all leaf certificates signed by kubo_master_ca_2021, kubo_ca_2018, and etcd_ca_2018:

    maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/kubo_master_ca_2021    
    maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
    maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
    
  6. Redeploy the cluster:

    bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST
    bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST
    
  7. After the cluster redeployment completes successfully, remove the transitional flag:

    maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/kubo_master_ca_2021    
    maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
    maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018
    
  8. Redeploy the cluster:

    bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST
    bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST
    

Please send any feedback you have to pks-feedback@pivotal.io.