Rotating Cluster Certificates

Page last updated:

This topic describes how to rotate certificates for Kubernetes clusters created by Tanzu Kubernetes Grid Integrated Edition (TKGI).

Overview of Rotating Cluster Certificates

Certificate authority (CA) certificates used by TKGI-created Kubernetes clusters and the leaf-level certificates that they issue include the following:

  • Cluster-specific certificates: Kubernetes cluster certificates that have values and expiration dates unique to each cluster.

    The following table lists the cluster-specific certificates and how to rotate them.

    Certificates When Used How to Rotate
    kubo_ca_2018, etcd_ca_2018, and their leaf certificates All clusters. See Rotate Cluster-Specific Certificates below.
    tls_nsx_t and tls_nsx_lb NSX-T only. These certificates must be registered with NSX Manager. See the Knowledge Base article How to rotate TKGi tls-nsx-t cluster certificate.

    Warning: Rotating NSX-T cluster certificates causes TKGI API downtime.


  • Shared cluster certificates: Certificates that are common to all TKGI-deployed Kubernetes clusters. Shared cluster certificates do not have unique values for each cluster. Some of them are used by the TKGI control plane for communication between the TKGI control plane and the clusters.

    To rotate shared cluster certificates, see Rotate Shared Cluster Certificates below.


For how to rotate certificates used only by the TKGI control plane, and not Kubernetes clusters, see Rotating TKGI Control Plane Certificates.

Rotate Cluster-Specific Certificates

This procedure rotates the following CA certificates and their leaf certificates:

  • kubo_ca_2018:
    • tls-metrics-server-2018
    • tls-kubelet-client-2018
    • tls-kubelet-2018
    • tls-kube-controller-manager-2018
  • etcd_ca_2018:
    • tls-etcdctl-root-2018-2
    • tls-etcdctl-flanneld-2018-2
    • tls-etcdctl-2018-2
    • tls-etcd-2018-2

All TKGI-deployed Kubernetes clusters use kubo_ca_2018, etcd_ca_2018, and their leaf certificates. Their values are unique to each cluster, and their expiration dates depend on the cluster creation date.

The procedure below uses the CredHub Maestro command line interface (CLI) to rotate the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates.

For more information about the CredHub Maestro CLI, see Getting Started with CredHub Maestro in the Ops Manager documentation.

Limitations:

  • In deployments that use NSX-T networking, clusters also have unique NSX-T certificates that must be registered with the NSX Manager. To rotate those, see Rotate Cluster-Specific NSX-T Certificates, below.
  • This procedure differs from the procedures to rotate shared cluster certificates or TKGI control plane certificates.

    Warning: Do not use the instructions in this section to rotate NSX-T certificates or shared Kubernetes cluster certificates.

Prerequisites

To rotate certificates using the CredHub Maestro CLI, you must have the following:

  • TKGI v1.9 or later. Earlier versions of TKGI are not compatible with the CredHub Maestro CLI.
  • Kubernetes clusters upgraded to TKGI v1.9.
  • Ops Manager v2.9 or later.
  • The pks.cluster.admin UAA scope.

The certificate rotation procedure with the CredHub Maestro CLI provided below has been tested on Ops Manager v2.9 and TKGI v1.9.

Downtime

Depending on cluster topology, rotating kubo_ca_2018 or etcd_ca_2018 may cause cluster downtime while cluster nodes restart:

  • Multiple control plane (master) and worker nodes: No downtime
  • Single control plane node: Cluster control plane downtime
  • Single worker node: Workload downtime

Prepare to Rotate

Before rotating your certificates, complete the following steps:

  1. Retrieve the Cluster UUID
  2. Access CredHub Maestro on the Ops Manager VM
  3. Identify Your Cluster Deployment Names
  4. Determine Which Certificates Are Expiring

Retrieve the Cluster UUID

This section describes how to retrieve the universally unique identifier (UUID) of a TKGI-provisioned Kubernetes cluster. You will use this UUID in Identify Your Cluster Deployment Names and Rotate the Certificates below.

To retrieve the UUID of a TKGI-provisioned Kubernetes cluster:

  1. Log in to TKGI. For instructions, see Logging in to Tanzu Kubernetes Grid Integrated Edition.

  2. To view the list of your deployed clusters, run tkgi clusters.

    For example:

    $ tkgi clusters

Name    Plan Name       UUID                                    Status       Action  
test    multi-master    ae681cd1-7ff4-4661-b12c-49a5b543f16f    succeeded    CREATE

  3. In the output of the tkgi clusters command, locate your target cluster and record its UUID. If you want to rotate the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates for multiple clusters, locate all your target clusters in the output and record the UUIDs.

  4. Proceed to Access CredHub Maestro on the Ops Manager VM below.

Access CredHub Maestro on the Ops Manager VM

To access the CredHub Maestro CLI on the Ops Manager VM:

  1. SSH into the Ops Manager VM. For instructions, see Log in to the Ops Manager VM with SSH in the Ops Manager documentation.

  2. Set the BOSH command line and CredHub environment variables on the Ops Manager VM.

    1. To set the BOSH environment variables, follow the instructions in Set the BOSH Environment Variables on the Ops Manager VM in the Ops Manager documentation.

      For example:

      $ export BOSH_CLIENT=ops_manager \
BOSH_CLIENT_SECRET=example_secret \
BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate \
BOSH_ENVIRONMENT=10.0.16.5 bosh

    2. To set the CredHub environment variables, export the following variables:

      • CREDHUB_SERVER is the URL of the BOSH Director CredHub server. This should be BOSH_ENVIRONMENT:8844.
      • CREDHUB_CLIENT is the name of the CredHub client. This is the same as BOSH_CLIENT.
      • CREDHUB_SECRET is the CredHub client secret. This is the same as BOSH_CLIENT_SECRET.
      • CREDHUB_CA_CERT is the path or value of the CredHub trusted CA certificate. This is the same as BOSH_CA_CERT.

      For example:

      $ export CREDHUB_SERVER=10.0.16.5:8844 \
CREDHUB_CLIENT=ops_manager \
CREDHUB_SECRET=example_secret \
CREDHUB_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate

Identify Your Cluster Deployment Names

  1. To identify your Kubernetes cluster deployment names, run:

    bosh deployments

    Kubernetes cluster deployment names begin with service-instance_. For example, service-instance_ae681cd1-7ff4-4661-b12c-49a5b543f16f, where ae681cd1-7ff4-4661-b12c-49a5b543f16f is the cluster UUID you retrieved in Retrieve the Cluster UUID above.

Determine Which Certificates Are Expiring

  1. To review the expiration dates of the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster:

    1. Run the following command:

      maestro list --expires-within TIME-PERIOD --deployment-name service-instance_CLUSTER-UUID

      Where:

      • TIME-PERIOD is the expiry window you want to filter. Valid units are d for days, w for weeks, m for months, and y for years. For example, 1y lists the certificates expiring within one year.
      • CLUSTER-UUID is the cluster UUID you retrieved in Retrieve the Cluster UUID above.

      For more information about how to check certificate expiration dates, see maestro list in the Ops Manager documentation.

    2. Locate the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates in the output of the maestro list command.

Rotate the Certificates

This section describes how to rotate the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster. To rotate these certificates for multiple clusters, repeat the procedure below for each cluster.

You can modify this procedure to rotate kubo_ca_2018 and etcd_ca_2018 separately, rather than at the same time.

To rotate the kubo_ca_2018 and etcd_ca_2018 CA certificates and their leaf certificates for a TKGI-provisioned Kubernetes cluster:

  1. Regenerate the kubo_ca_2018 and etcd_ca_2018 CA certificates:

    maestro regenerate ca --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
maestro regenerate ca --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018

    Where CLUSTER-UUID is the cluster UUID. You retrieved the UUID in Retrieve the Cluster UUID above.

    This step creates a new version of the CA certificates.

  2. If you are running Ops Manager v2.10 or later, skip to the next step. If you are running Ops Manager v2.9, mark the latest version of each CA certificate as transitional:

    maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
maestro update-transitional latest --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018

  3. Redeploy the cluster:

    1. Download the latest cluster deployment manifest:

      bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST

      Where PATH-TO-DEPLOYMENT-MANIFEST is the location where you want to save the cluster deployment manifest. For example, /tmp/manifest.yml.

    2. Deploy the cluster:

      bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST

  4. After the cluster redeployment completes successfully, mark the signing version of each CA certificate as transitional:

    maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
maestro update-transitional signing --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018

    This command also removes the transitional flag from the latest version of the CA certificates.

  5. Regenerate all leaf certificates signed by kubo_ca_2018 and etcd_ca_2018:

    maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
maestro regenerate leaf --signed-by /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018

  6. Redeploy the cluster:

    bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST
bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST

  7. After the cluster redeployment completes successfully, remove the transitional flag:

    maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/etcd_ca_2018
maestro update-transitional remove --name /p-bosh/service-instance_CLUSTER-UUID/kubo_ca_2018

  8. Redeploy the cluster:

    bosh -d service-instance_CLUSTER-UUID manifest > PATH-TO-DEPLOYMENT-MANIFEST
bosh -d service-instance_CLUSTER-UUID deploy PATH-TO-DEPLOYMENT-MANIFEST

Rotate Cluster-Specific NSX-T Certificates

To rotate the unique NSX-T certificates used by a TKGI cluster and register them with the NSX Manager, see the Knowledge Base article How to rotate TKGi tls-nsx-t cluster certificate.

Warning: Rotating NSX-T cluster certificates causes TKGI API downtime.

Rotate Shared Cluster Certificates

TKGI rotates shared cluster certificates automatically with selected tile upgrades. Most of these certificates have four- or five-year expiry periods, so users do not ordinarily need to rotate them.

After a shared certificate is rotated, all clusters must be updated. If the certificate is also used by the TKGI control plane, the TKGI control plane must be redeployed as well.

Certificate-specific notes:

  • kubo_odb_ca_2018: Shared by Kubernetes clusters and the TKGI control plane. If rotation is needed, contact Support.

Please send any feedback you have to pks-feedback@pivotal.io.