Advanced Certificate Rotation with CredHub Maestro

Page last updated:

This topic describes how to rotate certificates and certificate authorities (CAs) in BOSH CredHub using CredHub Maestro. The topic only covers rotating CredHub-managed certificates.

For more information about setting up and using CredHub Maestro, see Getting Started with CredHub Maestro.

When to Use CredHub Maestro for Certificate Rotation

VMware recommends that you use the Ops Manager API to perform bulk rotation of certificates. For more information, see Overview of Certificate Rotation.

However, there are some cases when you cannot use the Ops Manager API to do bulk certificate rotation. In these cases, you can use CredHub Maestro to rotate certificates.

The following describes when to use CredHub Maestro instead of the Ops Manager API for certificate rotation:

  • You have VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) installed and therefore cannot use the Ops Manager API to rotate CredHub-managed certificates. In TKGI v1.9 and later, these include the cluster-specfic kubo_ca_2018 and etcd_ca_2018 CAs and their leaf certificates.

    Warning: In TKGI v1.8 and earlier, you cannot use CredHub Maestro to rotate TKGI certificates. To rotate these certificates, contact Support.

  • You need to rotate one or more specific CredHub-managed certificates and do not want to rotate any other certificates managed by Ops Manager or CredHub.

Prerequisites

Before you rotate CredHub-managed certificates with CredHub Maestro:

  • Before rotating certificates in Ops Manager v2.10, you must reset any certificates that were manually set in CredHub on Ops Manager v2.6 and earlier to prevent them from being rotated with other certificates generated by CredHub. To find and reset any manually set certificates in CredHub, follow the procedure in Reviewing Manually Set Certificates in CredHub. For more information, see Reset Manually Set Certificates in CredHub Before Rotating Certificates in Ops Manager v2.10 Release Notes.

  • To ensure that there is no app availability downtime during a rotation, VMware recommends that you scale up all instance groups and use an external blobstore.

  • Check which tiles in your installation support certificate rotation with CredHub Maestro. Not every tile currently supports certificate rotation, so there is a potential for downtime. For more information about which tiles support certificate rotation, see CredHub Maestro Tile Compatibility.

Check Expiration Dates and Certificate Types

This section describes how to manually check the expiration dates of the CAs and leaf certificates managed by CredHub.

After identifying the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

To check certificate expiration dates and types:

  1. Retrieve a list of certificates expiring within a given timeframe by running:

    maestro list --expires-within TIME-PERIOD
    

    Where TIME-PERIOD is the unit for which you want to check the expiry window. Valid units are d for days, w for weeks, m for months, and y for years.

  2. After you identify the list of certificates that expire soon, follow the procedure in Rotate Certificates.

Rotate Certificates

This section explains how to rotate leaf certificates and CAs stored in CredHub. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated.

To rotate certificates, follow one of the following procedures:

Note: All CredHub Maestro rotation commands have a dry-run flag that can be used to preview the certificates to be modified.

Safety Checks

CredHub Maestro performs basic safety checks when rotating certificates to prevent unsafe operations.

To skip CredHub Maestro safety checks, you can pass the --skip-safety-check flag to CredHub Maestro certificate rotation commands.

For more information, see Troubleshooting CredHub Maestro Safety Violations During Certificate Rotation.

Rotate All CAs and Leaf Certificates

This section describes the procedure for bulk rotating all CAs and leaf certificates using CredHub Maestro.

VMware recommends that you use the Ops Manager API to perform bulk rotation of certificates unless you meet the conditions described in When to Use CredHub Maestro for Certificate Rotation.

Note: If any of your deployments use a Services TLS CA:
  • Run all CredHub Maestro commands except maestro regenerate leaf with --exclude /services/tls_ca.
  • Run maestro regenerate leaf with --exclude-signed-by /services/tls_ca.
The process for rotating the Services TLS CA requires additional steps to ensure that there is no downtime. To rotate the Services TLS CA, see Rotate the Services TLS CA and Its Leaf Certificates below.

To rotate all CAs and leaf certificates in CredHub:

  1. Regenerate all CAs. This creates a new version of every CA and marks these new CA versions as transitional. The transitional status indicates that these new CA versions will not yet sign any leaf certificates, though they will be trusted by clients. Run: maestro regenerate ca --all

  2. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.
  3. Mark the signing version of each CA as transitional by running:

    maestro update-transitional signing --all
    
  4. Regenerate all leaf certificates by running:

    maestro regenerate leaf --all
    
  5. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.
  6. Remove the transitional flag on all certificate versions. This removes the old, inactive CA versions on the next deploy. Run:

    maestro update-transitional remove --all
    
  7. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.

Rotating All Leaf Certificates

This section describes the procedure for bulk rotating all leaf certificates using CredHub Maestro.

VMware recommends that you use the Ops Manager API to perform bulk rotation of certificates unless you meet the conditions described in When to Use CredHub Maestro for Certificate Rotation above.

To rotate all leaf certificates in CredHub:

  1. Regenerate all leaf certificates by running:

    maestro regenerate leaf --all
    
  2. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.

    Rotate a Single CA and Its Leaf Certificates

    This section describes the procedure for rotating a single CA and its leaf certificates.

    To rotate a single CA and its leaf certificates: 1. Regenerate the CA you want to rotate. This creates a new version of this CA. Run:

    ```
    maestro regenerate ca --name "CA-NAME"
    ```
    Where `CA-NAME` is the name of the CA you want to rotate.
    <p class="note"><strong>Note:</strong> The CA name you provide must be the full path to the credential.</p>
    
    1. Mark the latest version of the single CA as transitional by running:

      maestro update-transitional latest --name "CA-NAME"
      

      Where CA-NAME is the name of the CA you want to mark as transitional.

    2. Redeploy all deployments that use the CA.

    3. Mark the signing version of the CA as transitional by running:

      maestro update-transitional signing --name "CA-NAME"
      

      Where CA-NAME is the name of the CA you want to mark as transitional.

    4. Regenerate all leaf certificates signed by the CA by running:

      maestro regenerate leaf --signed-by "CA-NAME"
      

      Where CA-NAME is the name of the CA for which you want to regenerate leaf certificates.

    5. Redeploy all deployments that use the CA.

    6. Remove the transitional flag from the CA. This removes the old, inactive CA version on the next deploy. Run:

      maestro update-transitional remove --name "CA-NAME"
      

      Where CA-NAME is the name of the CA you want to remove.

    7. Redeploy all deployments that use the CA.

Rotate the Services TLS CA and Its Leaf Certificates

This section describes the procedure for rotating the Services TLS CA and its leaf certificates.

Before you begin this rotation procedure, ensure that you have the following:

  • Only one Ops Manager foundation and not a multi-site or Wide Area Network (WAN) setup. For a multi-site or WAN setup, you must perform the Apply Changes steps across all your sites or WAN.

  • All certificates must include Subject Alternate Name (SAN) and Common Name (CN) entries.

  • Your services are configured for high availability to minimize potential app impact during CA rotation. Rotating the Services TLS CA requires an upgrade all on-demand service instances, which can cause service downtime.

  • Knowledge whether your deployment contains non-Java MySQL apps that rely on VCAP_SERVICES for TLS. You must rebind and restage these apps during the procedure, which can require additional time and coordination for the rotation. You do not need to rebind and restage MySQL apps that use jdbcURL or consume CA certificates from /etc/ssl/certs/ca-certificate.crt. For more information about determining which apps are affected, see Rotating /services/tls_ca certificate using CredHub transitional certificate rotation feature in the Knowledge Base.

To rotate the Services TLS CA and its leaf certificates:

  1. Check the CredHub Maestro Tile Compatibility document to ensure that all service tile versions currently deployed are compatible with Maestro.

    Warning: If you have any service tiles that are not compatible with CredHub Maestro, do not follow this procedure. If you do, you might experience extended downtime or data loss. Instead, follow the procedure in Rotating the Services TLS CA and Its Leaf Certificates.

  2. Regenerate the Services TLS CA and mark the latest version of the CA as transitional. This creates a new version of the Services TLS CA and indicates that the latest version is inactive, so that all leaf certificates trust the new CA.

    1. On the Services TLS CA, see if your Services TLS CA was CredHub-set by running:

      maestro list
      

      After running this command, you see one Services TLS CA listed per deployment, but it is the same certificate.

    2. If the Services TLS CA was CredHub-set, run:

      maestro regenerate ca --name "/services/tls_ca" --force
      
  3. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. On the Review Pending Changes page, enable the Select All Products checkbox.
    3. For each service tile, enable the Upgrade all service instances errand.

      Note: The name of the Upgrade all service instances errand may be slightly different between services.

    4. Click Apply Changes.
  4. For deployments that use MySQL for VMware Tanzu, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.

    Note: The bg-restage plugin enables zero-downtime blue-green restaging without access to the app source code. For more information, see bg-restage on GitHub.

  5. Mark the signing version of the Services TLS CA as transitional by running:

    maestro update-transitional signing --name "/services/tls_ca"
    
  6. Regenerate all service instance certificates signed by the Services TLS CA by running:

    maestro regenerate leaf --signed-by "/services/tls_ca"
    
  7. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. On the Review Pending Changes page, enable the Select All Products checkbox.
    3. For each service tile, enable the Upgrade all service instances errand.

      Note: The name of the Upgrade all service instances errand may be slightly different between services.

    4. Click Apply Changes.
  8. For deployments that use MySQL for VMware Tanzu, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.

  9. Remove the transitional flag from the Services TLS CA. This removes the old, inactive version of the Services TLS CA on the next deployment. Run:

    maestro update-transitional remove --name "/services/tls_ca"
    
  10. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. On the Review Pending Changes page, enable the Select All Products checkbox.
    3. For each service tile, enable the Upgrade all service instances errand.

      Note: The name of the Upgrade all service instances errand may be slightly different between services.

    4. Click Apply Changes.
  11. For deployments that use MySQL for VMware Tanzu, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.

Rotate Manually Set Certificates

This section describes the procedure for rotating certificates that are manually set in Credhub.

To rotate a manually set certificate:

  1. Create a new CA using the same parameters you used for the original CA.

  2. Use the CredHub CLI or CredHub API to set the new CA at the same path as the original. To download and install the CredHub CLI, see the CredHub CLI repository on GitHub. For more information about the CredHub API, see the CredHub API documentation.

  3. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.
  4. Mark the signing version of the single CA as transitional by running:

    maestro update-transitional signing --name "CA-NAME"
    
  5. Regenerate all leaf certificates that are signed by the CA you are rotating. Run:

    maestro regenerate leaf --signed-by "CA-NAME"
    

    Note: You might need to use the --force flag when regenerating leaf certificates if they were set by CredHub. Otherwise, you must provide a new leaf certificate at the same path.

  6. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.
  7. Remove the transitional flag from the single CA. This removes the old, inactive CA version on the next deploy. Run:

    maestro update-transitional remove --name "CA-NAME"
    
  8. Redeploy:

    1. Return to the Ops Manager Installation Dashboard.
    2. Click Review Pending Changes.
    3. On the Review Pending Changes page, enable the Select All Products checkbox.
    4. For each service tile you have deployed:
      1. Verify that the Recreate all service instances errand is disabled. You do not need to run this errand because when run by itself, CredHub Maestro does not update BOSH Agent certificates.
      2. Enable the Upgrade all service instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.

        Note: The names of the Recreate all service instances and the Upgrade all service instances errands may be slightly different between services.

    5. Click Apply Changes.

Delete Inactive Certificate Versions

Over the course of many rotations, it is possible for the CredHub database to fill up. In order to prevent this, you can periodically use CredHub Maestro’s garbage collect functionality to delete old, unused certificate versions.

You can run garbage collect on leaf certificates and CAs. Generally, deleting leaf certificates is enough to free up space in the database.

To see if you need to run garbage collect, do one of the following:

  • Manually check the BOSH Director’s VM vitals through your IaaS provider.

  • SSH into the BOSH Director and run df -h /var/vcap/store.

Warning: To enhance the safety of the garbage collect command, CredHub Maestro only deletes versions of a certificate older than the currently active version. To delete all inactive versions of CAs or leaf certificates, add the --force flag to the garbage collect command. VMware does not recommend using the --force flag during a rotation.

Delete All Leaf Certificates

To delete all leaf certificates:

  1. Determine which certificate versions you want to delete by running:

    maestro garbage-collect leaf --all --dry-run
    
  2. Delete the certificate versions by running:

    maestro garbage-collect leaf --all
    

Delete All CAs

To delete all CAs:

  1. Determine which CA versions you want to delete by running:

    maestro garbage-collect ca --all --dry-run
    
  2. Delete the CA versions by running:

    maestro garbage-collect ca --all
    

Delete a Single Leaf Certificate

To delete a single leaf certificate:

  1. Determine which certificate versions you want to delete by running:

    maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME" --dry-run
    
  2. Delete the certificate versions by running:

    maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME"
    

Delete a Single CA

To delete a single CA:

  1. Determine which CA you want to delete by running:

    maestro garbage-collect leaf --name "CA-NAME" --dry-run
    

    Where CA-NAME is the name of the CA you want to delete.

  2. Delete the CA by running:

    maestro garbage-collect leaf --name "CA-NAME"
    

    Where CA-NAME is the name of the CA you want to delete.