Advanced Certificate Rotation with CredHub Maestro

Page last updated:

This topic describes how to rotate certificates and Certificate Authorities (CAs) in CredHub using CredHub Maestro.

This topic only covers rotating CredHub-managed certificates. To rotate Ops Manager managed certificates, see Rotating Certificates.

Warning: Not every Pivotal Platform tile currently supports certificate rotation so there is a potential for downtime.

Overview

CredHub Maestro is a command-line interface (CLI) that facilitates rotations of certificates in CredHub. Using CredHub Maestro, you can:

  • Determine if any of your CredHub certificates are expiring soon

  • Rotate CredHub certificates

  • Clean up inactive certificate versions so that CredHub does not run out of disk space

For more information about setting up and using CredHub Maestro, see Getting Started with CredHub Maestro.

Prerequisites

Before you rotate CredHub-managed certificates:

  • All CredHub Maestro rotation workflows ignore CredHub-generated certificates by default. Before rotating certificates in Ops Manager v2.8, you must reset any certificates that were manually set in CredHub on Ops Manager v2.6 and earlier to prevent them from being rotated with other certificates generated by CredHub. To find and reset any manually set certificates in CredHub, follow the procedure in Reviewing Manually Set Certificates in CredHub. For more information, see Reset Manually Set Certificates in CredHub Before Rotating Certificates in Pivotal Operations Manager v2.8 Release Notes.

  • To ensure that there is no app availability downtime during a rotation, Pivotal recommends that you scale up all instance groups and use an external blobstore.

Check Expiration Dates and Certificate Types

This section describes how to manually check the expiration dates of the CAs and leaf certificates managed by CredHub.

After identifying the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

To check certificate expiration dates and types:

  1. Retrieve a list of certificates expiring within a given timeframe by running:

    maestro list --expires-within TIME-PERIOD
    

    Where TIME-PERIOD is the unit for which you want to check the expiry window. Valid units are d for days, w for weeks, m for months, and y for years.

  2. After you identify the list of certificates that expire soon, complete the Rotate Certificates procedure below.

Rotate Certificates

This section explains how to rotate leaf certificates and CAs stored in CredHub. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated.

To rotate certificates, do one of the following procedures:

Note: All CredHub Maestro rotation commands have a dry-run flag that can be used to preview the certificates to be modified.

Rotate All CAs and Leaf Certificates

This section describes the procedure for rotating all CAs and leaf certificates.

Note: If any of your deployments use a Services TLS CA:
  • Run all CredHub Maestro commands except maestro regenerate leaf with --exclude /services/tls_ca.
  • Run maestro regenerate leaf with --exclude-signed-by /services/tls_ca.
The process for rotating the Services TLS CA requires additional steps to ensure that there is no downtime. To rotate a Services TLS CA, see Rotate the Services TLS CA and Its Leaf Certificates.

To rotate all CAs and leaf certificates in CredHub:

  1. Regenerate all CAs. This creates a new version of every CA. Run:

    maestro regenerate ca --all
    
  2. Mark the latest version of each CA as transitional. This indicates that the certificate is inactive, so that all leaf certificates trust the new CA version. Run:

    maestro update-transitional latest --all
    
  3. Redeploy all deployments.

  4. Mark the signing version of each CA as transitional by running:

    maestro update-transitional signing --all
    
  5. Regenerate all leaf certificates by running:

    maestro regenerate leaf --all
    
  6. Redeploy all deployments.

  7. Remove the transitional flag on all certificate versions. This removes the old, inactive CA versions on the next deploy. Run:

    maestro update-transitional remove --all
    
  8. Redeploy all deployments.

Rotating All Leaf Certificates

This section describes the procedure for rotating all leaf certificates.

To rotate all leaf certificates in CredHub:

  1. Regenerate all leaf certificates by running:

    maestro regenerate leaf --all
    
  2. Redeploy all deployments.

Rotate a Single CA and Its Leaf Certificates

This section describes the procedure for rotating a single CA and its leaf certificates.

To rotate a single CA and its leaf certificates:

  1. Regenerate the CA you want to rotate. This creates a new version of this CA. Run:

    maestro regenerate ca --name "CA-NAME"
    

    Note: The CA name you provide must be the full path to the credential.

  2. Mark the latest version of the single CA as transitional by running:

    maestro update-transitional latest --name "CA-NAME"
    
  3. Redeploy all deployments that use the CA.

  4. Mark the signing version of the CA as transitional by running:

    maestro update-transitional signing --name "CA-NAME"
    
  5. Regenerate all leaf certificates signed by the CA by running:

    maestro regenerate leaf --signed-by "CA-NAME"
    
  6. Redeploy all deployments that use the CA.

  7. Remove the transitional flag from the CA. This removes the old, inactive CA version on the next deploy. Run:

    maestro update-transitional remove --name "CA-NAME"
    
  8. Redeploy all deployments that use the CA.

Rotate the Services TLS CA and Its Leaf Certificates

This section describes the procedure for rotating the Services TLS CA and its leaf certificates.

Warning: Rotating the Services TLA CA and its leaf certificates is not yet supported. Do not attempt to rotate these certificates.

Rotate CredHub-Set Certificates

This section describes the procedure for rotating CredHub-set certificates.

To rotate a CredHub-set certificate:

  1. Create a new CA using the same parameters you used for the original CA.

  2. Use the CredHub CLI or CredHub API to set the new CA at the same path as the original.

  3. Mark the latest version of the single CA as transitional. This indicates that the certificate is inactive. Run:

    maestro update-transitional latest --name "CA-NAME"
    

    Note: The CA name you provide must be the full path to the credential.

  4. Redeploy all deployments that use the CA you are rotating.

  5. Mark the signing version of the single CA as transitional by running:

    maestro update-transitional signing --name "CA-NAME"
    
  6. Regenerate all leaf certificates that are signed by the CA you are rotating. Run:

    maestro regenerate leaf --signed-by "CA-NAME"
    

    Note: You might need to use the --force flag when regenerating leaf certificates if they were set by CredHub. Otherwise, you must provide a new leaf certificate at the same path.

  7. Redeploy all deployments that use the CA you are rotating.

  8. Remove the transitional flag from the single CA. This removes the old, inactive CA version on the next deploy. Run:

    maestro update-transitional remove --name "CA-NAME"
    
  9. Redeploy all deployments that use the CA you are rotating.

Delete Inactive Certificate Versions

Over the course of many rotations, it is possible for the CredHub database to fill up. In order to prevent this, you can periodically use CredHub Maestro’s garbage collect functionality to delete old, unused certificate versions.

You can run garbage collect on leaf certificates and CAs. Generally, deleting leaf certificates is enough to free up space in the database.

To see if you need to run garbage collect, do one of the following:

  • Manually check the BOSH Director’s VM vitals through your IaaS provider.

  • SSH into the BOSH Director and run df -h /var/vcap/store.

Warning: To enhance the safety of the garbage collect command, CredHub Maestro only deletes versions of a certificate older than the currently active version. To delete all inactive versions of CAs or leaf certificates, add the --force flag to the garbage collect command. Pivotal does not recommend using the --force flag during a rotation.

Delete All Leaf Certificates

To delete all leaf certificates:

  1. Determine which certificate versions you want to delete by running:

    maestro garbage-collect leaf --all --dry-run
    
  2. Delete the certificate versions by running:

    maestro garbage-collect leaf --all
    

Delete All CAs

To delete all CAs:

  1. Determine which CA versions you want to delete by running:

    maestro garbage-collect ca --all --dry-run
    
  2. Delete the CA versions by running:

    maestro garbage-collect ca --all
    

Delete a Single Leaf Certificate

To delete a single leaf certificate:

  1. Determine which certificate versions you want to delete by running:

    maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME" --dry-run
    
  2. Delete the certificate versions by running:

    maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME"
    

Delete a Single CA

To delete a single CA:

  1. Determine which certificate versions you want to delete by running:

    maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME" --dry-run
    
  2. Delete the certificate versions by running:

    maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME"