Advanced Certificate Rotation with CredHub Maestro

Overview
Prerequisites
Check Expiration Dates and Certificate Types
Rotate Certificates
Delete Inactive Certificate Versions

Page last updated:

This topic describes how to rotate certificates and certificate authorities (CAs) in BOSH CredHub using CredHub Maestro.

This topic only covers rotating CredHub-managed certificates. To rotate Ops Manager-managed certificates, see Overview of Certificate Rotation.

Overview

CredHub Maestro is a command-line interface (CLI) that facilitates rotations of certificates in CredHub. Using CredHub Maestro, you can:

Determine if any of your CredHub certificates are expiring soon
Rotate CredHub certificates
Clean up inactive certificate versions so that CredHub does not run out of disk space

For more information about setting up and using CredHub Maestro, see Getting Started with CredHub Maestro.

Warning: Not every Pivotal Platform tile supports certificate rotation, so there is a potential for downtime. For more information about which tiles support certificate rotation, see CredHub Maestro Tile Compatibility.

Prerequisites

Before you rotate CredHub-managed certificates:

All CredHub Maestro rotation workflows ignore certificates manually set in CredHub by default. Before rotating certificates in Ops Manager v2.8, you must reset any certificates that were manually set in CredHub on Ops Manager v2.6 and earlier to prevent them from being rotated with other certificates generated by CredHub. To find and reset any manually set certificates in CredHub, follow the procedure in Reviewing Manually Set Certificates in CredHub. For more information, see Reset Manually Set Certificates in CredHub Before Rotating Certificates in Pivotal Operations Manager v2.8 Release Notes.
To ensure that there is no app availability downtime during a rotation, Pivotal recommends that you scale up all instance groups and use an external blobstore.
Check which tiles in your installation support certificate rotation with CredHub Maestro. Not every tile currently supports certificate rotation, so there is a potential for downtime. For more information about which tiles support certificate rotation, see CredHub Maestro Tile Compatibility.

Check Expiration Dates and Certificate Types

This section describes how to manually check the expiration dates of the CAs and leaf certificates managed by CredHub.

After identifying the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

To check certificate expiration dates and types:

Retrieve a list of certificates expiring within a given timeframe by running:
```
maestro list --expires-within TIME-PERIOD
```
Where TIME-PERIOD is the unit for which you want to check the expiry window. Valid units are d for days, w for weeks, m for months, and y for years.
After you identify the list of certificates that expire soon, follow the procedure in Rotate Certificates.

Rotate Certificates

This section explains how to rotate leaf certificates and CAs stored in CredHub. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated.

To rotate certificates, do one of the following procedures:

If multiple CAs are expiring soon, see Rotate All CAs and Leaf Certificates.
If the only certificates expiring are leaf certificates, see Rotate All Leaf Certificates.
To only rotate a subset of CAs, see Rotate a Single CA and Its Leaf Certificates.
To rotate the Services TLS CA, see Rotate the Services TLS CA and Its Leaf Certificates.
To rotate CredHub Set Certificates, see Rotate CredHub Set Certificates.

Note: All CredHub Maestro rotation commands have a dry-run flag that can be used to preview the certificates to be modified.

Rotate All CAs and Leaf Certificates

This section describes the procedure for rotating all CAs and leaf certificates.

Note: If any of your deployments use a Services TLS CA:

Run all CredHub Maestro commands except maestro regenerate leaf with --exclude /services/tls_ca.
Run maestro regenerate leaf with --exclude-signed-by /services/tls_ca.

The process for rotating the Services TLS CA requires additional steps to ensure that there is no downtime. To rotate a Services TLS CA, see Rotate the Services TLS CA and Its Leaf Certificates.

To rotate all CAs and leaf certificates in CredHub:

Regenerate all CAs. This creates a new version of every CA. Run:
```
maestro regenerate ca --all
```
Mark the latest version of each CA as transitional. This indicates that the certificate is inactive, so that all leaf certificates trust the new CA version. Run:
```
maestro update-transitional latest --all
```
Redeploy all deployments.
Mark the signing version of each CA as transitional by running:
```
maestro update-transitional signing --all
```
Regenerate all leaf certificates by running:
```
maestro regenerate leaf --all
```
Redeploy all deployments.
Remove the transitional flag on all certificate versions. This removes the old, inactive CA versions on the next deploy. Run:
```
maestro update-transitional remove --all
```
Redeploy all deployments.

Rotating All Leaf Certificates

This section describes the procedure for rotating all leaf certificates.

To rotate all leaf certificates in CredHub:

Regenerate all leaf certificates by running:
```
maestro regenerate leaf --all
```
Redeploy all deployments.

Rotate a Single CA and Its Leaf Certificates

This section describes the procedure for rotating a single CA and its leaf certificates.

To rotate a single CA and its leaf certificates:

Regenerate the CA you want to rotate. This creates a new version of this CA. Run:
```
maestro regenerate ca --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to rotate.
Note: The CA name you provide must be the full path to the credential.
Mark the latest version of the single CA as transitional by running:
```
maestro update-transitional latest --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to mark as transitional.
Redeploy all deployments that use the CA.
Mark the signing version of the CA as transitional by running:
```
maestro update-transitional signing --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to mark as transitional.
Regenerate all leaf certificates signed by the CA by running:
```
maestro regenerate leaf --signed-by "CA-NAME"
```
Where CA-NAME is the name of the CA for which you want to regenerate leaf certificates.
Redeploy all deployments that use the CA.
Remove the transitional flag from the CA. This removes the old, inactive CA version on the next deploy. Run:
```
maestro update-transitional remove --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to remove.
Redeploy all deployments that use the CA.

Rotate the Services TLS CA and Its Leaf Certificates

This section describes the procedure for rotating the Services TLS CA and its leaf certificates.

Before you begin this rotation procedure, ensure that you have the following:

Only one Ops Manager foundation and not a multi-site or Wide Area Network (WAN) setup. For a multi-site or WAN setup, you must perform the Apply Changes steps across all your sites or WAN.
All certificates must include Subject Alternate Name (SAN) and Common Name (CN) entries.
Set up your services for high availability to minimize potential app impact during CA rotation. Rotating the Services TLS CA requires an upgrade all on-demand service instances, which can cause service downtime.
Determined whether your deployment contains non-Java MySQL apps that rely on VCAP_SERVICES for TLS. You must rebind and restage these apps during the rotation. You do not need to rebind and restage MySQL apps that use jdbcURL or consume CA certificates from /etc/ssl/certs/ca-certificate.crt. For more information about determining which apps are affected, see this Knowlegebase Article.

To rotate the Services TLS CA and its leaf certificates:

Check the CredHub Maestro Tile Compatibility document to ensure that all service tile versions currently deployed are compatible with Maestro.
Warning: If you have any service tiles that are not compatible with CredHub Maestro, do not follow this procedure. If you do, you might experience extended downtime or data loss. Use the procedure documented in Rotating the Services TLS CA and Its Leaf Certificates instead.
Regenerate the Services TLS CA.
1. On the Services TLS CA, see if your Services TLS CA was CredHub-set by running:
```
maestro list
```
  After running this command, you see one Services TLS CA listed per deployment, but it is the same certificate.
2. If the Services TLS CA was CredHub-set, run:
```
maestro regenerate ca --name "/services/tls_ca" --force
```
Mark the latest version of the Services TLS CA as transitional by running:
```
maestro update-transitional latest --name "/services/tls_ca"
```
Tell Ops Manager to trust your new Services TLS CA certificate:
1. In Ops Manager, paste the certificate into the BOSH Director > Security > Trusted Certificates field.
2. Click Save.
3. In the PAS and Isolation Segment tiles, paste the certificate into the Networking > Certificate Authorities trusted by the Gorouter and Certificate Authorities trusted by the HAProxy fields.
4. Click Save.
Redeploy:
1. Return to the Ops Manager Installation Dashboard.
2. On the Review Pending Changes page, enable the Select All Products checkbox.
3. For each service tile, enable the Upgrade all service instances errand.
  Note: The name of the Upgrade all service instances errand may be slightly different between services.
4. Click Apply Changes.
For deployments that use MySQL for Pivotal Platform, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.

Note: The bg-restage plugin enables zero-downtime blue-green restaging without access to the app source code. For more information, see bg-restage on GitHub.
Mark the signing version of the Services TLS CA as transitional. Depending on your PAS version, do one of the following:
- If you are on PAS v2.9.3 or earlier, run:
```
maestro topology --name /services/tls_ca
```
  Then, record the certificate_id and the version_id of the version with signing: true and run:
```
credhub curl -p /api/v1/certificates/{certificate_id}/update_transitional_version -i -d '{"version": "{version_id}"}'
```
- If you are on PAS v2.9.4 or later, run:
```
maestro update-transitional signing --name "/services/tls_ca"
```
Regenerate all service instance certificates signed by the Services TLS CA by running:
```
maestro regenerate leaf --signed-by "/services/tls_ca"
```
Redeploy:
1. Return to the Ops Manager Installation Dashboard.
2. On the Review Pending Changes page, enable the Select All Products checkbox.
3. For each service tile, enable the Upgrade all service instances errand.
  Note: The name of the Upgrade all service instances errand may be slightly different between services.
4. Click Apply Changes.
For deployments that use MySQL for Pivotal Platform, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.
Remove the transitional flag from the Services TLS CA. This removes the old, inactive version of the Services TLS CA on the next deployment. Run:
```
maestro update-transitional remove --name "/services/tls_ca"
```
Redeploy:
1. Return to the Ops Manager Installation Dashboard.
2. On the Review Pending Changes page, enable the Select All Products checkbox.
3. For each service tile, enable the Upgrade all service instances errand.
  Note: The name of the Upgrade all service instances errand may be slightly different between services.
4. Click Apply Changes.
For deployments that use MySQL for Pivotal Platform, rebind and restage all non-Java MySQL apps that rely on VCAP_SERVICES for TLS.

Rotate Manually Set Certificates

This section describes the procedure for rotating certificates that are manually set in CredHub.

To rotate a manually set certificate:

Create a new CA using the same parameters you used for the original CA.
Use the CredHub CLI or CredHub API to set the new CA at the same path as the original. To download and install the CredHub CLI, see the CredHub CLI repository on GitHub. For more information about the CredHub API, see the CredHub API documentation.
Mark the latest version of the single CA as transitional. This indicates that the certificate is inactive. Run:
```
maestro update-transitional latest --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to mark as transitional.
Note: The CA name you provide must be the full path to the credential.
Redeploy all deployments that use the CA you are rotating.
Mark the signing version of the single CA as transitional by running:
```
maestro update-transitional signing --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to mark as transitional.
Regenerate all leaf certificates that are signed by the CA you are rotating. Run:
```
maestro regenerate leaf --signed-by "CA-NAME"
```
Where CA-NAME is the name of the CA for which you want to regenerate leaf certificates.
Note: You might need to use the --force flag when regenerating leaf certificates if they were set by CredHub. Otherwise, you must provide a new leaf certificate at the same path.
Redeploy all deployments that use the CA you are rotating.
Remove the transitional flag from the single CA. This removes the old, inactive CA version on the next deploy. Run:
```
maestro update-transitional remove --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to remove.
Redeploy all deployments that use the CA you are rotating.

Delete Inactive Certificate Versions

Over the course of many rotations, it is possible for the CredHub database to fill up. In order to prevent this, you can periodically use CredHub Maestro’s garbage collection functionality to delete old, unused certificate versions.

You can run garbage collection on leaf certificates and CAs. Generally, deleting leaf certificates is enough to free up space in the database.

To see if you need to run garbage collection, do one of the following:

Manually check the BOSH Director’s VM vitals through your IaaS provider.
SSH into the BOSH Director and run df -h /var/vcap/store.

Warning: To enhance the safety of the garbage-collect command, CredHub Maestro only deletes versions of a certificate older than the currently active version. To delete all inactive versions of CAs or leaf certificates, add the --force flag to the garbage-collect command. Pivotal does not recommend using the --force flag during a rotation.

Delete All Leaf Certificates

To delete all leaf certificates:

Determine which certificate versions you want to delete by running:
```
maestro garbage-collect leaf --all --dry-run
```
Delete the certificate versions by running:
```
maestro garbage-collect leaf --all
```

Delete All CAs

To delete all CAs:

Determine which CA versions you want to delete by running:
```
maestro garbage-collect ca --all --dry-run
```
Delete the CA versions by running:
```
maestro garbage-collect ca --all
```

Delete a Single Leaf Certificate

To delete a single leaf certificate:

Determine which certificate versions you want to delete by running:
```
maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME" --dry-run
```
Where LEAF-CERTIFICATE-NAME is the name of the leaf certificate you want to delete.
Delete the certificate versions by running:
```
maestro garbage-collect leaf --name "LEAF-CERTIFICATE-NAME"
```
Where LEAF-CERTIFICATE-NAME is the name of the leaf certificate you want to delete.

Delete a Single CA

To delete a single CA:

Determine which CA you want to delete by running:
```
maestro garbage-collect ca --name "CA-NAME" --dry-run
```
Where CA-NAME is the name of the CA you want to delete.
Delete the CA by running:
```
maestro garbage-collect ca --name "CA-NAME"
```
Where CA-NAME is the name of the CA you want to delete.

Create a pull request or raise an issue on the source for this page in GitHub