Troubleshooting CredHub Maestro Safety Violations During Certificate Rotation

Page last updated:

This topic provides operators with information on how to troubleshoot safety violation errors that are returned when rotating certificates with either the Ops Manager API or CredHub Maestro CLI.

Overview of CredHub Maestro Safety Checks

CredHub Maestro performs basic safety checks when rotating certificates to prevent unsafe operations.

The Ops Manager API invokes CredHub Maestro when rotating certificates. If a certificate rotation API command is unsafe, CredHub Maestro stops the command and returns one or more safety violations.

The Ops Manager API does not invoke CredHub Maestro and subsequently CredHub Maestro-related safety checks during certificate rotation if it detects an installation of Pivotal Application Service (PAS) v2.8.2 or earlier, any version of VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), or any version of TKGI (TKGI).

CredHub Maestro returns safety violations if a rotation step is run out of order. For example, you generate a new root certificate authority (CA) but attempt to activate it before clicking Apply Changes. Performing steps out of the correct order during certificate rotations can make your deployment unstable or risk downtime.

For information about certificate rotation using the Ops Manager API, see Overview of Certificate Rotation.

For information about using the CredHub Maestro CLI, see Getting Started with CredHub Maestro.

Skip Safety Checks

To skip CredHub Maestro safety checks in Ops Manager API calls, you can pass skip_safety_check=true as a parameter to certificate rotation API endpoints.

To skip CredHub Maestro safety checks in the CredHub Maestro CLI, specify the --skip-safety-check flag when running certificate rotation commands.

Warning: Skipping safety checks is strongly discouraged. Use the option only if absolutely necessary. For example, you might want to skip safety checks if your CA has already expired and your apps are experiencing downtime. Skipping safety checks can help bring your apps back online faster. If certificates have expired, contact Support for guidance.

Troubleshoot Certificate Rotation Safety Violations

This section lists known safety violations that may occur when using the Ops Manager API or CredHub Maestro CLI to rotate certificates. Each table includes the literal safety violation message, describes the safety violation, and presents possible solutions to resolve the underlying cause.

Multiple Signing Versions of a CA

Safety Violation
there is more than one signing version of a certificate authority

Related Commands This safety violation may appear after calling the following Ops Manager API endpoints:
  • POST /api/v0/certificate_authorities/generate
  • POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate
  • POST /api/v0/certificate_authorities/active/regenerate
  • DELETE /api/v0/certificate_authorities/{certificate_authority_guid}
Or after running one of the following commands:
  • maestro regenerate ca
  • maestro update-transitional signing
  • maestro regenerate leaf
  • maestro update-transitional remove
Cause CredHub Maestro expects that only one version of a CA is used to sign active leaf certificates. During the rotation process, Maestro moves forward from one CA version to another in bulk, but is not equipped to handle mixed-mode operations where multiple CA versions are actively in use.
Solution There are two possible problems with different solutions. To identify and resolve the root issue:
  1. Run maestro topology --name CERTIFICATE-NAME, where CERTIFICATE-NAME is the name and full path of the certificate returned in the error message.
  2. Check the command line output for the following:
    • If the signs field contains a child certificate that has two versions set to active: true with different product tile deployment names:
      1. Navigate to the Ops Manager Installation Dashboard.
      2. Click Review Pending Changes.
      3. Select the product tile that is associated with the older certificate version.
      4. Click Apply Changes.
    • If there are multiple child certificates and their active versions have different signed-by versions:
      1. Run maestro regenerate leaf --signed-by PARENT-CA-NAME, where PARENT-CA-NAME is the name of the signing CA.
      2. Navigate to the Ops Manager Installation Dashboard.
      3. Click Review Pending Changes.
      4. Select all products.
      5. Click Apply Changes.

Transitional Version of CA Detected

Safety Violation
certificate authorities can not have a transitional version

Related Commands This safety violation may appear after calling the POST /api/v0/certificate_authorities/generate Ops Manager API endpoint or after running the maestro regenerate ca command.
Cause CredHub Maestro has detected an existing transitional CA version. This means that the certificate rotation process has already started.
Solution Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

Multiple Active Child Certificates

Safety Violation
active child certificate version is not the latest non-transitional version or more than one active version exists

Related Commands This safety violation may appear after calling the following Ops Manager API endpoints:
  • POST /api/v0/certificate_authorities/generate
  • POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate
Or after running the following commands:
  • maestro regenerate ca
  • maestro update-transitional signing
Cause This error usually occurs if you have not applied your changes to all the deployments that are required to propagate the latest version of the child certificate. A child certificate can refer to a leaf certificate or an intermediate certificate.
Solution To resolve this error:

  1. Navigate to the Ops Manager Installation Dashboard.
  2. Click Review Pending Changes.
  3. Select all products.
  4. Click Apply Changes.
  5. After the deployment completes, confirm that the latest version is propagated by running maestro topology --name CERTIFICATE-NAME, where CERTIFICATE-NAME is the name of the certificate returned in the error message.

Latest CA is Not Transitional

Safety Violation
certificate authorities latest version is not transitional

Related Commands This safety violation may appear after calling the POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate Ops Manager API endpoint or after running the maestro update-transitional signing command.
Cause The latest version of each CA is expected to be transitional before the latest CA version becomes the future signing version. This allows trust to be incrementally added to the latest version before it ever gets used.
Solution Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

If you believe you are running the steps in the correct order, it is possible that a new CA has been created in CredHub after the rotation process has begun.

To work around this situation, take one of the following actions:
  • If the CA is new, exclude this CA by adding the --exclude=CA-NAME option for future commands during this rotation. For example, maestro update-transitional signing --exclude=CA-NAME.
  • If you need to rotate the CA now, re-run the earlier CredHub Maestro rotation steps maestro regenerate ca CA-NAME, where CA-NAME is the name of a CA that was not originally present. Repeat this step for all CAs not originally present. This gets all targeted CAs into the same state. Then, run the maestro update-transitional signing step again.

Signing Version Must Be Transitional

Safety Violation
signing version has to be transitional if there is a transitional certificate authority

Related Commands This safety violation may appear after calling the POST /api/v0/certificate_authorities/active/regenerate Ops Manager API endpoint or after running the maestro regenerate leaf command.
Cause The leaf certificates must only be signed by older versions of their appropriate CAs. At this point in the rotation, the older signing version for each CA should be transitional so that new leaf certificates are issued from the latest version.
Solution Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

If you believe you are running the steps in the correct order, it is possible that a new leaf was issued while the rotation process was in progress. You may also receive the there is more than one signing version error described in Multiple Signing Versions of a CA above. If this safety violation error appears, follow the mitigation steps for that error.

Active Certificates Not Signed By Latest CA Detected

Safety Violation
active child certificate version is signed by a certificate authority that is not the latest version

Related Commands This safety violation may appear after calling the DELETE /api/v0/certificate_authorities/{certificate_authority_guid} Ops Manager API endpoint or after running the maestro update-transitional remove command.
Cause There are still active intermediate CAs or leaf certificates that are not signed by the latest CA version.
Solution Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

Stale Leaf Certificates Detected

Safety Violation
active certificate version is not the latest non transitional version

Related Commands This safety violation may appear after calling the DELETE /api/v0/certificate_authorities/{certificate_authority_guid} Ops Manager API endpoint or after running the maestro update-transitional remove command.
Cause There are still stale leaf certificates in use with new versions ready to be deployed.
Solution Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

Transitional CA Detected as Latest CA

Safety Violation
transitional certificate authority is the latest version

Related Commands This safety violation may appear after calling the DELETE /api/v0/certificate_authorities/{certificate_authority_guid} Ops Manager API endpoint or after running the maestro update-transitional remove command.
Cause The latest version of one or more CAs is transitional, when CredHub Maestro is expecting the transitional version to be an older version.
Solution Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.

It is possible that you are attempting to halt a rotation in the early stages, likely after new transitional CAs have been issued but not yet used. Continue with the rotation at this point. If continuing with the rotation is impossible due to other deployment needs, reach out to Support for guidance.