Troubleshooting CredHub Maestro Safety Violations During Certificate Rotation
Page last updated:
This topic provides operators with information on how to troubleshoot safety violation errors that are returned when rotating certificates with either the Ops Manager API or CredHub Maestro CLI.
Overview of CredHub Maestro Safety Checks
CredHub Maestro performs basic safety checks when rotating certificates to prevent unsafe operations.
The Ops Manager API invokes CredHub Maestro when rotating certificates. If a certificate rotation API command is unsafe, CredHub Maestro stops the command and returns one or more safety violations.
The Ops Manager API does not invoke CredHub Maestro and subsequently CredHub Maestro-related safety checks during certificate rotation if it detects an installation of Pivotal Application Service (PAS) v2.8.2 or earlier, any version of VMware Tanzu Kubernetes Grid Integrated Edition (TKGI), or any version of TKGI (TKGI).
CredHub Maestro returns safety violations if a rotation step is run out of order. For example, you generate a new root certificate authority (CA) but attempt to activate it before clicking Apply Changes. Performing steps out of the correct order during certificate rotations can make your deployment unstable or risk downtime.
For information about certificate rotation using the Ops Manager API, see Overview of Certificate Rotation.
For information about using the CredHub Maestro CLI, see Getting Started with CredHub Maestro.
Skip Safety Checks
To skip CredHub Maestro safety checks in Ops Manager API calls, you can pass skip_safety_check=true
as a parameter to certificate rotation API endpoints.
To skip CredHub Maestro safety checks in the CredHub Maestro CLI, specify the --skip-safety-check
flag when running certificate rotation commands.
Warning: Skipping safety checks is strongly discouraged. Use the option only if absolutely necessary. For example, you might want to skip safety checks if your CA has already expired and your apps are experiencing downtime. Skipping safety checks can help bring your apps back online faster. If certificates have expired, contact Support for guidance.
Troubleshoot Certificate Rotation Safety Violations
This section lists known safety violations that may occur when using the Ops Manager API or CredHub Maestro CLI to rotate certificates. Each table includes the literal safety violation message, describes the safety violation, and presents possible solutions to resolve the underlying cause.
Multiple Signing Versions of a CA
Safety Violation |
there is more than one signing version of a certificate authority
|
---|---|
Related Commands | This safety violation may appear after calling the following Ops Manager API endpoints:
|
Cause | CredHub Maestro expects that only one version of a CA is used to sign active leaf certificates. During the rotation process, Maestro moves forward from one CA version to another in bulk, but is not equipped to handle mixed-mode operations where multiple CA versions are actively in use. |
Solution | There are two possible problems with different solutions. To identify and resolve the root issue:
|
Transitional Version of CA Detected
Safety Violation |
certificate authorities can not have a transitional version
|
---|---|
Related Commands | This safety violation may appear after calling the POST /api/v0/certificate_authorities/generate Ops Manager API endpoint or after running the maestro regenerate ca command. |
Cause | CredHub Maestro has detected an existing transitional CA version. This means that the certificate rotation process has already started. |
Solution | Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started. |
Multiple Active Child Certificates
Safety Violation |
active child certificate version is not the latest non-transitional version or more than one active version exists
|
---|---|
Related Commands | This safety violation may appear after calling the following Ops Manager API endpoints:
|
Cause | This error usually occurs if you have not applied your changes to all the deployments that are required to propagate the latest version of the child certificate. A child certificate can refer to a leaf certificate or an intermediate certificate. |
Solution | To resolve this error:
|
Latest CA is Not Transitional
Safety Violation |
certificate authorities latest version is not transitional
|
---|---|
Related Commands | This safety violation may appear after calling the POST /api/v0/certificate_authorities/{certificate_authority_guid}/activate Ops Manager API endpoint or after running the maestro update-transitional signing command. |
Cause | The latest version of each CA is expected to be transitional before the latest CA version becomes the future signing version. This allows trust to be incrementally added to the latest version before it ever gets used. |
Solution | Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.
If you believe you are running the steps in the correct order, it is possible that a new CA has been created in CredHub after the rotation process has begun. To work around this situation, take one of the following actions:
|
Signing Version Must Be Transitional
Safety Violation |
signing version has to be transitional if there is a transitional certificate authority
|
---|---|
Related Commands | This safety violation may appear after calling the POST /api/v0/certificate_authorities/active/regenerate Ops Manager API endpoint or after running the maestro regenerate leaf command. |
Cause | Before regenerating new leaf certificates, all the existing leaf certificates must only be signed by older versions of their appropriate CAs. These older CA versions are marked as transitional, which indicates that they are inactive but still needs to be trusted by clients. This means the new leaf certificates are signed by the newer CA version and clients trust both the old and new CA versions. |
Solution | Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.
If you believe you are running the steps in the correct order, it is possible that a new leaf was issued while the rotation process was in progress. You may also receive the there is more than one signing version error described in Multiple Signing Versions of a CA above. If this safety violation error appears, follow the mitigation steps for that error.
|
Active Certificates Not Signed By Latest CA Detected
Safety Violation |
active child certificate version is signed by a certificate authority that is not the latest version
|
---|---|
Related Commands | This safety violation may appear after calling the DELETE /api/v0/certificate_authorities/{certificate_authority_guid} Ops Manager API endpoint or after running the maestro update-transitional remove command. |
Cause | There are still active intermediate CAs or leaf certificates that are not signed by the latest CA version. |
Solution | Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started. |
Stale Leaf Certificates Detected
Safety Violation |
active certificate version is not the latest non transitional version
|
---|---|
Related Commands | This safety violation may appear after calling the DELETE /api/v0/certificate_authorities/{certificate_authority_guid} Ops Manager API endpoint or after running the maestro update-transitional remove command. |
Cause | There are still stale leaf certificates in use with new versions ready to be deployed. |
Solution | Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started. |
Transitional CA Detected as Latest CA
Safety Violation |
transitional certificate authority is the latest version
|
---|---|
Related Commands | This safety violation may appear after calling the DELETE /api/v0/certificate_authorities/{certificate_authority_guid} Ops Manager API endpoint or after running the maestro update-transitional remove command. |
Cause | The latest version of one or more CAs is transitional, when CredHub Maestro is expecting the transitional version to be an older version. |
Solution | Continue with the certificate rotation procedure. In most cases, this means clicking Apply Changes in the Ops Manager Installation Dashboard. Do not run earlier commands after the rotation process has started.
It is possible that you are attempting to halt a rotation in the early stages, likely after new transitional CAs have been issued but not yet used. Continue with the rotation at this point. If continuing with the rotation is impossible due to other deployment needs, reach out to Support for guidance. |