Rotating the Root CA and Leaf Certificates
Page last updated:
Warning: Pivotal Operations Manager v2.3 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.
This topic describes how to rotate the Ops Manager root certificate authority (CA) and leaf certificates using the Ops Manager API.
Overview
This procedure uses the Ops Manager API to rotate the Ops Manager root CA and leaf certificates. When you rotate the Ops Manager root CA, the Ops Manager API automatically rotates the BOSH NATS CA. To rotate only leaf certificates, see one of the leaf certificate rotation topics listed in Certificate Rotation Procedures in Overview of Certificate Rotation.
This procedure includes three redeploys. The first deploy adds the new CA certificates to Pivotal Cloud Foundry (PCF) components. The second deploy replaces the old leaf certificates with the new leaf certificates. The third deploy removes the old CA certificates, leaving only the new CA certificates.
Warning: You must complete these steps in the exact order specified. Otherwise, you may damage your deployment.
To rotate the Ops Manager root CA and leaf certificates:
Add a new Ops Manager root CA. For more information, see Add a New Root CA below.
Activate the Ops Manager root CA. For more information, see Activate the Root CA below.
Rotate configurable leaf certificates. For more information, see Rotate Configurable Leaf Certificates below.
Rotate non-configurable leaf certificates. For more information, see Rotate Non-Configurable Leaf Certificates from the New Root CA below.
Delete the old root CA. For more information, see Delete the Old Root CA below.
Warning: The Services TLS CA, /services/tls_ca
, cannot be rotated using this procedure. For information about how to rotate the Services TLS CA, see Rotating the Services TLS CA and Its Leaf Certificates.
Step 1: Add a New Root CA
This section describes how to add a new root CA for Ops Manager. The new root CA can be a PCF-generated CA or your own custom CA.
This procedure also automatically regenerates the BOSH NATS CA.
Warning: The Ops Manager API also excludes from rotation the CAs for certain versions of the MySQL, Redis, RabbitMQ, and PCF Cloud Cache tiles. For information about rotating these CAs, see the documentation for each tile.
To add a new root CA for Ops Manager and regenerate the BOSH NATS CA:
If you have not already done so, follow the procedure in Using Ops Manager API to target and authenticate with the Ops Manager UAA server. Record your Ops Manager access token, and use it for
UAA-ACCESS-TOKEN
in the following steps.Note: When you record your Ops Manager access token, remove any newline characters such as
\n
.Use Ops Manager to generate a new CA, or add your own custom CA:
Note: Elliptic Curve Digital Signature Algorithm (ECDSA) certificates are not supported in PCF.
To use your own custom CA: Call the Ops Manager API
certificate_authorities
endpoint by running:curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \ -X POST \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" \ -H "Content-Type: application/json" \ -d '{"cert_pem": "-----BEGIN CERTIFICATE-----CUSTOM-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----RSA-KEY"}' -i
Where:
OPS-MANAGER-FQDN
is the fully-qualified domain name (FQDN) of your Ops Manager deployment.CUSTOM-CERTIFICATE
is your custom CA.RSA-KEY
is your RSA key.UAA-ACCESS-TOKEN
is your UAA access token.
If the command succeeds, the API returns a response that includes the new CA. For example:HTTP/1.1 200 OK { "certificate_authorities": [ { "guid": "f7bc18f34f2a7a9403c3", "issuer": "CUSTOM-CERTIFICATE", "created_on": "2017-01-09", "expires_on": "2021-01-09", "active": true, "cert_pem": "-----\BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc" } ] }
To use a PCF-generated CA: Call the Ops Manager API
generate
endpoint by running:curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/generate" \ -X POST \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" \ -H "Content-Type: application/json" \ -d '{}' -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN
is your UAA access token.
If the command succeeds, the API returns a response that includes the new CA. For example:HTTP/1.1 200 OK { "guid": "f7bc18f34f2a7a9403c3", "issuer": "Pivotal", "created_on": "2017-01-19", "expires_on": "2021-01-19", "active": false, "cert_pem": "-----\BEGIN EXAMPLE CERTIFICATE-----\ MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84= -----\END EXAMPLE CERTIFICATE-----\" }
Confirm that the new CA is added by listing all the CAs for Ops Manager. Run:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \ -X GET \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN
is your UAA access token.
The API call returns the GUID of the newly added CA. For example:HTTP/1.1 200 OK { "certificate_authorities": [ { "guid": "f7bc18f34f2a7a9403c3", "issuer": "Pivotal", "created_on": "2017-01-09", "expires_on": "2021-01-09", "active": true, "cert_pem": "-----\BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc" } { "guid": "a8ee01e33e3e3e3303e3", "issuer": "Pivotal", "created_on": "2017-04-09", "expires_on": "2021-04-09", "active": false, "cert_pem": "-----\BEGIN CERTIFICATE-----\nzBBBC+eAAAe1gAwAAAeZ....etc" } ] }
Identify the newly added CA, which has
active
set tofalse
. Record its GUID.Navigate to
https://OPS-MANAGER-FQDN
in a browser, whereOPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.Log in to the Ops Manager Installation Dashboard.
Click the BOSH Director tile.
Select Director Config.
Enable the Recreate All VMs checkbox. This propagates the new CA to all VMs deployed by the BOSH Director to prevent downtime.
Return to the Ops Manager Installation Dashboard.
If you have service tiles installed, for each service tile:
Note: The names of the Upgrade All Service Instances and Recreate All Service Instances errands may be slightly different between services.
- Click the tile.
- Click the Errands tab.
- Enable the Upgrade All Service Instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
- If the service tile has the Recreate All Service Instances errand:
- Enable the Recreate All Service Instances errand.
- Click Review Pending Changes.
- Click Apply Changes.
If the service tile does not have the Recreate All Service Instances errand:
- Click Review Pending Changes.
- Click Apply Changes.
When the deploy finishes, manually push the BOSH NATS CA to each service instance. For each service instance, run:
bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
Where
SERVICE-INSTANCE-DEPLOYMENT
is the BOSH deployment name for the service instance.
If you do not have any service tiles installed:
- Click Review Pending Changes.
- Click Apply Changes.
When the deploy finishes and you have completed all necessary tasks to update service instances, continue to the next section, Activate the Root CA.
Step 2: Activate the Root CA
To activate the new root CA:
Call the Ops Manager API
activate
endpoint by running:curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/CERTIFICATE-GUID/activate" \ -X POST \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" \ -H "Content-Type: application/json" \ -d '{}' -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.CERTIFICATE-GUID
is the GUID of your CA that you retrieved in the previous section.UAA-ACCESS-TOKEN
is your UAA access token.
The API returns a successful response:HTTP/1.1 200 OK
List your CAs to confirm that the new Ops Manager root CA is active. Run:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \ -X GET \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN
is your UAA access token.
Examine the response to ensure that your new root CA has the
active
property set totrue
.
Step 3: Rotate Configurable Leaf Certificates
Any configurable leaf certificates generated from the Ops Manager root CA must be rotated. These are the leaf certificates that you generated using the Generate RSA Certificate button in the Ops Manager UI. For example, see Providing a Certificate for Your TLS Termination Point.
To regenerate and rotate each configurable leaf certificate that expires soon:
Find the text field where the leaf certificate is configured within the Ops Manager UI. You might need to look through multiple configuration panes to identify the location of the certificate configuration text field. Use the following fields to identify the location of the certificate configuration text field:
- The
product_guid
field in the Ops Manager API output can help identify the tile in which the certificate is configured. For example, the prefixp-bosh-
refers to the BOSH Director tile, and the prefixcf-
refers to the Pivotal Application Service (PAS) tile. - The
property_reference
field in the Ops Manager API output can help identify the configuration pane in which the certificate is configured. For example, theuaa.service_provider_key_credentials
property is configured in the UAA pane of the PAS tile.
- The
Copy a new value for the leaf certificate into the text field or generate a new leaf certificate using the Generate RSA Certificate button.
Click Save at the bottom of each pane in which you added new leaf certificates.
Step 4: Rotate Non-Configurable Leaf Certificates from the New Root CA
After activating the new Ops Manager root CA, you must rotate non-configurable leaf certificates from the new root CA.
To rotate non-configurable leaf certificates:
Call the Ops Manager API
regenerate
endpoint to regenerate all non-configurable leaf certificates and apply the new root CA to your existing BOSH Director. Run:curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/active/regenerate" \ -X POST \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" \ -H "Content-Type: application/json" \ -d '{}' -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN
is your UAA access token.
The API returns a successful response:HTTP/1.1 200 OK
Navigate to the Ops Manager Installation Dashboard.
Click the BOSH Director tile.
Select Director Config.
Enable the Recreate All VMs checkbox. This propagates the new CAs to all VMs deployed by the BOSH Director to prevent downtime.
Return to the Ops Manager Installation Dashboard.
If you have service tiles installed, for each service tile:
Note: The names of the Upgrade All Service Instances and Recreate All Service Instances errands may be slightly different between services.
- Click the tile.
- Click the Errands tab.
- Enable the Upgrade All Service Instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
- If the service tile has the Recreate All Service Instances errand:
- Enable the Recreate All Service Instances errand.
- Click Review Pending Changes.
- Click Apply Changes.
If the service tile does not have the Recreate All Service Instances errand:
- Click Review Pending Changes.
- Click Apply Changes.
When the deploy finishes, manually push the BOSH NATS CA to each service instance. For each service instance, run:
bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
Where
SERVICE-INSTANCE-DEPLOYMENT
is the BOSH deployment name for the service instance.
If you do not have any service tiles installed:
- Click Review Pending Changes.
- Click Apply Changes.
When the deploy finishes and you have completed all necessary tasks to update service instances, continue to the next section, Delete the Old Root CA.
Step 5: Delete the Old Root CA
After activating new CAs and rotating leaf certificates from the new CAs, you should delete the old Ops Manager root CA from your deployment.
If your old root CA or leaf certificates have been compromised, you must delete the old root CA immediately. Otherwise, the old root CA and its leaf certificates may still be trusted, which compromises your deployment’s security. If your old root CA or leaf certificates have not been compromised, Pivotal recommends that you delete the old root CA as soon as possible, but you can also delete it at a later time, such as in conjunction with another deploy.
Warning: Before you delete the old CA, ensure that you reviewed the Review Pending Changes page and clicked Apply Changes as part of Rotate Non-Configurable Leaf Certificates from the New Root CA above.
To delete the old Ops Manager root CA:
List your root CAs to retrieve the GUID of the old, inactive Ops Manager root CA. Run:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \ -X GET \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.UAA-ACCESS-TOKEN
is your UAA access token.
Delete the old Ops Manager root CA by running:
curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/OLD-CERTIFICATE-GUID" \ -X DELETE \ -H "Authorization: Bearer UAA-ACCESS-TOKEN" -i
Where:
OPS-MANAGER-FQDN
is the FQDN of your Ops Manager deployment.OLD-CERTIFICATE-GUID
is the GUID of the old Ops Manager root CA.UAA-ACCESS-TOKEN
is your UAA access token.
The API returns a successful response:HTTP/1.1 200 OK
Navigate to the Ops Manager Installation Dashboard.
Click the BOSH Director tile.
Select Director Config.
Enable the Recreate All VMs checkbox.
Return to the Ops Manager Installation Dashboard.
If you have service tiles installed, for each service tile:
Note: The names of the Upgrade All Service Instances and Recreate All Service Instances errands may be slightly different between services.
- Click the tile.
- Click the Errands tab.
- Enable the Upgrade All Service Instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
- If the service tile has the Recreate All Service Instances errand:
- Enable the Recreate All Service Instances errand. Running this errand pushes BOSH Agent certificate updates to service instances.
- Click Review Pending Changes.
- Click Apply Changes.
If the service tile does not have the Recreate All Service Instances errand:
- Click Review Pending Changes.
- Click Apply Changes.
When the deploy finishes, manually push BOSH Agent certificate updates to each service instance. For each service instance, run:
bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
Where
SERVICE-INSTANCE-DEPLOYMENT
is the BOSH deployment name for the service instance.
If you do not have any service tiles installed:
- Click Review Pending Changes.
- Click Apply Changes.