Rotating Certificates

Page last updated:

This topic describes how to check the expiration date of and rotate certificate authorities (CAs) and leaf certificates in Ops Manager.

This includes the Ops Manager root CA, BOSH NATS CA, and leaf certificates stored in Ops Manager and CredHub that are visible to the Ops Manager API.

Warning: The rotation procedures described in this topic do not work if the certificates have already expired. If the certificates have expired, contact Support for guidance.

For information about rotating IPsec certificates, see Rotating IPsec Certificates.

For information about using trusted third-party certificates for both apps hosted on Ops Manager and internal Ops Manager components, see Setting Trusted Certificates.

Overview

The Ops Manager API manages and lists internal CAs and leaf certificates that enable Ops Manager components to communicate with each other securely using TLS. It can also list certificates used externally, such as SAML certificates that authenticate to an external identity provider (IDP).

For more information about the CAs and leaf certificates visible to the Ops Manager API, see Certificate Types.

Rotate CAs and leaf certificates before they expire to avoid downtime for your foundation.

Procedure

This section explains how to rotate the Ops Manager root CA, BOSH NATS CA, and leaf certificates in Ops Manager.

Note: The BOSH NATS CA is automatically rotated when you rotate the Ops Manager root CA. For more information, see Certificate Types.

To rotate certificates in Ops Manager, you first check the expiration dates of all certificates. Then, based on the types of certificates that expire soon, you follow a certificate rotation procedure to replace expiring certificates and redeploy BOSH to apply changes.

To rotate CAs and leaf certificates:

  1. Check the expiration dates of the Ops Manager root CA and leaf certificates, and identify the types of leaf certificates that require rotation. For more information, see Check Expiration Dates and Certificate Types below.

  2. Rotate the certificates that expire soon. For more information, see Rotate Certificates below.

Check Expiration Dates and Certificate Types

This section describes how to check the expiration dates of the CAs and leaf certificates that the Ops Manager API lists and manages. It also explains how to identify the types of certificates that require manual rotation.

After identifying the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

  1. Follow the procedure in Using Ops Manager API to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for UAA-ACCESS-TOKEN in the steps below.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. To retrieve the Ops Manager certificates, use curl to call the /deployed/certificates endpoint of the Ops Manager API. Run:

    curl "https://OPS-MANAGER-FQDN/api/v0/deployed/certificates" \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN"
    

    Where:

    • OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) of your Ops Manager deployment.
    • UAA-ACCESS-TOKEN is the access_token value you recorded in the previous step.

      More options:
      • To limit command output to certificates that expire within a given time interval, append ?expires_within=TIME to the endpoint, replacing TIME with an integer-letter code.
        • Valid letter codes are d for days, w for weeks, m for months, and y for years. For example, querying to https://OPS-MANAGER-FQDN/api/v0/deployed/certificates?expires_within=6m searches for certificates expiring within six months.
      • To make the JSON output more readable, you can pipe your curl command to jq or another text editor with JSON formatting.
  3. The deployed/certificates output lists all CAs and leaf certificates visible to the Ops Manager API, whether they are stored in Ops Manager directly or stored in CredHub. To determine the expiration date and type for each certificate listed:

    1. Determine the expiration date from the valid_until value. For example, the root CA listed below expires on August 12, 2020:
      {
      "configurable": true,
      "is_ca": true,
      "property_reference": ".properties.root_ca.fb10484dd5541a273c9d",
      "property_type": "rsa_cert_credentials",
      "product_guid": "ops_manager",
      "location": "ops_manager",
      "variable_path": null,
      "issuer": "/CN=ToolsmithsCA",
      "valid_from": "2019-08-13T15:30:22Z",
      "valid_until": "2020-08-12T15:30:22Z"
      },
      
    2. For any certificates expiring soon, use the following rules to identify their types:
      • Non-rotatable certificates: Non-rotatable certificates have the following property value:
        • variable_path is /opsmgr/bosh_dns/tls_ca.
      • Non-configurable certificates: Non-configurable leaf certificates have the following property values:
        • variable_path is not /opsmgr/bosh_dns/tls_ca.
        • configurable is false.
        • location is either ops_manager or credhub.
        • is_ca is false.
      • Configurable certificates: Configurable leaf certificates have the following property values:
        • configurable is true.
        • is_ca is false.
      • Ops Manager root CA and matching BOSH NATS CA: CAs have the following property values:
        • is_ca is true.
        • location is ops_manager.
  4. After you identify the list of certificates that expire soon, follow one of the procedures in Rotate Certificates below.

Rotate Certificates

This section includes procedures for rotating CAs and leaf certificates. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated. For more information about identifying the types of certificates in your foundation that expire soon, see Check Expiration Dates and Certificate Types above.

Warning: The rotation procedures described in this topic do not work if the certificates have already expired. If the certificates have expired, contact Support for guidance.

To rotate certificates, follow one of these procedures:

Rotate the Root CA and Leaf Certificates

Warning: The Services CA certificate, /services/tls_ca, cannot be rotated using this procedure. For information about how to rotate the /services/tls_ca certificate, see Rotating the Services TLS CA and Its Leaf Certificates.

This procedure uses the Ops Manager API to rotate the Ops Manager root CA and non-configurable leaf certificates. When you rotate the Ops Manager root CA, the Ops Manager API automatically rotates the BOSH NATS CA.

If you used the Generate RSA Certificate feature to rotate the configurable certificates, this procedure instructs you to rotate the configurable leaf certificates by regenerating them.

To prevent system downtime, this procedure includes two BOSH deploys. The first BOSH deploy applies new certificates to Ops Manager components. The second BOSH deploy deletes the old certificates.

Note: The names of the Upgrade All Service Instances and Recreate All Service Instances errands may be slightly different between services.

Warning: You must complete these steps in the exact order specified. Otherwise, you may damage your deployment.

To rotate the Ops Manager root CA and leaf certificates:

  1. Add a new Ops Manager root CA. For more information, see Add a New Root CA below.

  2. Activate the Ops Manager root CA. For more information, see Activate the Root CA below.

  3. Rotate leaf configurable certificates. For more information, see Rotate Configurable Leaf Certificates below.

  4. Rotate non-configurable leaf certificates. For more information, see Rotate Non-Configurable Leaf Certificates from the New Root CA below.

  5. Delete the old root CA. For more information, see Delete the Old Root CA below.

Step 1: Add a New Root CA

This section describes how to add a new root CA for Ops Manager. The new root CA can be a Pivotal-generated CA or your own custom CA.

This procedure also automatically regenerates the BOSH NATS CA.

Warning: The CAs for certain versions of the MySQL, Redis, RabbitMQ, and Pivotal Cloud Cache tiles are excluded from rotation. For more information about rotating these CAs, see the tile-specific documentation.

To add a new root CA for Ops Manager and regenerate the BOSH NATS CA:

  1. If you have not already done so, follow the procedure in Using Ops Manager API to target and authenticate with the Ops Manager UAA server. Record your Ops Manager access token, and use it for UAA-ACCESS-TOKEN in the following steps.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. Use Ops Manager to generate a new CA, or add your own custom CA:

    Note: Elliptic Curve Digital Signature Algorithm (ECDSA) certificates are not supported in Ops Manager.

    • To use your own custom CA: Call the Ops Manager API certificate_authorities endpoint. Run:

      curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
          -X POST \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
          -H "Content-Type: application/json" \
          -d '{"cert_pem": "-----BEGIN CERTIFICATE-----\CUSTOM-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----\RSA-KEY"}'
      

      Where:

      • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
      • CUSTOM-CERTIFICATE is your custom CA.
      • RSA-KEY is your RSA key.
      • UAA-ACCESS-TOKEN is your UAA access token.

        If the command succeeds, the API returns a response that includes the new CA certificate. For example:
        HTTP/1.1 200 OK
        {
        "certificate_authorities": [
          {
            "guid": "f7bc18f34f2a7a9403c3",
            "issuer": "YOUR-CA",
            "created_on": "2017-01-09",
            "expires_on": "2021-01-09",
            "active": true,
            "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
          }
        ]
        }
        
    • To use a Pivotal-generated CA: Call the Ops Manager API generate endpoint. Run:

      curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/generate" \
            -X POST \
            -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
            -H "Content-Type: application/json" \
            -d '{}'
      

      Where:

      • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.

        If the command succeeds, the API returns a response that includes the new CA certificate. For example:
        HTTP/1.1 200 OK
        {
        "guid": "f7bc18f34f2a7a9403c3",
        "issuer": "Pivotal",
        "created_on": "2017-01-19",
        "expires_on": "2021-01-19",
        "active": false,
        "cert_pem": "-----BEGIN EXAMPLE CERTIFICATE-----
        MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ
        EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa
        EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B
        EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT
        EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt
        EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp
        EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw
        EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd
        EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV
        EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
        EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ
        EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY
        EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ
        EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP
        EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA
        EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84=
          -----END EXAMPLE CERTIFICATE-----
          "
        
  3. Confirm that the new CA is added by listing all the root CAs for Ops Manager by running:

    curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
          -X GET \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN"
    

    Where:

    • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
    • UAA-ACCESS-TOKEN is your UAA access token.

      The API call returns the GUID of the newly added CA. For example:
      HTTP/1.1 200 OK
      {
      "certificate_authorities": [
        {
          "guid": "f7bc18f34f2a7a9403c3",
          "issuer": "Pivotal",
          "created_on": "2017-01-09",
          "expires_on": "2021-01-09",
          "active": true,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
        }
        {
          "guid": "a8ee01e33e3e3e3303e3",
          "issuer": "Pivotal",
          "created_on": "2017-04-09",
          "expires_on": "2021-04-09",
          "active": false,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nzBBBC+eAAAe1gAwAAAeZ....etc"
        }
      ]
      }
      
  4. Identify the newly added CA, which has active set to false. Record its GUID.

  5. Navigate to https://OPS-MANAGER-FQDN in a browser, where OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.

  6. Log in to the Ops Manager Installation Dashboard.

  7. Click the BOSH Director tile.

  8. Select Director Config.

  9. Enable the Recreate All VMs checkbox. This propagates the new CA to all VMs deployed by the BOSH Director to prevent downtime.

  10. Return to the Ops Manager Installation Dashboard.

  11. If you have service tiles installed, for each service tile:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Upgrade All Service Instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
    4. If the service tile has the Recreate All Service Instances errand:
      1. Enable the Recreate All Service Instances errand.
      2. Click Review Pending Changes.
      3. Click Apply Changes.
    5. If the service tile does not have the Recreate All Service Instances errand:

      1. Click Review Pending Changes.
      2. Click Apply Changes.
      3. When the deploy finishes, manually push the BOSH NATS CA to each of its service instances. For each service instance, run:

        bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
        

        Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.

  12. If you do not have any service tiles installed:

    1. Click Review Pending Changes.
    2. Click Apply Changes.
  13. When the deploy finishes and you have completed all necessary tasks to update service instances, continue to the next section, Activate the Root CA.

Step 2: Activate the Root CA

To activate the new root CA:

  1. Make an Ops Manager API call that activates the new CAs. Run:

    curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/CERTIFICATE-GUID/activate" \
      -X POST \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
      -H "Content-Type: application/json" \
      -d '{}'
    

    Where:

    • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
    • CERTIFICATE-GUID is the GUID of your CA that you retrieved in the previous section.
    • UAA-ACCESS-TOKEN is your UAA access token.

      The API returns a successful response:
      HTTP/1.1 200 OK
  2. List your root CAs again to confirm that the new Ops Manager root CA is active. Run:

    curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
          -X GET \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN"
    

    Where:

    • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
    • UAA-ACCESS-TOKEN is your UAA access token.
  3. Examine the response to ensure that your new CA has the active property set to true.

Step 3: Rotate Configurable Leaf Certificates

Any configurable certificates generated from the Ops Manager root CA must be rotated. These are the certificates that you generated using Generate RSA Certificate in the Ops Manager UI. For example, see Providing a Certificate for Your TLS Termination Point.

To regenerate and rotate configurable leaf certificates, do the following for each configurable certificate that expires soon:

  1. Find the text field where the certificate is configured within the Ops Manager UI. You might need to look through multiple configuration panes to identify the location of the certificate configuration text field. Use the following fields to identify the location of the certificate configuration text field:

    • The product_guid field in the Ops Manager API output can help identify the tile in which the certificate is configured. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the TAS for VMs tile.
    • The property_reference field in the Ops Manager API output can help identify the configuration pane in which the certificate is configured. For example, the uaa.service_provider_key_credentials property is configured in the UAA pane of the TAS for VMs tile.
  2. Copy a new value for the certificate into the text field or generate a new certificate using the Generate RSA Certificate button.

  3. Click Save at the bottom of each pane in which you added new certificates.

Step 4: Rotate Non-Configurable Leaf Certificates from the New Root CA

After activating the new Ops Manager root CA, you must rotate non-configurable leaf certificates from the new root CA.

To rotate non-configurable leaf certificates from the new root CA:

  1. Make a call to the Ops Manager API to regenerate all non-configurable certificates. Run:

      curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/active/regenerate" \
            -X POST \
            -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
            -H "Content-Type: application/json" \
            -d '{}'
    

    Where:

    • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
    • UAA-ACCESS-TOKEN is your UAA access token.

      The API returns a successful response:
      HTTP/1.1 200 OK
  2. Navigate to the Ops Manager Installation Dashboard.

  3. Click the BOSH Director tile.

  4. Select Director Config.

  5. Enable the Recreate All VMs checkbox. This propagates the new CAs to all VMs deployed by the BOSH Director to prevent downtime.

  6. Return to the Ops Manager Installation Dashboard.

  7. If you have service tiles installed, for each service tile:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Upgrade All Service Instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
    4. If the service tile has the Recreate All Service Instances errand:
      1. Enable the Recreate All Service Instances errand.
      2. Click Review Pending Changes.
      3. Click Apply Changes.
    5. If the service tile does not have the Recreate All Service Instances errand:

      1. Click Review Pending Changes.
      2. Click Apply Changes.
      3. When the deploy finishes, manually push the BOSH NATS CA to each of its service instances. For each service instance, run:

        bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
        

        Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.

  8. If you do not have any service tiles installed:

    1. Click Review Pending Changes.
    2. Click Apply Changes.
  9. When the deploy finishes and you have completed all necessary tasks to update service instances, continue to the next section, Delete the Old Root CA.

Step 5: Delete the Old Root CA

After activating new CAs and rotating leaf certificates from the new CAs, you should delete the old Ops Manager root CA from your foundation.

If your old root CA or leaf certificates have been compromised, you must delete the old root CA immediately. Otherwise, the old root CA and its leaf certificates may still be trusted, which compromises your foundation’s security. If your old root CA or leaf certificates have not been compromised, VMware recommends that you delete the old root CA as soon as possible, but you can also delete it at a later time, such as in conjunction with another deploy.

Warning: Before you delete the old CA, ensure that you reviewed the Review Pending Changes page and clicked Apply Changes as part of Rotate Non-Configurable Leaf Certificates from the New Root CA above.

To delete the old Ops Manager root CA:

  1. List your root CAs to retrieve the GUID of the old, inactive Ops Manager root CA. Run:

    curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities" \
          -X GET \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN"
    

    Where:

    • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
    • UAA-ACCESS-TOKEN is your UAA access token.
  2. Use curl to make an API call to delete the old Ops Manager root CA. Run:

    curl "https://OPS-MANAGER-FQDN/api/v0/certificate_authorities/OLD-CERTIFICATE-GUID" \
          -X DELETE \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN"
    

    Where:

    • OPS-MANAGER-FQDN is the FQDN of your Ops Manager deployment.
    • OLD-CERTIFICATE-GUID is the GUID of the old Ops Manager root CA.
    • UAA-ACCESS-TOKEN is your UAA access token.

      The API returns a successful response:
      HTTP/1.1 200 OK
  3. Navigate to the Ops Manager Installation Dashboard.

  4. Click the BOSH Director tile.

  5. Select Director Config.

  6. Enable the Recreate All VMs checkbox.

  7. Return to the Ops Manager Installation Dashboard.

  8. If you have service tiles installed, for each service tile:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Upgrade All Service Instances errand. Running this errand is necessary to push CredHub certificate updates to each service instance.
    4. If the service tile has the Recreate All Service Instances errand:
      1. Enable the Recreate All Service Instances errand. Running this errand pushes BOSH agent certificate updates to service instances.
      2. Click Review Pending Changes.
      3. Click Apply Changes.
    5. If the service tile does not have the Recreate All Service Instances errand:

      1. Click Review Pending Changes.
      2. Click Apply Changes.
      3. When the deploy finishes, manually push BOSH agent certificate updates to each service instance. For each service instance, run:

        bosh -d SERVICE-INSTANCE-DEPLOYMENT recreate
        

        Where SERVICE-INSTANCE-DEPLOYMENT is the BOSH deployment name for the service instance.

  9. If you do not have any service tiles installed:

    1. Click Review Pending Changes.
    2. Click Apply Changes.