Rotating Certificates

Page last updated:

This topic describes how to check the expiration date of and rotate Certificate Authorities (CAs) and leaf certificates in Pivotal Platform.

This includes the Pivotal Ops Manager root CA, BOSH NATS CA, and other CAs and leaf certificates stored in Ops Manager that are visible to the Ops Manager API.

Warning: The rotation procedures described in this topic do not work if the certificates have already expired. If the certificates have expired, contact Pivotal Support for guidance.

For information about rotating IPsec certificates, see Rotating IPsec Certificates.

For information about using trusted third-party certificates for both apps hosted on Pivotal Platform and internal Pivotal Platform components, see Setting Trusted Certificates.

For information about how to rotate certificates managed by CredHub and the Services TLS CA, see Advanced Certificate Rotation with CredHub Maestro.

Overview

The Ops Manager API manages and lists internal Certificate Authorities (CAs) and leaf certificates that enable Pivotal Platform components to communicate with each other securely using TLS. It can also list certificates used externally, such as SAML certificates that authenticate to an external identity provider.

For more information about the CAs and leaf certificates visible to the Ops Manager API, see Certificate Types.

Rotate CAs and leaf certificates before they expire to avoid downtime for your foundation. To rotate certificates in Pivotal Platform, first check the expiration dates of all certificates. Then, based on the types of certificates that expire soon, follow a certificate rotation procedure to replace expiring certificates and redeploy BOSH to apply changes.

Check Expiration Dates and Certificate Types

This section describes how to manually check the expiration dates of the CAs and leaf certificates that the Ops Manager API lists and manages. It also explains how to identify the types of certificates that require manual rotation.

To configure Concourse to automatically monitor expiring certificates, you can use Platform Automation. For more information, see expiring-certificates in the Platform Automation documentation.

After identifying the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

To check certificate expiration dates and types:

  1. Perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the steps below.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. To retrieve the certificates, use curl to call the /deployed/certificates endpoint of the Ops Manager API:

    curl "https://OPS-MAN-FQDN/api/v0/deployed/certificates" \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    

    Where:

    • OPS-MAN-FQDN is the URL of your Ops Manager.
    • YOUR-UAA-ACCESS-TOKEN is the access_token value you recorded in the previous step:

    More options:

    • To limit command output to certificates that expire within a given time interval, append ?expires_within=TIME to the endpoint, replacing TIME with an integer-letter code.
      • Valid letter codes are d for days, w for weeks, m for months, and y for years.
      • For example, querying to https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=6m searches for certificates expiring within six months.
    • To make the JSON output more readable, you can pipe your curl command to jq or another text editor with JSON formatting.

  3. The deployed/certificates output lists all CAs and leaf certificates visible to the Ops Manager API. For each certificate listed, determine its expiration date and type.

    1. Determine the expiration date from the valid_until value. For example, the root CA listed below expires on August 12, 2020:
      {
      "configurable": true,
      "is_ca": true,
      "property_reference": ".properties.root_ca.fb10484dd5541a273c9d",
      "property_type": "rsa_cert_credentials",
      "product_guid": "ops_manager",
      "location": "ops_manager",
      "variable_path": null,
      "issuer": "/CN=ToolsmithsCA",
      "valid_from": "2019-08-13T15:30:22Z",
      "valid_until": "2020-08-12T15:30:22Z"
      },
      
    2. For any certificates expiring soon, use the following rules to identify their types:
      • Non-Rotatable Certificates: Non-Rotatable certificates have the following property value:
        • variable_path is /opsmgr/bosh_dns/tls_ca
      • Non-Configurable Certificates: Non-Configurable leaf certificates have the following property values:
        • variable_path is not /opsmgr/bosh_dns/tls_ca
        • configurable is false
        • location is ops_manager
        • is_ca is false
      • Configurable Certificates: Configurable leaf certificates have the following property values:
        • configurable is true
        • is_ca is false
      • CAs: CAs have the following property values:
        • is_ca is true
        • location is ops_manager
  4. After you identify the list of certificates that expire soon, complete the Rotate Certificates procedure below.

Rotate Certificates

This section explains how to rotate certificates in Pivotal Platform, including the Ops Manager root CA, BOSH NATS CA, and leaf certificates. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated.

To rotate certificates, do one of the following procedures:

  • To rotate non-rotatable certificates, call Pivotal Support.

  • To rotate CAs stored in Ops Manager, follow the Rotate CAs and Leaf Certificates procedure. This procedure also rotates all non-configurable leaf certificates and the BOSH NATS CA. You can also optionally rotate configurable leaf certificates as part of this procedure.

  • To rotate non-configurable leaf certificates, but not the Ops Manager root CA or BOSH NATS CA, follow the Rotate Non-Configurable Leaf Certificates procedure.

  • To rotate configurable leaf certificates, but not the Ops Manager root CA or BOSH NATS CA, follow the Rotate Configurable Leaf Certificates procedure.

Rotate CAs and Leaf Certificates

This procedure uses the Ops Manager API to rotate the Ops Manager root CA and non-configurable leaf certificates. When you rotate the Ops Manager root CA, the Ops Manager API also automatically rotates the BOSH NATS CA.

To prevent system downtime, this procedure includes two BOSH deploys. The first BOSH deploy applies new certificates to Pivotal Platform components. The second BOSH deploy deletes the old certificates.

Warning: You must complete these steps in the exact order specified. Otherwise, you may damage your deployment.

To rotate CAs and non-configurable leaf certificates:

  1. Add a new Ops Manager root CA. This step also automatically adds new CAs for all CAs visible to the Ops Manager API. See Add New CAs.

  2. Activate the Ops Manager root CA. This step also automatically activates any other new CAs added in the previous step. See Activate the CAs.

  3. (Optional) Rotate configurable leaf certificates. See Rotate Configurable Leaf Certificates.

  4. Rotate non-configurable leaf certificates. See Rotate Non-Configurable Leaf Certificates from the New CAs.

  5. Delete the old CA. See Delete the Old CA.

Step 1: Add New CAs

Follow this procedure to add a new root CA for Ops Manager. The new root CA can be a Pivotal-generated CA or your own custom CA.

This procedure also automatically regenerates the BOSH NATS CA.

To add new CAs:

  1. If you haven’t already, perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the following steps.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. Use Ops Manager to generate a new CA, or add your own custom CA:

    Note: Elliptic Curve Digital Signature Algorithm (ECDSA) certificates are not supported in Pivotal Platform.

    • To use your own custom CA: Call the Ops Manager API certificate_authorities endpoint. Run:
      curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"cert_pem": "-----BEGIN CERTIFICATE-----\YOUR-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----\YOUR-KEY"}'
      
      Where:
      • YOUR-CERTIFICATE is your custom CA.
      • YOUR-KEY is your RSA key.
      • YOUR-UAA-ACCESS-TOKEN is your UAA access token.

      If the command succeeds, the API returns a response that includes the new CA certificate:

      HTTP/1.1 200 OK
      {
      "certificate_authorities": [
        {
          "guid": "f7bc18f34f2a7a9403c3",
          "issuer": "YOUR-CA",
          "created_on": "2017-01-09",
          "expires_on": "2021-01-09",
          "active": true,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
        }
      ]
      }
      
    • To use a Pivotal-generated CA: Call the Ops Manager API generate endpoint. Run:
      curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/generate" \
          -X POST \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
          -H "Content-Type: application/json" \
          -d '{}'
      
      Where YOUR-UAA-ACCESS-TOKEN is your UAA access token.

      If the command succeeds, the API returns a response that includes the new CA certificate:
        HTTP/1.1 200 OK
      {
        "guid": "f7bc18f34f2a7a9403c3",
        "issuer": "Pivotal",
        "created_on": "2017-01-19",
        "expires_on": "2021-01-19",
        "active": false,
        "cert_pem": "-----BEGIN EXAMPLE CERTIFICATE-----
        MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ
        EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa
        EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B
        EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT
        EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt
        EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp
        EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw
        EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd
        EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV
        EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
        EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ
        EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY
        EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ
        EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP
        EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA
        EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84=
          -----END EXAMPLE CERTIFICATE-----
          "
      
  3. Confirm that the new CA is added by listing all the root CAs for Ops Manager. Run:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    The API call returns the GUID of the newly added CA. For example:
    HTTP/1.1 200 OK
    {
      "certificate_authorities": [
        {
          "guid": "f7bc18f34f2a7a9403c3",
          "issuer": "Pivotal",
          "created_on": "2017-01-09",
          "expires_on": "2021-01-09",
          "active": true,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
        }
        {
          "guid": "a8ee01e33e3e3e3303e3",
          "issuer": "Pivotal",
          "created_on": "2017-04-09",
          "expires_on": "2021-04-09",
          "active": false,
          "cert_pem": "-----BEGIN CERTIFICATE-----\zBBBC+eAAAe1gAwAAAeZ....etc"
        }
      ]
    }
    

  4. Identify the newly added CA, which has active set to false. Record its GUID.

  5. Navigate to https://OPS-MAN-FQDN in a browser and log in to Ops Manager.

  6. Click the BOSH Director tile in the Installation Dashboard.

  7. Select the Director Config pane.

  8. Select Recreate All VMs. This propagates the new CA to all VMs to prevent downtime.

  9. Go back to the Installation Dashboard. For each service tile you have installed:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Recreate All Service Instances errand if provided. If you do not see this errand, contact Pivotal Support.
  10. Click Review Pending Changes, then Apply Changes. When the deploy finishes, continue to the next section.

Step 2: Activate the CAs

This procedure activates the CAs that you added in Step 1: Add New CAs.

To activate the new CAs:

  1. Make an Ops Manager API call that activates the new CAs. Run:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/CERT-GUID/activate" \
      -X POST \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
      -H "Content-Type: application/json" \
      -d '{}'
    
    Where CERT-GUID is the GUID of your CA that you retrieved in the previous section.

    The API returns a successful response:
    HTTP/1.1 200 OK

  2. List your CAs to confirm that the new Ops Manager root CA is active. Run:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Examine the response to ensure that your new CA has active set to true.

(Optional) Step 3: Rotate Configurable Leaf Certificates

If there are any configurable certificates in your foundation expiring soon, Pivotal recommends that you rotate them now. Rotating configurable leaf certificates now ensures that your next BOSH deploy includes both new configurable and non-configurable leaf certificates.

To rotate configurable leaf certificates, do the following for each configurable certificate that expires soon:

  1. Find the text field where the certificate is configured within the Ops Manager interface. You might have to look through multiple configuration panes to identify the location of the certificate configuration text field. Use the following fields to identify the location of the certificate configuration text field:

    • The product_guid field in the API output can help identify in which tile the certificate is configured. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the Pivotal Platform tile.
    • The property_reference field in the API output can help identify in which Settings pane the certificate is configured. For example, the uaa.service_provider_key_credentials property is configured in the UAA pane of the Pivotal Platform tile.
  2. Paste a new value for the certificate into the text field.

  3. Click Save at the bottom of each pane in which you added new certificates.

Step 4: Rotate Non-Configurable Leaf Certificates from the New CAs

After activating the new Ops Manager root CA and other CAs, you must rotate non-configurable leaf certificates from the new CAs.

To rotate non-configurable leaf certificates:

  1. Make an API call to regenerate all non-configurable certificates and apply the new root CA to your existing BOSH Director. Run:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{}'
    
    The API returns a successful response:
    HTTP/1.1 200 OK

  2. Click the BOSH Director tile in the Installation Dashboard.

  3. Select the Director Config pane.

  4. Select Recreate All VMs. This propagates the new CAs to all VMs to prevent downtime.

  5. Go back to the Installation Dashboard. For each service tile you have installed:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Recreate All Service Instances errand if provided. If you do not see this errand, contact Pivotal Support.
  6. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes to perform a second redeploy.

Step 5: Delete the Old CAs

After activating new CAs and rotating leaf certificates from the new CAs, Pivotal recommends that you delete the old Ops Manager root CA from your foundation.

Note: To delete CredHub certificates, use the Maestro garbage collection feature. For information about Maestro certificate garbage collection, see the Delete Inactive Certificate Versions section of the Advanced Certificate Rotation with CredHub Maestro topic.

Warning: Ensure that you reviewed the Review Pending Changes page and clicked Apply Changes as part of Step 4: Rotate Non-Configurable Leaf Certificates from the New CAs before you delete the old CA.

To delete the old Ops Manager root CA:

  1. List your root CAs to retrieve the GUID of the old, inactive Ops Manager root CA. Run:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    

  2. Use curl to make an API call to delete the old Ops Manager root CA. Run:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/OLD-CERT-GUID" \
        -X DELETE \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Where OLD-CERT-GUID is the GUID of the old Ops Manager root CA.

    The API returns a successful response.
    HTTP/1.1 200 OK

  3. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes.

Rotate Non-Configurable Leaf Certificates

This procedure rotates non-configurable leaf certificates visible to the Ops Manager API.

Warning: This procedure does not rotate the Ops Manager root CA. To rotate both the root CA and non-configurable leaf certificates, see Rotate CAs and Leaf Certificates.

To rotate non-configurable leaf certificates:

  1. Use curl to make an API call to regenerate all non-configurable certificates and apply the new CA to your existing BOSH Director:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{}'
    
    The API returns a successful response:
    HTTP/1.1 200 OK

  2. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes to perform a second redeploy.

Rotate Configurable Leaf Certificates

Configurable certificates are generated by the user and pasted into Ops Manager configuration panes where needed. Examples include certificates that terminate SSL traffic into Pivotal Platform, or authenticate a Single Sign-On (SSO) service plan to an external SAML server.

For instructions on how to rotate SAML certificates for both Pivotal Platform and the SSO service, see Rotate Identity Provider SAML Certificates.

Warning: This procedure does not rotate the Ops Manager root CA. To rotate both the root CA and non-configurable leaf certificates, see Rotate CAs and Leaf Certificates.

To rotate configurable leaf certificates:

  1. If you have not already done so, use the Ops Manager API deployed/certificates endpoint to retrieve information about your expiring configurable certificates, as described in the procedures Check Expiration Dates and Certificate Types. The information for each configurable certificate looks like this:

    {
      "configurable": true,
      "property_reference": ".properties.networking_poe_ssl_certs[0].certificate",
      "property_type": "rsa_cert_credentials",
      "product_guid": "cf-36066831c119c39736d3",
      ...
      "valid_until": "2019-01-25T22:07:58Z"
    },
    

  2. For each configurable certificate that expires soon:

    1. Find the text field the certificate is configured within the Ops Manager interface.
      • The product_guid field in the API output can help identify which tile the certificate is configured in. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the Pivotal Platform tile.
      • The property_reference field in the API output can often help identify which Settings pane the certificate is configured in. For example, the uaa.service_provider_key_credentials property is configured in the UAA pane of the Pivotal Platform tile.
      • You might have to look through multiple configuration panes to identify where a certificate is configured.
    2. Paste a new value for the certificate into the field.
    3. Click Save at the bottom of each pane in which you have provided new certificates.
  3. If you are rotating configurable certificates within the Rotate CAs and Leaf Certificates procedure, continue to the next step. Otherwise, if you are rotating configurable leaf certificates only, return to the Installation Dashboard, click Review Pending Changes, and click Apply Changes.

Rotate Identity Provider SAML Certificates

SAML service provider credentials are one example of configurable certificates in Pivotal Platform. When Pivotal Platform is configured to use SAML as an identity provider, it uses a configurable CA certificate to authenticate to an external SAML server, by generating ephemeral certificates that Pivotal Platform includes in its outbound request message headers. This CA has a two-year expiration period.

In addition, the SSO service shares the use of Pivotal Platform SAML certificates for every SAML external Identity Provider (IdP) integration, such as trust, partnership, or Federation. You must rotate these in lockstep with Pivotal Platform.

The following procedure provides an example of how to rotate certificates for each IdP, including temporarily disabling certificate validation on the IdP side during the rotation.

The Knowledge Base article Pivotal Platform Advisory - SAML Service Provider Credential Certificates Expire after 2 Years provides more information about rotating SAML certificates.

SAML service provider credentials are only required for your Pivotal Platform deployment if all of these conditions are met:

  • You are using SSO in production for login to Pivotal Platform or using the SSO service for login to apps.
  • You are using SAML identity providers for Pivotal Platform or SSO service plans.
  • You had Ops Manager generate a certificate for you by clicking the Generate RSA Certificate button.
  • You are validating the signature of SAML authentication request with your identity provider.

To regenerate and rotate SAML service provider certificates without disrupting Pivotal Platform or your apps using the SSO service:

  1. Disable certificate validation in your identity provider.

  2. For Pivotal Platform, follow the procedure in the table below that corresponds to your use case. This includes downloading and importing a new certificate and updated SAML metadata in your identity provider.

    Solution Name Procedure
    CA Single Sign-On aka CA SiteMinder Configuring CA as an Identity Provider
    PingFederate Configuring PingFederate as an Identity Provider
    Active Directory Federation Services Configuring ADFS as an Identity Provider

  3. For the SSO service, follow the procedure in the table below that corresponds to your use case. This includes downloading the SAML Service Provider metadata for each SAML identity provider integration, such as trust, partnership, or Federation, and importing the updated SAML Service Provider metadata in your identity provider.

    Solution Name Procedure
    ADFS Configuring a Single Sign-On Service Provider
    CA SSO Configuring a Single Sign-On Service Provider
    Okta Configure Okta as an Identity Provider
    PingFederate Configure PingFederate as an Identity Provider
    Additional Documentation Integration Guides

  4. Re-enable certificate validation in your identity provider.