Rotating Certificates

This topic describes how to check the expiration date of and rotate the root Certificate Authorities (CAs) and leaf certificates in Pivotal Cloud Foundry (PCF) that are visible to the Ops Manager API.

Note: The rotation procedures described in this topic do not work if the certificates have already expired. If the certificates have expired, contact Pivotal Support for guidance.

For information about rotating IPsec certificates, see Rotating IPsec Certificates.

For information about using trusted third-party certificates for both apps hosted on PCF and internal PCF components, see Setting Trusted Certificates.

Overview

The Ops Manager API manages and lists internal Certificate Authorities (CAs) and leaf certificates that enable PCF components to communicate with each other securely using TLS. It can also list certificates used externally, such as SAML certificates that authenticate to an external identity provider.

For more information about root CAs and leaf certificates, see Certificate Types.

Rotate root CAs and leaf certificates before they expire to avoid downtime for your foundation.

Procedure

This section provides an overview of the procedure for rotating the root CA and leaf certificates in PCF.

To rotate certificates in PCF, you first check the expiration dates of all certificates. Then, based on the types of certificates that expire soon, you follow a certificate rotation procedure to replace expiring certificates and redeploy BOSH to apply changes.

To rotate root CAs and leaf certificates in PCF, do the following:

  1. Check the expiration dates of the root CA and leaf certificates, and identify the types of leaf certificates that require rotation. See Check Expiration Dates and Certificate Types.

  2. Rotate the certificates that expire soon. See Rotate Certificates.

Check Expiration Dates and Certificate Types

This section describes how to check the expiration dates of the root CA and leaf certificates that the Ops Manager API lists and manages. It also explains how to identify the types of leaf certificates that expire soon and require manual rotation.

After identifying the types of certificates that expire soon, you can determine which certificate rotation procedure to follow.

Check Root CA Expiration Date

This procedure describes how to check the expiration date for the Ops Manager root CA. The Ops Manager root CA expires four years after creation.

To check the Ops Manager root CA expiration date, do the following:

  1. Perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the steps below.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. To retrieve the Ops Manager root CA, use curl to make an Ops Manager API call to the https://OPS-MAN-FQDN/api/v0/certificate_authorities endpoint. For example:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    

  3. To make the JSON output more readable, you can pipe it to jq or another text editor with JSON formatting.

  4. In the certificate_authorities list returned, if there is more than one, find the CA with "active": true.

  5. To determine its expiration date of the active CA, refer to its expires_on value. For example, the root CA shown below expires on September 5, 2019:

    {
    "certificate_authorities": [
    {
      "guid": "9c9a110c8f82a1e4aaca",
      "issuer": "Pivotal",
      "created_on": "2017-09-05T22:47:53Z",
      "expires_on": "2019-09-05T22:47:53Z",
      "active": true,
      "cert_pem": "-----BEGIN CERTIFICATE-----\
      [...]
      \n-----END CERTIFICATE-----\n"
    

Check Leaf Certificate Expiration Dates and Types

This procedure describes how to check the expiration dates of leaf certificates using the deployed/certificates endpoint. It also describes how to identify non-configurable, configurable, and non-rotatable leaf certificates in the output from the deployed/certificates endpoint.

To check the expiration dates of leaf certificates and identify the types of leaf certificates that expire soon, do the following:

  1. If you haven’t already, perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the steps below.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. To check the system for certificates that expire within a given time interval, use curl to call the https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=TIME API endpoint, replacing TIME with an integer-letter code. Valid letter codes are d for days, w for weeks, m for months, and y for years.

    For example, the following command searches for certificates expiring within six months:

    curl "https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=6m" \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Where YOUR-UAA-ACCESS-TOKEN is the access_token value you recorded in the previous step.

  3. To make the JSON output more readable, pipe it to jq or another text editor with JSON formatting.

  4. To determine the expiration date of each certificate, refer to its expires_on value.

  5. Identify the types of leaf certificates requiring rotation. The following rules identify the type of each leaf certificate:

    • Non-Rotatable Certificates: Non-Rotatable leaf certificates have the following property value: variable_path is /opsmgr/bosh_dns/tls_ca
    • Non-Configurable Certificates: Non-Configurable leaf certificates have the following property values and are not non-rotatable as identified above:
      • configurable is false
      • location is either ops_manager or credhub
    • Configurable Certificates: Configurable leaf certificates have the following property value: configurable is true

Rotate Certificates

This section includes procedures for rotating root CAs and leaf certificates. There are different rotation procedures for each type of certificate that requires rotation.

You can determine which rotation procedure to follow based on the types of certificates in your foundation that must be rotated. For more information about identifying the types of certificates in your foundation that expire soon, see Check Expiration Dates and Certificate Types.

To rotate certificates, do one of the following procedures:

Rotate the Root CA and Leaf Certificates

This procedure uses the Ops Manager API to rotate the Ops Manager root CA and non-configurable leaf certificates. Rotating the Ops Manager root CA automatically rotates all non-configurable leaf certificates.

To prevent system downtime, this procedure includes two BOSH deploys. The first BOSH deploy applies new certificates to PCF components. The second BOSH deploy deletes the old certificates.

Warning: You must complete these steps in the exact order specified. Otherwise, you may damage your deployment.

To rotate the root CA and leaf certificates, do the following:

  1. Add a new root CA. See Add a New Root CA.

  2. Activate the root CA. Activate the Root CA.

  3. (Optional) Rotate leaf configurable certificates. See Rotate Configurable Leaf Certificates.

  4. Rotate non-configurable leaf certificates. See Rotate Non-Configurable Leaf Certificates from the New Root.

  5. (Optional) Delete the old CA. See Delete the Old CA.

Step 1: Add a New Root CA

Follow this procedure to add a new root CA for Ops Manager. The new root CA can be a Pivotal-generated CA or your own custom CA.

Note: This procedure regenerates the NATS CA in unison with the Ops Manager root CA.

To add a new root CA for Ops Manager, do the following:

  1. If you haven’t already, perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the following steps.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. Use Ops Manager to generate a new CA, or else add your own custom CA:

    Note: Elliptic Curve Digital Signature Algorithm (ECDSA) certificates are not supported in PCF.

    • To use your own custom CA, call the Ops Manager API generate endpoint as follows:
      curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{"cert_pem": "-----BEGIN CERTIFICATE-----\YOUR-CERTIFICATE", "private_key_pem": "-----BEGIN RSA PRIVATE KEY-----\YOUR-KEY"}'
      
      Where:
      • YOUR-CERTIFICATE is your custom CA.
      • YOUR-KEY is your RSA key.
      • YOUR-UAA-ACCESS-TOKEN is your UAA access token.

      If the command succeeds, the API returns a response that includes the new CA certificate:

      HTTP/1.1 200 OK
      {
      "certificate_authorities": [
        {
          "guid": "f7bc18f34f2a7a9403c3",
          "issuer": "YOUR-CA",
          "created_on": "2017-01-09",
          "expires_on": "2021-01-09",
          "active": true,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
        }
      ]
      }
      
    • To use a Pivotal-generated CA, call the Ops Manager API generate endpoint as follows:
      curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/generate" \
          -X POST \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
          -H "Content-Type: application/json" \
          -d '{}'
      
      Where YOUR-UAA-ACCESS-TOKEN is your UAA access token.

      If the command succeeds, the API returns a response that includes the new CA certificate:
        HTTP/1.1 200 OK
      {
        "guid": "f7bc18f34f2a7a9403c3",
        "issuer": "Pivotal",
        "created_on": "2017-01-19",
        "expires_on": "2021-01-19",
        "active": false,
        "cert_pem": "-----BEGIN EXAMPLE CERTIFICATE-----
        MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ
        EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa
        EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B
        EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT
        EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt
        EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp
        EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw
        EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd
        EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV
        EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
        EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ
        EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY
        EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ
        EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP
        EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA
        EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84=
          -----END EXAMPLE CERTIFICATE-----
          "
      
  3. Confirm that your new CA has been added by listing all of the root CAs for Ops Manager:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    The API call returns the GUID of the newly added CA. For example:
    HTTP/1.1 200 OK
    {
      "certificate_authorities": [
        {
          "guid": "f7bc18f34f2a7a9403c3",
          "issuer": "Pivotal",
          "created_on": "2017-01-09",
          "expires_on": "2021-01-09",
          "active": true,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
        }
        {
          "guid": "a8ee01e33e3e3e3303e3",
          "issuer": "Pivotal",
          "created_on": "2017-04-09",
          "expires_on": "2021-04-09",
          "active": false,
          "cert_pem": "-----BEGIN CERTIFICATE-----\zBBBC+eAAAe1gAwAAAeZ....etc"
        }
      ]
    }
    
    Identify your newly added CA, which has active set to false. Record its GUID.

  4. Navigate to https://OPS-MAN-FQDN in a browser and log in to Ops Manager.

  5. Click the BOSH Director tile in the Installation Dashboard.

  6. Select the Director Config pane.

  7. Select Recreate All VMs. This propagates the new CA to all VMs to prevent downtime.

  8. Go back to the Installation Dashboard. For each service tile you have installed, do the following:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Recreate All Service Instances errand if provided. If you do not see this errand, contact Pivotal Support.
  9. Click Review Pending Changes, then Apply Changes. When the deploy finishes, continue to the next section.

Step 2: Activate the Root CA

To activate the new root CA, do the following:

  1. Use curl to make an Ops Manager API call that activates the new CA:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/CERT-GUID/activate" \
      -X POST \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
      -H "Content-Type: application/json" \
      -d '{}'
    
    Where CERT-GUID is the GUID of your CA that you retrieved in the previous section.

    The API returns a successful response:
    HTTP/1.1 200 OK

  2. List your root CAs again to confirm that the new CA is active:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Examine the response to ensure that your new CA has active set to true.

(Optional) Step 3: Rotate Configurable Leaf Certificates

If there are any configurable certificates in your foundation expiring soon, Pivotal recommends rotating them now. Rotating configurable leaf certificates now ensures that your next BOSH deploy includes both new configurable and non-configurable leaf certificates.

To rotate configurable leaf certificates, do the following for each configurable certificate that expires soon:

  1. Find the text field where the certificate is configured within the Ops Manager interface. You might have to look through multiple configuration panes to identify the location of the certificate configuration text field. Use the following fields to identify the location of the certificate configuration text field:
    • The product_guid field in the API output can help identify in which tile the certificate is configured. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the PAS tile.
    • The property_reference field in the API output can help identify in which Settings pane the certificate is configured. For example, the uaa.service_provider_key_credentials property is configured in the PAS tile > UAA pane.
  2. Paste a new value for the certificate into the text field.
  3. Click Save at the bottom of each pane in which you added new certificates.

Step 4: Rotate Non-Configurable Leaf Certificates from the New Root

After activating the new root CA, you must rotate non-configurable leaf certificates from the root CA.

To rotate non-configurable leaf certificates from the new root CA, do the following:

  1. Use curl to make an API call to regenerate all non-configurable certificates and apply the new CA to your existing BOSH Director:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{}'
    
    The API returns a successful response:
    HTTP/1.1 200 OK

  2. Click the BOSH Director tile in the Installation Dashboard.

  3. Select the Director Config pane.

  4. Select Recreate All VMs. This propagates the new CA to all VMs to prevent downtime.

  5. Go back to the Installation Dashboard. For each service tile you have installed, do the following:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Recreate All Service Instances errand if provided. If you do not see this errand, contact Pivotal Support.
  6. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes to perform a second redeploy.

(Optional) Step 5: Delete the Old CA

After activating a new root CA and rotating leaf certificates from the new root CA, Pivotal recommends deleting the old root CA from your foundation.

Warning: Include the Review Pending Changes and Apply Changes from Step 4: Rotate Non-Configurable Leaf Certificates from the New Root before you delete the old CA.

To delete the old root CA, do the following:

  1. List your root CAs to retrieve the GUID of your old, inactive CA:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    

  2. Use curl to make an API call to delete your old CA:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/OLD-CERT-GUID" \
        -X DELETE \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Where OLD-CERT-GUID is the GUID of your old, inactive CA.

    The API returns a successful response.
    HTTP/1.1 200 OK

  3. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes.

Rotate Non-Configurable Leaf Certificates

This procedure rotates non-configurable leaf certificates visible to the Ops Manager API, whether they are managed and stored by Ops Manager directly, or by CredHub at Ops Manager request.

Warning: This procedure does not rotate the Ops Manager root CA. To rotate both the root CA and non-configurable leaf certificates, see Rotate the Root CA and Leaf Certificates.

To rotate non-configurable leaf certificates, do the following:

  1. Use curl to make an API call to regenerate all non-configurable certificates and apply the new CA to your existing BOSH Director:

    curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{}'
    
    The API returns a successful response:
    HTTP/1.1 200 OK

  2. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes to perform a second redeploy.

Rotate Configurable Leaf Certificates

Configurable certificates are generated by the user and pasted into Ops Manager configuration panes where needed. Examples include certificates that terminate SSL traffic into PAS, or authenticate a Single Sign-On (SSO) service plan to an external SAML server.

For instructions on how to rotate SAML certificates for both PAS and the SSO service, see Rotate Identity Provider SAML Certificates.

Warning: This procedure does not rotate the Ops Manager root CA. To rotate both the root CA and non-configurable leaf certificates, see [Rotate the Root CA and Leaf Certificates](#rotate-ca).

To rotate configurable leaf certificates, do the following:

  1. If you haven’t already, use the Ops Manager API deployed/certificates endpoint to retrieve information about your expiring configurable certificates, as described in the procedures Check Expiration Dates and Certificate Types.
    The information for each configurable certificate looks like this:

    {
      "configurable": true,
      "property_reference": ".properties.networking_poe_ssl_certs[0].certificate",
      "property_type": "rsa_cert_credentials",
      "product_guid": "cf-36066831c119c39736d3",
      ...
      "valid_until": "2019-01-25T22:07:58Z"
    },
    

  2. For each configurable certificate that expires soon:

    1. Find the text field the certificate is configured within the Ops Manager interface.
      • The product_guid field in the API output can help identify which tile the certificate is configured in. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the PAS tile.
      • The property_reference field in the API output can often help identify which Settings pane the certificate is configured in. For example, the uaa.service_provider_key_credentials property is configured in the PAS tile > UAA pane.
      • You might have to look through multiple configuration panes to identify where a certificate is configured.
    2. Paste a new value for the certificate into the field
    3. Click Save at the bottom of each pane in which you have provided new certificates.
  3. If you are rotating configurable certificates within the Rotate Root and Leaf Certificates procedure, continue to the next step. Otherwise, if you are rotating configurable leaf certificates only, return to the Installation Dashboard, click Review Pending Changes, and click Apply Changes.

Rotate Identity Provider SAML Certificates

SAML service provider credentials are one example of configurable certificates in PCF. When PAS is configured to use SAML as an identity provider, it uses a configurable CA certificate to authenticate to an external SAML server, by generating ephemeral certificates that PAS includes in its outbound request message headers. This CA has a two-year expiration period.

In addition, the SSO service shares the use of PAS SAML certificates for every SAML external Identity Provider (IdP) integration, such as trust, partnership, or Federation. You must rotate these in lockstep with PAS.

The following procedure provides an example of how to rotate certificates for each IdP, including temporarily disabling certificate validation on the IdP side during the rotation.

The Knowledge Base article PCF Advisory - SAML Service Provider Credential Certificates Expire after 2 Years provides more information about rotating SAML certificates.

SAML service provider credentials are only required for your PAS deployment if all of these conditions are met:

  • You are using SSO in production for login to PAS or using the SSO service for login to apps.
  • You are using SAML identity providers for PAS or SSO service plans.
  • You had Ops Manager generate a certificate for you by clicking the Generate RSA Certificate button.
  • You are validating the signature of SAML authentication request with your identity provider.

To regenerate and rotate SAML service provider certificates without disrupting PAS or your apps using the SSO service, do the following:

  1. Disable certificate validation in your identity provider.

  2. For PAS, follow the procedure in the table below that corresponds to your use case. This includes downloading and importing a new certificate and updated SAML metadata in your identity provider.

    Solution Name Procedure
    CA Single Sign-On aka CA SiteMinder Configuring CA as an Identity Provider
    PingFederate Configuring PingFederate as an Identity Provider
    Active Directory Federation Services Configuring ADFS as an Identity Provider

  3. For the SSO service, follow the procedure in the table below that corresponds to your use case. This includes downloading the SAML Service Provider metadata for each SAML identity provider integration, such as trust, partnership, or Federation, and importing the updated SAML Service Provider metadata in your identity provider.

    Solution Name Procedure
    ADFS Configuring a Single Sign-On Service Provider
    CA SSO Configuring a Single Sign-On Service Provider
    Okta Configure Okta as an Identity Provider
    PingFederate Configure PingFederate as an Identity Provider
    Additional Documentation Integration Guides

  4. Re-enable certificate validation in your identity provider.