Managing Certificates

This topic describes how to manage the internal Certificate Authorities (CAs) and certificates in Pivotal Cloud Foundry (PCF) that are visible to the Ops Manager API. The topic includes:

  • How to check the expiration dates of CAs and certificates (“certs”).
  • How to rotate different types of CAs and certs as needed.
  • A master procedure for checking and rotating all of the Ops Manager CAs and certs that require manual checking and rotation.

WARNING: You must be using Ops Manager version 2.1.6 or higher to avoid a critical issue. Upgrade to at least that patch version before running any of the following certificate rotation procedures.

For information about rotating IPsec certificates, see Rotating IPsec Certificates.

For information about using trusted third-party certificates for both apps hosted on PCF and internal PCF components, see Setting Trusted Certificates.

Introduction

The Ops Manager API manages and lists internal certificates that enable PCF components to communicate with each other securely, using TLS. It can also list certificates used externally, such as SAML certs that authenticate to an external identity provider.

To keep PCF running, you must keep track of which certs are set to expire soon, and rotate them before they expire. To do this, you can run the Check and Rotate Certs procedure.

Certificate Types

Four types of PCF internal certs require planned rotation:

  • The Ops Manager root CA issues other certs that PCF uses.

  • Non-configurable certs are leaf certs issued directly by the Ops Manager root CA, or by intermediate CAs in a chain of trust originated by the root CA.

    • The Ops Manager API’s deployed/certificates endpoint returns information about these.
    • The Ops Manager API’s active/regenerate endpoint rotates these.
    • They are created by a CA stored in Ops Manager itself, or created and stored by CredHub and managed by Ops Manager calls to the CredHub API.
  • Configurable certs are leaf certs supplied by the user and pasted into configuration fields in Ops Manager.

    • For convenience, some configuration panes include a Generate RSA Certificate link that supplies valid certificates, but users can obtain configurable certs from elsewhere.
    • The Ops Manager API’s deployed/certificates endpoint returns information about these, along with information about non-configurable certs.
  • Non-Rotatable certs are five specific leaf certs that, like non-configurable certs, are issued by the root CA and returned by the deployed/certificates endpoint.

    • Unlike non-configurable certs, non-rotatable certs cannot currently be rotated by the Ops Manager API.

In addition to the types above, some Pivotal products issue their own tile certificates that are not managed by or visible to the Ops Manager API. These tile certs do not require planned rotation because they rotate automatically with product upgrades.

Pivotal Application Service (PAS) and Pivotal Container Service (PKS) both use tile certs in addition to their Ops Manager certs.

Master Procedure: Check and Rotate Certs

Ops Manager’s root CA expires after four years after creation, but most non-configurable certs expire after two years. Configurable certs generated by Ops Manager also typically expire after two years.

Rotating the Ops Manager root CA automatically rotates all configurable certs, but a shorter and easier procedure rotates all configurable certs without also rotating the root CA.

The following master procedure checks expiration dates of different types of internal certs and rotates them only as necessary. You can run this procedure only when records show that your certs will expire soon, or else periodically, to comply your organization’s security compliance policies:

  1. Follow the Check Ops Manager Root CA Expiration Date procedure and record your root CA expiration date.

  2. Follow the Check Leaf Certificate Expiration Dates procedure and record any leaf certs that are expiring soon.

  3. Follow the Identify Non-Configurable, Configurable, and Unrotatable Leaf Certs to determine which types of leaf certs require rotation.

  4. Proceed as follows to rotate your certs as needed:

    1. If you have any non-rotatable certs expiring soon, call Pivotal Support.
    2. If your root CA expires soon, follow the Rotate Root and Leaf Certificates procedure.
      • If you also have configurable certs expiring soon, Pivotal recommends that you include the Step 3 (Optional): Rotate Configurable Certs step, which avoids one deploy. But you can also rotate your configurable certs and re-deploy separately later.
    3. If you have non-configurable leaf certs expiring soon, but not your root CA, follow the Rotate Non-Configurable Certificates procedure.
    4. If you have configurable leaf certs expiring soon, but not your root CA, follow the Rotate Configurable Certificates procedure.

Check Expiration Dates and Certificate Types

The following procedures check the expiration dates and types of CAs and certs that the Ops Manager API lists and manages.

Check Ops Manager Root CA Expiration Date

  1. Perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the steps below.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. To retrieve the Ops Manager root CA, use curl to make an Ops Manager API call to the https://OPS-MAN-FQDN/api/v0/certificate_authorities endpoint. For example:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    

  3. To make the JSON output more readable, you can pipe it to jq or another text editor with JSON formatting.

  4. In the certificate_authorities list returned, if there is more than one, find the CA with "active": true.

  5. To determine its expiration date of the active CA, refer to its expires_on value. For example, the root CA shown below expires on September 5, 2019:

    {
    "certificate_authorities": [
    {
      "guid": "9c9a110c8f82a1e4aaca",
      "issuer": "Pivotal",
      "created_on": "2017-09-05T22:47:53Z",
      "expires_on": "2019-09-05T22:47:53Z",
      "active": true,
      "cert_pem": "-----BEGIN CERTIFICATE-----\
      [...]
      \n-----END CERTIFICATE-----\n"
    

Check Leaf Certificate Expiration Dates

The non-configurable certificates in your deployment expire every two years, as may your configurable certs. To retrieve information about the expiration dates for these certificates, do the following:

  1. If you haven’t already, perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the steps below.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. To check the system for certificates that expire within a given time interval, use curl to call the https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=TIME API endpoint, replacing TIME with an integer-letter code. Valid letter codes are d for days, w for weeks, m for months, and y for years.

    For example, the following command searches for certificates expiring within six months:

    $ curl "https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=6m" \
          -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Replace YOUR-UAA-ACCESS-TOKEN with the access_token value you recorded in the previous step.

  3. To make the JSON output more readable, you can pipe it to jq or another text editor with JSON formatting.

  4. To determine the expiration date of each certificate, refer to its expires_on value.

  5. To determine the type of each certificate in the output, follow the Identify Non-Configurable, Configurable, and Unrotatable Leaf Certs procedure.

Identify Non-Configurable, Configurable, and Unrotatable Leaf Certs

Output from the deployed/certificates endpoint, such as the output generated in the Check Leaf Certificate Expiration Dates procedure, combines information about non-configurable, configurable, and unrotatable leaf certs into a single list.

You need to manually rotate these different certificate types in different ways. The following rules identify the type of each cert requiring rotation:

  • Non-Rotatable cert property values fulfill one of the following conditions:

    • property_reference includes nats_client_ca, nats_server_certificate, nats_director_client_certificate, or nats_health_monitor_client_certificate
    • variable_path is /opsmgr/bosh_dns/tls_ca

    If you have non-rotatable certs expiring soon, call Pivotal Support.

  • Non-Configurable certs have the following property values, but are not non-rotatable as identified above:

    • configurable is false
    • location is either ops_manager or credhub
  • Configurable certs have the following property value:

    • configurable is true

Rotate CAs and Leaf Certificates

The following procedures rotate CAs and certificates that are listed or managed by the Ops Manager API.

Note: The rotation procedures described in this topic does not work when your certificates have already expired. If your certificates have expired, contact Pivotal Support for guidance.

Rotate Root and Leaf Certificates

This procedure uses the Ops Manager API to rotate the Ops Manager root certificate and the intermediate CAs and non-configurable leaf certificates underneath it. It also includes the option to rotate configurable certs.

PCF users never need to manually rotate intermediate CAs, because they rotate automatically when the root CA is rotated.

To prevent system downtime, this procedure includes two BOSH redeploys. When you click Apply Changes for the first time, BOSH applies new certificates to PCF components alongside the old ones. The second Apply Changes then deletes the old ones. Each successful redeploy verifies that the certificate rotation process is proceeding correctly.

WARNING: You must complete these steps in the exact order specified. Otherwise, you risk damaging your deployment.

Step 1: Add a New Root CA

  1. If you haven’t already, perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN in the following procedures.

    Note: When you record your Ops Manager access token, remove any newline characters such as \n.

  2. Use Ops Manager to generate a new CA, or else add your own custom CA.

    • To use your own custom CA, follow the instructions in Adding a Custom Certificate Authority.
    • To use a Pivotal-generated CA, call the Ops Manager API generate endpoint as follows:
      $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/generate" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{}'
      
      If the command succeeds, the API returns a response that includes the new CA certificate:
        HTTP/1.1 200 OK
      {
        "guid": "f7bc18f34f2a7a9403c3",
        "issuer": "Pivotal",
        "created_on": "2017-01-19",
        "expires_on": "2021-01-19",
        "active": false,
        "cert_pem": "-----BEGIN EXAMPLE CERTIFICATE-----
        MIIC+zCCAeOgAwIBAgIBADANBgkqhkiG9w0BAQsFADAfMQswCQYDVQQGEwJVUzEQ
        EXAMPLEoCgwHUGl2b3RhbDAeFw0xNDarthgyMTQyMjVaFw0yMTAxMTkyMTQyMjVa
        EXAMPLEoBgNVBAYTAlVTMRAwDgYDVVaderdQaXZvdGFsMIIBIjANBgkqhkiG9w0B
        EXAMPLEoAQ8AMIIBCgKCAQEAyV4OhPIIZTEym9OcdcNVip9Ev0ijPPLo9WPLUMzT
        EXAMPLEo/TgD+DP09mwVXfqwBlJmoj9DqRED1x/6bc0Ki/BAFo/P4MmOKm3QnDCt
        EXAMPLEooqgA++2HYrNTKWJ5fsXmERs8lK9AXXT7RKXhktyWWU3oNGf7zo0e3YKp
        EXAMPLEoh1NwIbNcGT1AurIDsxyOZy1HVzBLTisMyDogJmSCLsOw3qUDQjatjXKw
        EXAMPLEojG3nv2hvD4/aTOiHuKM3+AGbnaS2MdIOvFOh/7Y79tUp89csK0gs6uOd
        EXAMPLEohe4DcKw5CzUTfHKNXgHyeoVOBPcVQTp4lJp1iQIDAQABo0IwQDAdBgNV
        EXAMPLEoyH4y7VEuImLStXM0CKR8uVqxX/gwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
        EXAMPLEoBAMCAQYwDQYJKoZIhvcNAQELBQADggEBALmHOPxdyBGnuR0HgR9V4TwJ
        EXAMPLEoGLKVT7am5z6G2Oq5cwACFHWAFfrPG4W9Jm577QtewiY/Rad/PbkY0YSY
        EXAMPLEokrfNjxjxI0H2sr7qLBFjJ0wBZHhVmDsO6A9PkfAPu4eJvqRMuL/xGmSQ
        EXAMPLEoCynMNz7FgHyFbd9D9X5YW8fWGSeVBPPikcONdRvjw9aEeAtbGEh8eZCP
        EXAMPLEob33RuR+CTNqThXY9k8d7/7ba4KVdd4gP8ynFgwvnDQOjcJZ6Go5QY5HA
        EXAMPLEoPFW8pAYcvWrXKR0rE8fL5o9qgTyjmO+5yyyvWIYrKPqqIUIvMCdNr84=
          -----END EXAMPLE CERTIFICATE-----
          "
      
  3. Confirm that your new CA has been added by listing all of the root CAs for Ops Manager:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    The API call returns something like the following:
    HTTP/1.1 200 OK
    {
      "certificate_authorities": [
        {
          "guid": "f7bc18f34f2a7a9403c3",
          "issuer": "Pivotal",
          "created_on": "2017-01-09",
          "expires_on": "2021-01-09",
          "active": true,
          "cert_pem": "-----BEGIN CERTIFICATE-----\nMIIC+zCCAeOgAwIBAgI....etc"
        }
        {
          "guid": "a8ee01e33e3e3e3303e3",
          "issuer": "Pivotal",
          "created_on": "2017-04-09",
          "expires_on": "2021-04-09",
          "active": false,
          "cert_pem": "-----BEGIN CERTIFICATE-----\zBBBC+eAAAe1gAwAAAeZ....etc"
        }
      ]
    }
    
    Identify your newly added CA, which has active set to false. Record its GUID.

  4. Navigate to https://OPS-MAN-FQDN in a browser and log in to Ops Manager.

  5. Click the BOSH Director tile in the Installation Dashboard.

  6. Select the Director Config pane.

  7. Select Recreate All VMs. This propagates the new CA to all VMs to prevent downtime.

  8. Go back to the Installation Dashboard. For each service tile you have installed, do the following:

    1. Click the tile.
    2. Click the Errands tab.
    3. Enable the Recreate All Service Instances errand if provided.
  9. Click Review Pending Changes, then Apply Changes. When the deploy finishes, continue to the next section.

Step 2: Activate the New CA

  1. Use curl to make an Ops Manager API call that activates the new CA, replacing CERT-GUID with the GUID of your CA that you retrieved in the previous section:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/CERT-GUID/activate" \
      -X POST \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
      -H "Content-Type: application/json" \
      -d '{}'
    
    The API returns a successful response:
    HTTP/1.1 200 OK

  2. List your root CAs again to confirm that the new CA is active:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    Examine the response to ensure that your new CA has active set to true.

Step 3 (Optional) : Rotate Configurable Certs

If you have configurable certs expiring soon, this is a good time to follow the Rotate Configurable Certs procedure, so that your next Apply Changes step includes both new configurable certs and new non-configurable certs. You can also rotate your configurable certs later, separately and perform an additional redeploy.

Step 4: Rotate Non-Configurable Certs from the New Root

  1. Follow the Rotate Non-Configurable Certificates procedure to regenerate new non-configurable certs from the new root CA.

WARNING: Be sure to include the Review Pending Changes and Apply Changes step of the Rotate Non-Configurable Certificates procedure before you proceed to deleting the old CA.

Step 5: Delete the Old CA

  1. List your root CAs to retrieve the GUID of your old, inactive CA:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities" \
        -X GET \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    

  2. Use curl to make an API call to delete your old CA, replacing OLD-CERT-GUID with the GUID of your old, inactive CA:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/:OLD-CERT-GUID" \
        -X DELETE \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"
    
    The API returns a successful response.
    HTTP/1.1 200 OK

  3. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes.

Rotate Non-Configurable Certificates

This procedure regenerates non-configurable leaf certificates visible to the Ops Manager API, whether they are managed and stored by Ops Manager directly, or by CredHub at Ops Manager request.

Run by itself, this procedure does not rotate or otherwise affect the Ops Manager root CA.

  1. Use curl to make an API call to regenerate all non-configurable certificates and apply the new CA to your existing BOSH Director:

    $ curl "https://OPS-MAN-FQDN/api/v0/certificate_authorities/active/regenerate" \
        -X POST \
        -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN" \
        -H "Content-Type: application/json" \
        -d '{}'
    
    The API returns a successful response:
    HTTP/1.1 200 OK

  2. Navigate to Ops Manager, click Review Pending Changes, and click Apply Changes to perform a second redeploy.

Rotate Configurable Certificates

Configurable certificates are generated by the user and pasted into Ops Manager configuration panes where needed. Examples include certs that terminate SSL traffic into PAS, or authenticate a Single Sign-On (SSO) service plan to an external SAML server.

Pivotal recommends only rotating configurable certificates within the more inclusive Rotate Root and Leaf Certificates procedure. But if you are sure that only your configurable certificates need rotation, and no others, you can run this procedure by itself and click Apply Changes at the end.

To rotate configurable certs:

  1. If you haven’t already, use the Ops Manager API deployed/certificates endpoint to retrieve information about your expiring configurable certs, as described in the procedures Check Leaf Certificate Expiration Dates and Identify Non-Configurable, Configurable, and Unrotatable Leaf Certs.
    The information for each configurable certificate looks like this:

    {
      "configurable": true,
      "property_reference": ".properties.networking_poe_ssl_certs[0].certificate",
      "property_type": "rsa_cert_credentials",
      "product_guid": "cf-36066831c119c39736d3",
      ...
      "valid_until": "2019-01-25T22:07:58Z"
    },
    

  2. For each configurable certificate that expires soon:

    1. Find the text field the certificate is configured within the Ops Manager interface.
      • The product_guid field in the API output can help identify which tile the certificate is configured in. For example, the prefix p-bosh- refers to the BOSH Director tile, and the prefix cf- refers to the PAS tile.
      • The property_reference field in the API output can often help identify which Settings pane the cert is configured in. For example, the uaa.service_provider_key_credentials property is configured in the PAS tile > UAA pane.
      • You might have to look through multiple configuration panes to identify where a certificate is configured.
    2. Paste a new value for the certificate into the field
    3. Click Save at the bottom of each pane in which you have provided new certificates.
  3. If you are rotating configurable certs within the Rotate Root and Leaf Certificates procedure, continue to the next step. Otherwise, if you are rotating configurable certs only, return to the Installation Dashboard, click Review Pending Changes, and click Apply Changes.

Configurable Cert Example: Rotate SAML CA

SAML service provider credentials are one example of configurable certs in PCF. When PAS is configured to use SAML as an identity provider, it uses a configurable CA certificate to authenticate to an external SAML server, by generating ephemeral certs that PAS includes in its outbound request message headers. This CA has a two-year expiration period.

SAML service provider credentials are only required for your PAS deployment if all of these conditions are met:

  • You are using SSO in production for apps.
  • You are using SAML identity providers for SSO service plans.
  • You had Ops Manager generate a certificate for you by clicking the Generate RSA Certificate button.
  • You are validating the signature of SAML authentication request with your identity provider.

To regenerate and rotate SAML service provider certificates without disrupting PAS, do the following:

  1. Disable certificate validation in your identity provider.

  2. Navigate to UAA > SAML Service Provider Credentials in your PAS tile.

  3. Click Generate RSA Certificate.

  4. Following the procedure in the Configure SAML as an Identity Provider for PCF section of the Configuring Authentication and Enterprise SSO for PAS topic to import the new certificate to your identity provider. These steps will vary depending on which SAML provider you are using.

  5. Re-enable certificate validation in your identity provider.

Other Certificate Operations

To manage and retrieve information about certificates in your deployment, use the API calls in this section.

If you haven’t already, perform the steps in the Using Ops Manager API topic to target and authenticate with the Ops Manager User Account and Authentication (UAA) server. Record your Ops Manager access token, and use it for YOUR-UAA-ACCESS-TOKEN.

Note: When you record your Ops Manager access token, remove any newline characters such as \n.

Generate a Single RSA Certificate

To generate and return a new RSA certificate signed by the root CA, use curl to make the following API call:

$ curl "https://OPS-MAN-FQDN/api/v0/certificates/generate \
      -X POST \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"

Retrieve the Ops Manager Root Certificate

To return the Ops Manager root certificate as a file, use curl to make the following API call:

$ curl "https://OPS-MAN-FQDN/download_root_ca_cert \
      -X GET \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"

To return the Ops Manager root certificate as JSON, use curl to make the following API call:

$ curl "https://OPS-MAN-FQDN/api/v0/security/root_ca_certificate \
      -X GET \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"

List all RSA Certificates

To return metadata from all deployed RSA certificates visible to Ops Manager, except the root CAs, use curl to make the following API call:

$ curl "https://OPS-MAN-FQDN/api/v0/deployed/certificates \
      -X GET \
      -H "Authorization: Bearer YOUR-UAA-ACCESS-TOKEN"

Create a pull request or raise an issue on the source for this page in GitHub