Managing Certificates

This topic describes how to check expiration dates and rotate CA certificates used by PCC for TLS communication and managed by the operator.

This topic does not describe how to rotate PCC certificates for individual service instances. Rotating a CA certificate as described here regenerates all the intermediate certificates for service instances.

Overview

When you installed PCC, you generated a CA certificate to BOSH CredHub by following the procedures in Preparing for TLS. You must manually rotate this CA certificate before the certificate expires or if it is compromised.

You must be running a version of PCC that supports certificate rotation. Find your PCC version in the table below. You must be running the minimum patch version or later to perform CA certificate rotation.

PCC version
(Major.Minor)
Minimum patch version
that supports CA rotation
1.9 1.9.0
1.8 1.8.2
1.7 1.7.2
1.6 1.6.4

To manually rotate certificates:

  1. Access BOSH CredHub
  2. Check Expiration Dates
  3. Rotate Certificates Manually

Warning: Do not attempt to rotate a CA certificate on your own. Contact Pivotal Support and perform the following procedures with their help.

PCC Certificates

PCC uses the following certificates, shown here with CredHub paths:

  • /services/tls_ca: CredHub CA that signs certificates for every PCC service instance
  • /p-bosh/service-instance_GUID/gemfire-locator-certificate
  • /p-bosh/service-instance_GUID/gemfire-server-certificate

where GUID is the name of the service instance GUID. When rotating certificates for PCC, the operator needs to rotate only /services/tls_ca as described below. The errand upgrade-all-service-instances regenerates all certificates for service instances, plus the certificates of the locator and server VMs of all the service instances.

Access BOSH CredHub

Pivotal Cloud Foundry (PCF) stores some of its internal CA and non-CA certificates in the BOSH CredHub credentials store. For more information, see BOSH CredHub.

To access the BOSH CredHub credentials store, you must retrieve credentials from BOSH Director and then use the credentials to log in to CredHub from the Ops Manager VM.

To access the BOSH CredHub:

  1. In the Ops Manager Installation Dashboard, click the BOSH Director tile.

  2. Click the Credentials tab.

  3. In the BOSH Director section, click the link to the BOSH Commandline Credentials. CredHub Credentials

  4. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
    For example:

    {"credential":"BOSH_CLIENT=ops_manager
    BOSH_CLIENT_SECRET=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
    BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
    BOSH_ENVIRONMENT=10.0.0.5 bosh "}
    

    The BOSH_CLIENT is the BOSH CredHub client name (“ops_manager” in this example) and the BOSH_CLIENT_SECRET is the BOSH CredHub client secret.

  5. Following the procedure in Gather Credential and IP Address Information to obtain the information needed to log in to the BOSH Director VM. Record the IP address for the Ops Manager Director and the Director Credentials.

  6. Log in to the Ops Manager VM by following the procedure in Log in to the Ops Manager VM with SSH.

  7. From the Ops Manager VM, set the API target of the CredHub CLI to your BOSH CredHub server by running the following command:

    credhub api https://BOSH-DIRECTOR:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

    where BOSH-DIRECTOR is the IP address of the BOSH Director VM you recorded above.

    For example:

    $ credhub api https://10.0.0.5:8844 \
      --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

  8. Log in to CredHub by running the following command:

    credhub login --client-name=CREDHUB-CLIENT-NAME --client-secret=CREDHUB-CLIENT-SECRET
    

    Where:

    • CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT earlier in this procedure (“ops_manager” in this example).
    • CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET earlier in this procedure.

    For example:

    $ credhub login \
    --client-name=ops_manager \
    --client-secret=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
    

Check Expiration Dates

You can check the expiration date for CA and PCC certificates.

To check certificate expiration dates:

  1. Log in to CredHub by following the procedure in Access BOSH CredHub above.

  2. Run the following command:

    credhub get -n CERTIFICATE-NAME -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"
    

    where CERTIFICATE-NAME is the name of the certificate you are checking.

    For example:

    $ credhub get -n /services/tls_ca -j | jq -r .value.ca | \
        openssl x509 -text -noout | grep -A 2 "Validity"
    Validity
              Not Before: Jan 23 19:04:58 2019 GMT
              Not After : Jan 23 19:04:58 2020 GMT
    

    The above command retrieves the certificate from CredHub, decrypts it, parses its JSON structure, and finds its Validity properties. The expiration date for the certificate is the value of Not After.

Rotate CA Certificates Manually

Warning: This procedure assumes that you use only one foundation and do not have a WAN setup. For WAN setup, all of the apply changes steps will need to be applied to your entire WAN. Also, all certificates must include SAN and CN entries.

To manually rotate the CA certificate:

  1. Obtain or Generate a New CA Certificate
  2. Add New Certificates to Ops Manager
  3. Apply Changes
  4. Rebind Your Apps
  5. Set New CA Certificate to /services/tls_ca
  6. Remove the Old CA Certificate
  7. Apply Changes
  8. Rebind Your Apps

Obtain or Generate a New CA Certificate

If you have a TLS cluster, obtain the CA (certificate authority) that signs its certificate. You will add this to the Ops Manager settings in the next step.

If you use self-signed certificates, generate a new self-signed certificate with a separate name in CredHub. In this example we generate the new certificate with the name “/services/new_ca”:

$ credhub generate \
    --name="/services/new_ca" \
    --type="certificate" \
    --no-overwrite \
    --is-ca \
    --common-name="servicesCA"

If you use an intermediate certificate signed by a root CA in credhub, generate a new certificate signed by the CredHub root CA:

$ credhub generate \
    --name="/services/new_ca" \
    --type="certificate" \
    --no-overwrite \
    --is-ca \
    --common-name="servicesCA" \
    --ca=/path-to-root-ca

Add New Certificates to Ops Manager

To add new and old CA certificates to Ops Manager:

  1. Log in to CredHub.
  2. Obtain the current CA with credhub get --name=/services/tls_ca -k ca.
  3. Obtain the new CA, either from a pre-existing file, or from your new CredHub location using credhub get --name=/services/new_ca -k ca.
  4. Make sure both ca values are pasted into Ops Manager -> BOSH Director -> Security -> Trusted Certificates.
  5. Make sure both ca values are pasted into PAS tile -> Networking -> Certificate Authorities Trusted by Router and HAProxy
  6. Click Save.

Apply Changes

Warning: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

To apply these changes, do the following:

  1. Navigate back to the Installation Dashboard.
  2. Click Review Pending Changes.
  3. Click ERRANDS on PCC.
  4. Select Upgrade All Service Instances. Upgrade All Service Instances
  5. Click Apply Changes.

Rebind Your Apps

If your apps do not use the trusted store to validate server certificates, you must rebind apps to the service instance to receive the updated CA certificate. Apps must be written in Spring to use the trusted store.

To rebind your app:

  1. Stop your app by running the following command:

    cf stop YOUR-APP
    

    Where YOUR-APP is the name of your app.

  2. Unbind your app from the service instance by running the following command:

    cf unbind-service YOUR-APP YOUR-SERVICE-INSTANCE
    

    Where:

    • YOUR-APP is the name of your app.
    • YOUR-SERVICE-INSTANCE is the name of your service instance.
  3. Rebind your app to the service instance by running the following command:

    cf bind-service YOUR-APP YOUR-SERVICE-INSTANCE
    

    Where:

    • YOUR-APP is the name of your app.
    • YOUR-SERVICE-INSTANCE is the name of your service instance.
  4. Restage your app by running the following command:

    cf restage YOUR-APP
    

    Where YOUR-APP is the name of your app.

Set New CA Certification to /services/tls_ca

To have all instances use the new CA Certificate, do the following:

If you have an existing certificate, obtain the CA certificate and private key file corresponding to the CA that you applied in the previous step. Then, run the following command:

credhub set \
    --name="/services/tls_ca" \
    --type="certificate" \
    --certificate=PEM-PATH/root.pem \
    --private=CERT-KEY

where:

  • PEM-PATH is the path to the root.pem file.
  • CERT-KEY is the private key for the certificate.

If you have created a new self-signed or intermediary certificate in /services/new_ca, set /services/tls_ca to the new CredHub value /services/new_ca that you generated in the previous step:

$ credhub get -n /services/new_ca -k ca > new_ca.ca
$ credhub get -n /services/new_ca -k certificate > new_ca.certificate
$ credhub get -n /services/new_ca -k private_key > new_ca.private_key
$ credhub set -n /services/tls_ca \
    --type=certificate \
    --root=new_ca.ca \
    --certificate=new_ca.certificate \
    --private=new_ca.private_key

Remove the Old CA Certificate

After your apps have reconnected to service instances with certificates generated by the new CA, remove the old CA certificate:

  1. Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile.

  2. Click Security.

  3. Delete the old CA certificate in Trusted Certificates.

  4. Click Save.

  5. Navigate to the Installation Dashboard in Ops Manager and click the PAS tile.

  6. Click Networking.

  7. Delete the old CA certificate in Certificate Authorities Trusted by Router and HAProxy

  8. Click Save.

Apply Changes

Warning: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

To apply these changes, do the following:

  1. Navigate back to the Installation Dashboard.
  2. Click Review Pending Changes.
  3. Click ERRANDS on PCC.
  4. Select Upgrade All Service Instances. Upgrade All Service Instances
  5. Click Apply Changes.

Rebind Your Apps

Again, if your apps do not use the trusted store to validate server certificates, you must rebind your apps to incorporate this last round of changes. See Rebind Your Apps above.