LATEST VERSION: 2.5 - RELEASE NOTES

Rotating CA Certificates

Page last updated:

This topic describes how to rotate certificate authority (CA) certificates. If you are using TLS for MySQL for PCF, you provided a CA certificate to BOSH CredHub when performing the procedures in Preparing for TLS. Operators must rotate these CA certificates before they expire or if they are compromised.

WARNING: Do not attempt to rotate a CA certificate on your own. Contact Pivotal Support and perform the following procedures with their help.

Overview

To rotate CA certificates, you must do the following:

  1. Add a New CA Certificate
  2. Regenerate Internal Certificates
  3. Rebind Your App
  4. Remove Old the CA Certificate

Add a New CA Certificate

WARNING: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

To add a new CA certificate, follow the procedures you must do the following:

  1. Find the CredHub Credentials in Ops Manager
  2. Regenerate and Add the New CA Certificate

Find the CredHub Credentials in Ops Manager

You need the BOSH CredHub client name and client secret to complete the Regenerate and Add a New CA Certificate procedure below.

To find the BOSH CredHub client name and client secret, do the following:

  1. In the Ops Manager Installation Dashboard, click the BOSH Director tile.
  2. Click the Credentials tab.
  3. In the BOSH Director section, click the link to the BOSH Commandline Credentials. CredHub Credentials
  4. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
    For example:

    {"credential":"BOSH_CLIENT=ops_manager
    BOSH_CLIENT_SECRET=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
    BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
    BOSH_ENVIRONMENT=10.0.0.5 bosh "}
    

    The BOSH_CLIENT is the BOSH CredHub client name and the BOSH_CLIENT_SECRET is the BOSH CredHub client secret.

Regenerate and Add the New CA Certificate

To regenerate and add the CA Certificate to Ops Manager, do the following:

  1. To log in to the Ops Manager VM, do the following procedures:

    1. To gather the information needed to log in to the BOSH Director VM, see Gather Credential and IP Address Information.
    2. To log in to the Ops Manager VM, see Log in to the Ops Manager VM with SSH.
  2. From the Ops Manager VM, to set the API target of the CredHub CLI to your BOSH CredHub server, run the following:

    credhub api https://BOSH-DIRECTOR:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

    Where BOSH-DIRECTOR is the IP address of the BOSH Director VM you recorded in step 1.a above.

    For example:

    $ credhub api https://10.0.0.5:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

  3. To log in to CredHub, run the following:

    credhub login --client-name=CREDHUB-CLIENT-NAME --client-secret=CREDHUB-CLIENT-SECRET
    

    Where:

    • CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT in step 4 in Find the CredHub Credentials above.
    • CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET in step 4 in Find the CredHub Credentials above.

      For example:
      $ credhub login \
      --client-name=credhub \
      --client-secret=abcdefghijklm123456789
      
  4. To generate a CA certificate or provide an existing one, do one of the following:

    Note: Your PCF deployment can have multiple CA certificates. Pivotal recommends using a dedicated CA certificate for services.

    • If you do not have a CA certificate, to generate a CA certificate run the following:

      credhub regenerate --name="/services/tls_ca" 
      
    • If you have an existing CA certificate that you want to use, create a new file called root.pem with the contents of the certificate. Then run the following command:

      credhub set \
          --name="/services/tls_ca" \
          --type="certificate" \
          --certificate=PEM-PATH/root.pem \
          --private=CERT-KEY...
      

      Where:

      • PEM-PATH is the path to the root.pem file.
      • CERT-KEY is the private key for the certificate.
  5. To extract and print the certificate portion from the CA certificate, run the following command:

    bosh int <(credhub get \
    --name=/services/tls_ca) \
    --path /value/certificate
    
  6. Copy the output.

  7. Navigate to the Installation Dashboard in Ops Manager and select the BOSH Director tile. Click Security.

  8. Append the contents of the new CA certificate to the old CA certificate under Trusted Certificates. Do not remove the old CA certificate. Click Save.

  9. Return to the Installation Dashboard in Ops Manager and click Apply Changes. This restarts all the VMs in your PCF deployment and applies your CA certificate.

Regenerate Internal Certificates

To regenerate your internal certificates and redeploy your service instances, follow the procedure below:

  1. To log in to CredHub, run the following:

    credhub login --client-name=CREDHUB-CLIENT-NAME \
    --client-secret=CREDHUB-CLIENT-SECRET
    

    Where:

    • CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT in step 4 in Find the CredHub Credentials above.
    • CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET in step 4 in Find the CredHub Credentials above.

      For example:
      $ credhub login \
      --client-name=credhub \
      --client-secret=abcdefghijklm123456789
      
  2. To regenerate all certificates signed by the old CA certificate, run the following

    curl "https://BOSH-DIRECTOR/api/v1/bulk-regenerate"" -X POST \
    -d '{ "signed_by": "/services/tls_ca" }'\
    -H "authorization: bearer $(credhub --token)"\
    -H 'content-type: application/json'
    

    Where BOSH-DIRECTOR is the IP address of the BOSH Director VM.

  3. To redeploy all service instances with the new certificates, run the following:

    bosh -d p-mysql-GUID run-errand upgrade-all-service-instances  
    

    Where pivotal-mysql-GUID is the BOSH deployment name for your MySQL service broker.

Rebind Your App

If your apps do not use the trusted store to validate server certificates, developers must re-bind apps to the service instance to receive the updated CA certificate. Apps not written in Java and Spring do not use the trusted store.

If you are rotating your CA certificate because it was compromised, you must remove your old CA certificate before rebinding your app. If you have not removed your old CA certificate, see Remove Old the CA Certificate below.

To re-bind your app, do the following:

  1. To stop your app, run the following command:

    cf stop YOUR-APP
    

    Where YOUR-APP is the name of your app.

  2. To unbind your app from the service instance, run the following command:

    cf unbind-service YOUR-APP YOUR-SERVICE-INSTANCE
    

    Where:

    • YOUR-APP is the name of your app.
    • YOUR-SERVICE-INSTANCE is the name of your service instance.
  3. To re-bind your app to the service instance, run the following command:

    cf bind-service YOUR-APP YOUR-SERVICE-INSTANCE
    

    Where:

    • YOUR-APP is the name of your app.
    • YOUR-SERVICE-INSTANCE is the name of your service instance.
  4. To restage your app, run the following command:

    cf restage YOUR-APP
    

    Where YOUR-APP is the name of your app.

Remove Old the CA Certificate

WARNING! This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

After your apps have reconnected to service instances with certificates generated by the new CA, do the following to remove the old CA certificate:

  1. Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile. Click Security.

  2. Remove the old CA certificate in Trusted Certificates and click Save.

  3. Return to the Installation Dashboard in Ops Manager and click Apply Changes. This restarts all the VMs in your deployment.

  4. If you are rotating your CA certificate because it was compromised, you must rebind your apps to update the CA certificate. See Rebind Your App above.

Create a pull request or raise an issue on the source for this page in GitHub