Rotating Certificates

Page last updated:

This topic describes how to check expiration dates and rotate certificates used by MySQL for PCF.

Types of Certificates

MySQL for PCF uses the following types of certificates:

• TLS certificates managed by the operator: If you are using TLS for MySQL for PCF, you provided a CA certificate to BOSH CredHub when you did the procedures in Preparing for TLS. Operators must manually rotate this TLS CA certificate before the certificate expires or if it is compromised.

• Internal certificates managed by MySQL for PCF: When you upgrade to MySQL for PCF v2.6, MySQL for PCF automatically rotates the internal certificates for single node and leader-follower service instances. The new certificates do not expire for 5 years. For more information, see Rotate Internal Certificates below.

  Note: MySQL for PCF v2.6 does not rotate certificates for highly available (HA) clusters service instances. To rotate these certificates, you must upgrade to MySQL for PCF v2.7. See Upgrading MySQL for PCF.

For a list of the certificates used by MySQL for PCF, see Certificates Used by MySQL for PCF below.

Warning: Do not attempt to rotate a CA certificate on your own. Contact Pivotal Support and perform the following procedures with their help.

Certificates Used by MySQL for PCF

The following table lists the certificates used by MySQL for PCF:

In the above table, GUID is the GUID for the service instance. To find the GUID of your service instance, do the procedure in Find Information about Your Service Instance.

Check Expiration Dates

Pivotal Cloud Foundry (PCF) stores the certificates used by MySQL for PCF and then use the credentials to log in to CredHub from the Ops Manager VM. You can use the CredHub API to check the expiration date for the certificates used by MySQL for PCF. For more information, see BOSH CredHub.

To use the CredHub API, you must retrieve credentials from BOSH Director and then use the credentials to log in to CredHub from the Ops Manager VM.

To check certificate expiration dates:

1. If the CredHub CLI is not installed on your local machine, install the CredHub CLI by doing the instructions in credhub-cli on GitHub.

2. In the Ops Manager Installation Dashboard, click the BOSH Director tile.

3. Click the Credentials tab.

4. In the BOSH Director section, click the link to the BOSH Commandline Credentials.

5. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
   For example:

   {"credential":"BOSH_CLIENT=ops_manager
   BOSH_CLIENT_SECRET=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
   BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
   BOSH_ENVIRONMENT=10.0.0.5 bosh "}
   

   The BOSH_CLIENT is the BOSH CredHub client name and the BOSH_CLIENT_SECRET is the BOSH CredHub client secret.

6. Record the information needed to log in to the BOSH Director VM by doing the procedure in Gather Credential and IP Address Information.

7. Log in to the Ops Manager VM by doing the procedure in Log in to the Ops Manager VM with SSH.

8. From the Ops Manager VM, set the API target of the CredHub CLI to your BOSH CredHub server by running the following command:

   credhub api https://BOSH-DIRECTOR:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
   

   Where BOSH-DIRECTOR is the IP address of the BOSH Director VM you recorded above.
   
   For example:

   $ credhub api https://10.0.0.5:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
   

9. Log in to CredHub by running the following command:

   credhub login --client-name=CREDHUB-CLIENT-NAME --client-secret=CREDHUB-CLIENT-SECRET
   

   Where:
   

   • CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT in step 5 of this procedure.
   • CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET in step 5 of this procedure.

   For example:

   $ credhub login \
   --client-name=credhub \
   --client-secret=abcdefghijklm123456789
   

10. For each certificate in the Certificates Used by MySQL for PCF list above, run the following command:

    credhub get -n CERTIFICATE-NAME -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"
    

    Where CERTIFICATE-NAME is the name of the certificate you are checking.
    
    For example:

    $ credhub get -n /services/tls_ca -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"
    Validity
              Not Before: Jan 23 19:04:58 2019 GMT
              Not After : Jan 23 19:04:58 2020 GMT
    

    The above command retrieves the certificate from CredHub, parses its JSON structure, decrypts it, and finds its Validity properties. The expiration date for the certificate is the value of Not After. The expiration date can be different for different certificates.
    
    

11. If any of your certificates are about to expire or have expired, do one of the following:

    • If your TLS certificates are about to expire or have expired, do the procedures in Rotate the TLS CA Certificate below. This rotates both the TLS certificates and the internal certificates.
      
      
    • If your internal certificates for single node or leader-follower service instances are about to expire or have expired, do the procedure in Rotate Internal Certificates below.
      
      
    • If only your internal certificates for HA clusters are about to expire or have expired, upgrade to MySQL for PCF v2.7. See Upgrading MySQL for PCF.

Rotate the TLS CA Certificate

This following procedures rotate a TLS CA certificate and regenerate the certificates signed by the TLS CA certificate. Only the certificates for single node and leader-follower service instances are rotated. To rotate certificates for HA clusters, upgrade to MySQL for PCF v2.7. See Upgrading MySQL for PCF.

To manually rotate the TLS CA certificate:

1. Add a New TLS CA Certificate
2. Regenerate Internal Certificates
3. Remove the Old TLS CA Certificate
4. Rebind Your App

Add a New TLS CA Certificate

Warning: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

To add a new TLS CA Certificate to Ops Manager:

1. Log in to CredHub by doing steps 1 through 9 in Check Expiration Dates above.

2. Set the TLS CA certificate by doing one of the following:

   Note: Your PCF deployment can have multiple CA certificates. Pivotal recommends using a dedicated CA certificate for each service.

   • If you do not have a TLS CA certificate: generate a new certificate by running the following command:

     credhub regenerate --name="/services/tls_ca"
     

   • If you have an existing certificate: create a new root.pem file with the contents of the certificate. Then, run the following command:

     credhub set \
         --name="/services/tls_ca" \
         --type="certificate" \
         --certificate=PEM-PATH/root.pem \
         --private=CERT-KEY...
     

     Where:

     • PEM-PATH is the path to the root.pem file.
     • CERT-KEY is the private key for the certificate.

3. Extract and output the certificate portion from the TLS CA certificate by running the following command:

   bosh int <(credhub get \
   --name=/services/tls_ca) \
   --path /value/certificate
   

4. Record the output.

5. Navigate to the Ops Manager Installation Dashboard and click the BOSH Director tile.

6. Click Security.

7. Append the contents of the new TLS CA certificate to the old CA certificate under Trusted Certificates. Do not remove the old TLS CA certificate.

8. Click Save.

9. (Optional) Distribute the new TLS CA certificate to your VMs and regenerate each certificate using the new TLS CA certificate by doing the following:

   1. Navigate back to the Installation Dashboard.
   2. Click Review Pending Changes.
   3. Click ERRANDS.
   4. Select Upgrade all On-demand MySQL Service Instances.

   Note: If you do not select Upgrade all On-demand MySQL Service Instances, you must manually run the upgrade-all-services-instances errand after applying changes. To manually run this errand, see Regenerate Internal Certificates below.

10. Return to the Installation Dashboard in Ops Manager and click Apply Changes.

Regenerate Internal Certificates

Note: The following procedure manually runs the upgrade-all-service-instances errand. If you selected the Upgrade all On-demand MySQL Service Instances in Add a New TLS CA Certificate above, skip the following procedure.

To regenerate your internal certificates and redeploy your service instances:

1. Log in to CredHub by doing steps 1 through 9 in Check Expiration Dates above.

2. Regenerate all certificates signed by the old TLS CA certificate by running the following command:

   curl "https://BOSH-DIRECTOR:8844/api/v1/bulk-regenerate" -X POST \
   -d '{ "signed_by": "/services/tls_ca" }' \
   -H "authorization: $(credhub --token)" \
   -H 'content-type: application/json'
   

   Where BOSH-DIRECTOR is the IP address of the BOSH Director VM.

3. Redeploy all service instances with the new internal certificates by running the following command:

   bosh -d p-mysql-GUID run-errand upgrade-all-service-instances
   

   Where pivotal-mysql-GUID is the BOSH deployment name for your MySQL service broker.

Remove the Old CA Certificate

Warning: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

If you are rotating your TLS CA certificate because it was compromised, you must remove your old TLS CA certificate before rebinding your app.

To remove the old TLS CA certificate:

1. Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile.

2. Click Security.

3. Delete the old TLS CA certificate in Trusted Certificates.

4. Click Save.

5. Return to the Installation Dashboard in Ops Manager and click Apply Changes. This restarts all the VMs in your deployment.

Rebind Your App

If your apps do not use the trusted store to validate certificates, developers must rebind apps to the service instance to receive the updated TLS CA certificate. Apps not written in Java and Spring do not use the trusted store.

To rebind your app:

1. Stop your app by running the following command:

   cf stop YOUR-APP
   

   Where YOUR-APP is the name of your app.

2. Unbind your app from the service instance by running the following command:

   cf unbind-service YOUR-APP YOUR-SERVICE-INSTANCE
   

   Where:

   • YOUR-APP is the name of your app.
   • YOUR-SERVICE-INSTANCE is the name of your service instance.

3. Re-bind your app to the service instance by running the following command:

   cf bind-service YOUR-APP YOUR-SERVICE-INSTANCE
   

   Where:

   • YOUR-APP is the name of your app.
   • YOUR-SERVICE-INSTANCE is the name of your service instance.

4. Restage your app by running the following command:

   cf restage YOUR-APP
   

   Where YOUR-APP is the name of your app.

Rotate Internal Certificates

When you upgrade to MySQL for PCF v2.6, MySQL for PCF automatically rotates the internal certificates single node and leader-follower service instances. The new certificates do not expire for 5 years. The below commands upgrade all of your MySQL for service instances and distribute the new certificates to your service instances.

To ensure that the internal certificates rotate, do one of the following:

• If you have not upgraded to MySQL for PCF v2.6, do the following:

  1. Navigate to the Ops Manager Installation Dashboard and click the MySQL for Pivotal Cloud Foundry v2 tile.
  2. Click Errands.

  3. Under Post-Deploy Errands, ensure that Upgrade all On-demand MySQL Service Instances is set to Default (On).

     

  4. Click Save.

• If you have already upgraded to MySQL for PCF v2.6 without the Upgrade all On-demand MySQL Service Instances errand on, run the following command:

  bosh -d p-mysql-GUID run-errand upgrade-all-service-instances
  

  Where pivotal-mysql-GUID is the BOSH deployment name for your MySQL service broker.

  For a list of certificates used by MySQL for PCF, see Certificates Used by MySQL for PCF above.