Rotating Certificates

Page last updated:

This topic describes how to check expiration dates and rotate certificates used by MySQL for PCF.

Types of Certificates

MySQL for PCF uses the following types of certificates:

• TLS certificates managed by the operator: If you are using TLS for MySQL for PCF, you provided a CA certificate to BOSH CredHub when you did the procedures in Preparing for TLS. Operators must manually rotate this TLS CA certificate before the certificate expires or if it is compromised.

• Internal certificates managed by MySQL for PCF: When you upgrade to MySQL for PCF v2.6, MySQL for PCF automatically rotates the internal certificates for single node and leader-follower service instances. The new certificates do not expire for 5 years. For more information, see Rotate Internal Certificates below.

  Note: MySQL for PCF v2.6 does not rotate certificates for highly available (HA) clusters service instances. To rotate these certificates, you must upgrade to MySQL for PCF v2.7. See Upgrading MySQL for PCF.

For a list of the certificates used by MySQL for PCF, see Certificates Used by MySQL for PCF below.

Warning: Do not attempt to rotate a CA certificate on your own. Contact Pivotal Support and perform the following procedures with their help.

Certificates Used by MySQL for PCF

The following table lists the certificates used by MySQL for PCF:

In the above table, GUID is the GUID for the service instance. To find the GUID of your service instance, do the procedure in Find Information about Your Service Instance.

Workflow

Depending on which of your certificates are expiring, you must complete different procedures. The following flowchart describes the workflow for rotating certificate:

View a larger version of this flowchart

To rotate your certificates:

1. Determine which of your certificates are expiring. See Check Expiration Dates below.
2. If any of your certificates are about to expire or have expired, do one of the following:
   • If your TLS certificates are about to expire or have expired: Do the procedures in Rotate CA Certificates Manually below. This rotates both the TLS certificates and the internal certificates.
   • If your internal certificates for single node or leader-follower service instances are about to expire or have expired: Do the procedure in Rotate Internal Certificates below.
     
     
   • If only your internal certificates for HA clusters are about to expire or have expired: Upgrade to MySQL for PCF v2.7. See Upgrading MySQL for PCF.

     Access BOSH CredHub

Ops Manager stores some of its internal CA (certificate authority) and non-CA certificates in the BOSH CredHub credentials store. For more information, see BOSH CredHub.

To access the BOSH CredHub credentials store, you must retrieve credentials from the BOSH Director and then use the credentials to log in to CredHub from the Ops Manager VM.

To access the BOSH CredHub:

1. In the Ops Manager Installation Dashboard, click the BOSH Director tile.

2. Click the Credentials tab.

3. In the BOSH Director section, click the link to the BOSH Commandline Credentials.

4. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
   For example:

   {"credential":"BOSH_CLIENT=ops_manager
   BOSH_CLIENT_SECRET=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
   BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
   BOSH_ENVIRONMENT=10.0.0.5 bosh "}
   

   The BOSH_CLIENT is the BOSH CredHub client name and the BOSH_CLIENT_SECRET is the BOSH CredHub client secret.

5. Follow the procedure in Gather Credential and IP Address Information to obtain the information needed to log in to the BOSH Director VM. Record the IP address for the BOSH Director and the Director Credentials.

6. Log in to the Ops Manager VM by following the procedure in Log in to the Ops Manager VM with SSH.

7. From the Ops Manager VM, set the API target of the CredHub CLI to your BOSH CredHub server by running the following command:

   credhub api  \
   https://BOSH-DIRECTOR-IP:8844 \
   --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
   

   Where BOSH-DIRECTOR-IP is the IP address of the BOSH Director VM you recorded above.
   
   For example:

   $ credhub api \
   https://10.0.0.5:8844 \
   --ca-cert=/var/tempest/workspaces/default/root_ca_certificate

8. Log in to CredHub by running the following command:

   credhub login \
   --client-name=CREDHUB-CLIENT-NAME \
   --client-secret=CREDHUB-CLIENT-SECRET
   

   Where:
   

   • CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT earlier in this procedure.
   • CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET earlier in this procedure.

   For example:

   $ credhub login \
   --client-name=credhub \
   --client-secret=abcdefghijklm123456789

Check Expiration Dates

You can check the expiration date for CA and MySQL for PCF certificates.

To check certificate expiration dates:

1. Log in to CredHub by following the procedure in Access BOSH CredHub above.

2. Run:

   credhub get -n OLD-CERT -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"
   

   Where OLD-CERT is the CredHub name or path to the certificate you are checking.
   
   For example:

   $ credhub get -n /services/tls_ca -j | jq -r .value.ca | \
       openssl x509 -text -noout | grep -A 2 "Validity"
   Validity
             Not Before: Jan 23 19:04:58 2019 GMT
             Not After : Jan 23 19:04:58 2020 GMT
   

   The above command retrieves the certificate from CredHub, decrypts it, parses its JSON structure, and finds its Validity properties. The expiration date for the certificate is the value of Not After.
   
   

3. If any of your certificates are about to expire or have expired, do one of the following:

   • If your TLS certificates are about to expire or have expired: Do the procedures in Rotate CA Certificates Manually below. This rotates both the TLS certificates and the internal certificates.
   • If only your internal certificates are about to expire or have expired: Follow the procedure in Rotate Internal Certificates below.

Rotate CA Certificates Manually

Warning: This procedure assumes that you use only one foundation and do not have a Wide Area Network (WAN) setup. For a WAN setup, all of the apply changes steps need to be applied to your entire WAN. Also, all certificates must include Subject Alternate Name (SAN) and Common Name (CN) entries.

To manually rotate the CA certificate:

1. Obtain or Generate a New CA Certificate
2. Add New Certificates to Ops Manager
3. Apply Changes
4. Rebind Your Apps
5. Set New CA Certificate
6. Remove the Old CA Certificate
7. Apply Changes
8. Rebind Your Apps

Obtain or Generate a New CA Certificate

To obtain or generate a new CA certificate:

1. Check if a certificate already exists by running:

   credhub get -n NEW-CERT

   Where NEW-CERT is your chosen CredHub path or name for the new certificate.

2. If a certificate already exists, delete the certificate by running:

   credhub delete -n NEW-CERT

3. Do one of the following:

   • If you can obtain a CA (certificate authority): Obtain the CA that signs its certificate. You add this to the Ops Manager settings in the following section.

   • If you use self-signed certificates: Generate a new self-signed certificate with a separate name or path in CredHub:

     credhub generate \
     --name="NEW-CERT" \
     --type="certificate" \
     --no-overwrite \
     --is-ca \
     --common-name="COMMON-NAME"
     

     Where:

     • NEW-CERT is your chosen CredHub path or name to the new certificate.
     • COMMON-NAME is the common name of the generated certificate.

     For example:

     $ credhub generate \
       --name="/services/new_ca" \
       --type="certificate" \
       --no-overwrite \
       --is-ca \
       --common-name="servicesCA"

   • If you use an intermediate certificate signed by a root CA in credhub: Generate a new certificate signed by the CredHub root CA by running:

     credhub generate \
       --name="NEW-CERT" \
       --type="certificate" \
       --no-overwrite \
       --is-ca \
       --common-name="COMMON-NAME" \
       --ca=PATH-TO-ROOT-CA

     Where:
     • COMMON-NAME is the common name of the generated certificate.

     • PATH-TO-ROOT-CA is the name of CA used to sign the generated certificate.

       For example:

       $ credhub generate \
         --name="/services/new_ca" \
         --type="certificate" \
         --no-overwrite \
         --is-ca \
         --common-name="servicesCA" \
         --ca=/path-to-root-ca

Add Certificates to Ops Manager

To add your new and current CA certificates to Ops Manager:

1. Log in to CredHub by following the procedure in Access BOSH CredHub above.

2. Get the current CA by running:

   credhub get --name=OLD-CERT -k ca
   

   Where OLD-CERT is the CredHub name or path to the certificate to later replace.
   
   For example:

   credhub get --name=/services/tls_ca -k ca
   

3. Get the new CA, either from a pre-existing file, or from your new CredHub location by running:

   credhub get --name=NEW-CERT -k ca
   

   Where NEW-CERT is the CredHub generated or pre-existing certificate created in the previous section.
   
   For example:

   credhub get --name=/services/new_ca -k ca
   

4. Paste both CA values into Ops Manager > BOSH Director > Security > Trusted Certificates.

5. Click Save.

6. Paste both CA values into PAS tile > Networking > Certificate Authorities Trusted by Router and HAProxy.

7. Click Save.

Apply Changes

Warning: This procedure involves restarting all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.

To apply these changes:

1. Navigate back to the Installation Dashboard.
2. Click Review Pending Changes.
3. Expand errands.
4. Select the errand to upgrade all service instances.
5. Click Apply Changes.

Rebind Your Apps

If your apps do not use the trusted store to validate server certificates, you must rebind apps to the service instance to receive the updated CA certificate. Apps must be written in Spring to use the trusted store.

Warning: If a developer rebinds an app to the MySQL for Pivotal Platform service after unbinding, they must also rebind any existing custom schemas to the app. When you rebind an app, stored code, programs, and triggers break. For more information about binding custom schemas, see Use Custom Schemas.

To rebind your app:

1. Stop your app by running the following command:

   cf stop APP-NAME
   

   Where APP-NAME is the name of your app.

2. Unbind your app from the service instance by running the following command:

   cf unbind-service APP-NAME SERVICE-INSTANCE-NAME
   

   Where:

   • APP-NAME is the name of your app.
   • SERVICE-INSTANCE-NAME is the name of your service instance.

3. Rebind your app to the service instance by running the following command:

   cf bind-service APP-NAME SERVICE-INSTANCE-NAME
   

   Where:

   • APP-NAME is the name of your app.
   • SERVICE-INSTANCE-NAME is the name of your service instance.

4. Restage your app by running the following command:

   cf restage APP-NAME
   

   Where APP-NAME is the name of your app.

Set New CA Certification

To have all instances use the new CA Certificate:

1. Do one of the following:

   • If you have an existing certificate: Obtain the CA certificate and private key file corresponding to the CA that you applied in the previous step. Then, run:

     credhub set \
     --name="OLD-CERT" \
     --type="certificate" \
     --certificate=PEM-PATH/root.pem \
     --private=CERT-KEY
     

     Where:

     • OLD-CERT is the CredHub name or path to the certificate to replace.
     • PEM-PATH is the path to the root.pem file.
     • CERT-KEY is the private key for the new certificate.

     For example:

     $ credhub set \
       --name="/services/tls_ca" \
       --type="certificate" \
       --certificate=PEM-PATH/root.pem \
       --private=CERT-KEY
     

   • If you have created a new self-signed or intermediary certificate: Set OLD-CERT to the new CredHub value NEW-CERT that you generated in the previous step:

     credhub get -n NEW-CERT -k ca > new_ca.ca
     credhub get -n NEW-CERT -k certificate > new_ca.certificate
     credhub get -n NEW-CERT -k private_key > new_ca.private_key
     credhub set -n OLD-CERT \
     --type=certificate \
     --root=new_ca.ca \
     --certificate=new_ca.certificate \
     --private=new_ca.private_key
     

     For example:

     $ credhub get -n /services/new_ca -k ca > new_ca.ca
     $ credhub get -n /services/new_ca -k certificate > new_ca.certificate
     $ credhub get -n /services/new_ca -k private_key > new_ca.private_key
     $ credhub set -n /services/tls_ca \
       --type=certificate \
       --root=new_ca.ca \
       --certificate=new_ca.certificate \
       --private=new_ca.private_key
     

Remove the Old CA Certificate

After your apps have reconnected to service instances with certificates generated by the new CA, remove the old CA certificate:

1. Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile.

2. Click Security.

3. Delete the old CA certificate in Trusted Certificates.

4. Click Save.

5. Navigate to the Installation Dashboard in Ops Manager and click the PAS tile.

6. Click Networking.

7. Delete the old CA certificate in Certificate Authorities Trusted by Router and HAProxy

8. Click Save.

Apply Changes

Warning: This procedure involves restarting all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.

To apply these changes:

1. Navigate back to the Installation Dashboard.
2. Click Review Pending Changes.
3. Expand errands.
4. Select the errand to upgrade all service instances.
5. Click Apply Changes.

Rebind Your Apps

Again, if your apps do not use the trusted store to validate server certificates, you must rebind your apps to incorporate this last round of changes. See Rebind Your Apps above.

Rotate Internal Certificates

When you upgrade to MySQL for PCF v2.6, MySQL for PCF automatically rotates the internal certificates single node and leader-follower service instances. The new certificates do not expire for 5 years. The below commands upgrade all of your MySQL for service instances and distribute the new certificates to your service instances.

To ensure that the internal certificates rotate, do one of the following:

• If you have not upgraded to MySQL for PCF v2.6: Do the following:

  1. Navigate to the Ops Manager Installation Dashboard and click the MySQL for Pivotal Cloud Foundry v2 tile.
  2. Click Errands.

  3. Under Post-Deploy Errands, ensure that Upgrade all On-demand MySQL Service Instances is set to Default (On).

     

  4. Click Save.

• If you have already upgraded to MySQL for PCF v2.6 without the Upgrade all On-demand MySQL Service Instances errand on: Run the following command:

  bosh -d p-mysql-GUID run-errand upgrade-all-service-instances
  

  Where pivotal-mysql-GUID is the BOSH deployment name for your MySQL service broker.

  For a list of certificates used by MySQL for PCF, see Certificates Used by MySQL for PCF above.