Rotating Certificates

Note: This version of MySQL for Pivotal Platform is no longer supported because it has reached the End of General Support phase. To stay up to date with the latest software and security updates, upgrade to a supported version.

Page last updated:

This topic describes how to check expiration dates and rotate certificates used by MySQL for PCF.

Types of Certificates

MySQL for PCF uses the following types of certificates:

  • TLS certificates managed by the operator: If you are using TLS for MySQL for PCF, you provided a CA certificate to BOSH CredHub when you did the procedures in Preparing for TLS. Operators must rotate this TLS CA certificate before the certificate expires or if it is compromised.
  • Internal certificates managed by MySQL for PCF: In MySQL for PCF v2.3, you cannot rotate internal certificates. To rotate these certificates, you must upgrade to MySQL for PCF v2.6. See Upgrading MySQL for PCF. To check the expiration date of your internal certificates, see Check Expiration Date below.

For a list of the certificates used by MySQL for PCF certificates, see Certificates Used by MySQL for PCF below.

Warning: Do not attempt to rotate a CA certificate on your own. Contact Pivotal Support and perform the following procedures with their help.

Certificates Used by MySQL for PCF

The following table lists the certificates used by MySQL for PCF:

Certificate
TLS Certificates
/services/tls_ca
mysql_tls
mysql_server_tls
Internal Certificates
/pivotal_mysql_GUID/agent_ca
/pivotal_mysql_GUID/agent_client_ssl
/pivotal_mysql_GUID/agent_server_ssl
/service_instance_GUID/agent_ca
/service_instance_GUID/lf_agent_client_ssl
/service_instance_GUID/lf_agent_server_ssl
/service_instance_GUID/agent_client_ssl
/service_instance_GUID/agent_server_ssl
/service_instance_GUID/streaming_backup_ca
/service_instance_GUID/streaming_backup_server_cert

In the above table, GUID is the GUID for the service instance. To find the GUID of your service instance, do the procedure in Find Information about Your Service Instance.

Check Expiration Dates

Pivotal Cloud Foundry (PCF) stores the certificates used by MySQL for PCF and then use the credentials to log in to CredHub from the Ops Manager VM. You can use the CredHub API to check the expiration date for the certificates used by MySQL for PCF. However, you can only rotate the TLS CA certificate in MySQL for PCF v2.3. See Rotate the TLS CA Certificate below.

Warning: If your internal certificates expire, service bindings and backups can fail.

To use the CredHub API, you must retrieve credentials from BOSH Director and then use the credentials to log in to CredHub from the Ops Manager VM.

For more information about CredHub, see BOSH CredHub.

To check certificate expiration dates:

  1. If the CredHub CLI is not installed on your local machine, install the CredHub CLI by doing the instructions in credhub-cli on GitHub.

  2. In the Ops Manager Installation Dashboard, click the BOSH Director tile.

  3. Click the Credentials tab.

  4. In the BOSH Director section, click the link to the BOSH Commandline Credentials. CredHub Credentials

  5. Record the values for BOSH_CLIENT and BOSH_CLIENT_SECRET.
    For example:

    {"credential":"BOSH_CLIENT=ops_manager
    BOSH_CLIENT_SECRET=abCdE1FgHIjkL2m3n-3PqrsT4EUVwXy5
    BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate
    BOSH_ENVIRONMENT=10.0.0.5 bosh "}
    

    The BOSH_CLIENT is the BOSH CredHub client name and the BOSH_CLIENT_SECRET is the BOSH CredHub client secret.

  6. Record the information needed to log in to the BOSH Director VM by doing the procedure in Gather Credential and IP Address Information.

  7. Log in to the Ops Manager VM by doing the procedure in Log in to the Ops Manager VM with SSH.

  8. From the Ops Manager VM, set the API target of the CredHub CLI to your BOSH CredHub server by running the following command:

    credhub api https://BOSH-DIRECTOR:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

    Where BOSH-DIRECTOR is the IP address of the BOSH Director VM you recorded above.

    For example:

    $ credhub api https://10.0.0.5:8844 --ca-cert=/var/tempest/workspaces/default/root_ca_certificate
    

  9. Log in to CredHub by running the following command:

    credhub login --client-name=CREDHUB-CLIENT-NAME --client-secret=CREDHUB-CLIENT-SECRET
    

    Where:

    • CREDHUB-CLIENT-NAME is the value you recorded for BOSH_CLIENT in step 5 of this procedure.
    • CREDHUB-CLIENT-SECRET is the value you recorded for BOSH_CLIENT_SECRET in step 5 of this procedure.

    For example:

    $ credhub login \
    --client-name=credhub \
    --client-secret=abcdefghijklm123456789
    

  10. For each certificate in the Certificates Used by MySQL for PCF list above, run the following command:

    credhub get -n CERTIFICATE-NAME -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"
    

    Where CERTIFICATE-NAME is the name of the certificate you are checking.

    For example:

    $ credhub get -n /services/tls_ca -j | jq -r .value.ca | openssl x509 -text -noout | grep -A 2 "Validity"
    Validity
              Not Before: Jan 23 19:04:58 2019 GMT
              Not After : Jan 23 19:04:58 2020 GMT
    

    The above command retrieves the certificate from CredHub, parses its JSON structure, decrypts it, and finds its Validity properties. The expiration date for the certificate is the value of Not After. The expiration date can be different for different certificates.

  11. If any of your certificates are about to expire or have expired, do the following:

Rotate the TLS CA Certificate

This following procedures rotate a TLS CA certificate. You cannot rotate your internal certificates with these procedures. To rotate internal certificates, upgrade to MySQL for PCF v2.6. See Upgrading MySQL for PCF.

To rotate CA certificates, you must do the following:

  1. Add a New CA Certificate
  2. Remove the Old TLS CA Certificate
  3. Rebind Your App

Add a New TLS CA Certificate

Warning: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

To add a new TLS CA Certificate to Ops Manager:

  1. Log in to CredHub by doing steps 1 through 9 in Check Expiration Dates above.

  2. Set the TLS CA certificate by doing one of the following:

    Note: Your PCF deployment can have multiple CA certificates. Pivotal recommends using a dedicated CA certificate for each service.

    • If you do not have a TLS CA certificate: generate a new certificate by running the following command:

      credhub regenerate --name="/services/tls_ca"
      
    • If you have an existing certificate: create a new root.pem file with the contents of the certificate. Then, run the following command:

      credhub set \
          --name="/services/tls_ca" \
          --type="certificate" \
          --certificate=PEM-PATH/root.pem \
          --private=CERT-KEY...
      

      Where:

      • PEM-PATH is the path to the root.pem file.
      • CERT-KEY is the private key for the certificate.
  3. Extract and output the certificate portion from the TLS CA certificate by running the following command:

    bosh int <(credhub get \
    --name=/services/tls_ca) \
    --path /value/certificate
    
  4. Record the output.

  5. Navigate to the Ops Manager Installation Dashboard and click the BOSH Director tile.

  6. Click Security.

  7. Append the contents of the new TLS CA certificate to the old CA certificate under Trusted Certificates. Do not remove the old TLS CA certificate.

  8. Click Save.

  9. Return to the Installation Dashboard in Ops Manager and click Apply Changes.

Remove the Old CA Certificate

Warning: This procedure involves restarting all of the VMs in your PCF deployment to apply a CA certificate. The operation can take a long time to complete.

If you are rotating your TLS CA certificate because it was compromised, you must remove your old TLS CA certificate before rebinding your app.

To remove the old TLS CA certificate:

  1. Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile.

  2. Click Security.

  3. Delete the old TLS CA certificate in Trusted Certificates.

  4. Click Save.

  5. Return to the Installation Dashboard in Ops Manager and click Apply Changes. This restarts all the VMs in your deployment.

Rebind Your App

If your apps do not use the trusted store to validate certificates, developers must rebind apps to the service instance to receive the updated TLS CA certificate. Apps not written in Java and Spring do not use the trusted store.

To rebind your app:

  1. Stop your app by running the following command:

    cf stop YOUR-APP
    

    Where YOUR-APP is the name of your app.

  2. Unbind your app from the service instance by running the following command:

    cf unbind-service YOUR-APP YOUR-SERVICE-INSTANCE
    

    Where:

    • YOUR-APP is the name of your app.
    • YOUR-SERVICE-INSTANCE is the name of your service instance.
  3. Re-bind your app to the service instance by running the following command:

    cf bind-service YOUR-APP YOUR-SERVICE-INSTANCE
    

    Where:

    • YOUR-APP is the name of your app.
    • YOUR-SERVICE-INSTANCE is the name of your service instance.
  4. Restage your app by running the following command:

    cf restage YOUR-APP
    

    Where YOUR-APP is the name of your app.