Rotating the Services TLS CA and Its Leaf Certificates

Page last updated:

This topic provides an overview of the Services TLS Certificate Authority (CA) and a procedure for rotating the Services TLS CA and its leaf certificates.

Services TLS CA Overview

In an Ops Manager and Pivotal Application Service (PAS) deployment, the Services TLS CA is the certificate that service brokers use to sign leaf certificates for TLS-enabled service instances.

Service instances include instances created by on-demand services such as MySQL for Pivotal Platform, Pivotal Cloud Cache, Redis for Pivotal Platform, and RabbitMQ for Pivotal Platform. You may have one or more of these service tiles installed in your deployment. All deployed service tiles in a foundation use the same Services TLS CA certificate, which is stored in BOSH CredHub with the name /services/tls_ca.

When an operator creates an on-demand TLS enabled service instance or updates an existing non-TLS on-demand service instance to use TLS, the following takes place:

  1. BOSH generates a leaf certificate signed by the Services TLS CA.

  2. BOSH then stores this new leaf certificate in BOSH CredHub with the format /p-bosh/service-instance_SERVICE-GUID/unique-name-per-service-type.

  3. BOSH supplies the private key and certificates to the VMs of that service instance.

In the following procedure, BOSH rotates leaf certificates when you select the upgrade all service instances errand for an on-demand service tile and apply changes in Ops Manager.

Prerequisites for Rotating the Services TLS CA

Before you begin this rotation procedure, ensure that you have the following:

  • Access to the Ops Manager VM, BOSH CredHub and CredHub CLI. For more information, see Accessing BOSH CredHub with the CredHub CLI.
  • Only one Ops Manager foundation and not a multi-site or Wide Area Network (WAN) setup. For a multi-site or WAN setup, you must perform the Apply Changes steps across all your sites or WAN.
  • All certificates include Subject Alternate Name (SAN) and Common Name (CN) entries.
  • Your services are set for high availability to minimize potential app impact during CA rotation. Rotating the Services TLS CA requires an upgrade all on-demand service instances, which can cause service downtime.
Warning: There is a potential for app downtime in the following specific cases:

  • If you have non-Java apps that use MySQL for Pivotal Platform and rely on VCAP_SERVICES for TLS, you will experience app downtime if you perform the CA rotation described in this procedure. To avoid app downtime, follow the procedure in this Knowledgebase Article instead of the procedure in this topic. Only follow the procedure in this topic if all of your MySQL apps use jdbcURL or consume CA certificates from /etc/ssl/certs/ca-certificate.crt.
  • If you are using Redis for Pivotal Platform v2.2 or v2.3 and use this procedure to rotate the Services TLS CA certificate, you may experience downtime for apps that bind to Redis service instances. To workaround the issue, see this Knowledgebase Article.

Procedure for Rotating the Services TLS CA

This section provides the procedure to rotate the Services TLS CA and update all its leaf certificates.

Verify the Expiration Status of Services TLS CA Certificate

Before starting the rotation, check that the Services TLS CA has not already expired.

To check the expiration date:

  1. Log in to CredHub by following the procedure in Accessing BOSH CredHub with the CredHub CLI.

  2. Run:

    credhub get -n /services/tls_ca -j | jq -r .value.ca \
     | openssl x509 -text -noout | grep -A 2 "Validity"
    

    For example:

    Validity
              Not Before: Jan 23 19:04:58 2019 GMT
              Not After : Jan 23 19:04:58 2020 GMT
    

    The above command retrieves the certificate from CredHub, decrypts it, parses its JSON structure, and finds its Validity properties. The expiration date for the certificate is the value of Not After.

  3. If your Services TLS CA certificate has already expired, follow the procedure in the Knowledgebase Article to rotate the certificate instead of the procedure in this topic.

Obtain or Generate a New Services TLS CA Certificate

To obtain or generate a new Services TLS CA certificate:

  1. Check if CredHub has a new temporary certificate from a previous rotation attempt. Run:

    credhub get -n /services/new_ca

    Note: This procedure assumes that /services/new_ca is the CredHub location used when generating the temporary certificate. If you used a different location, specify that location instead.

  2. If a older temporary certificate already exists, delete that certificate by running:

    credhub delete -n /services/new_ca

  3. Do one of the following:

    • If you have a TLS cluster: Obtain the CA that signs its certificate. You add this to the Ops Manager settings in the next high-level step.
    • If you use self-signed certificates: Generate a new self-signed certificate with a new name or path in CredHub:

      credhub generate \
      --name="/services/new_ca" \
      --type="certificate" \
      --no-overwrite \
      --is-ca \
      --duration=1825 \
      --common-name="opsmgr-services-tls-ca"
      

      Where:

      • name is the CredHub path or name of the new certificate. You can use /services/new_ca or substitute a different name as long as you use the value consistently in the rest of the procedure.
      • common-name is the common name of the generated certificate. You can use opsmgr-services-tls-ca or substitute a different common name as long as you use the value consistently in the rest of the procedure.
      • duration is set to 1825, which is a recommended value of five years. If you do not specify a duration, the default value is 365 or one year.
    • If you use an intermediate certificate signed by a root CA in CredHub: Generate a new certificate signed by the CredHub root CA by running:

      credhub generate \
      --name="/services/new_ca" \
      --type="certificate" \
      --no-overwrite \
      --is-ca \
      --duration=1825 \
      --common-name="opsmgr-services-tls-ca" \
      --ca=/PATH-TO-ROOT-CA
      

      Where:

      • name is the CredHub path or name of the new certificate. You can use /services/new_ca or substitute a different name as long as you use the value consistently in the rest of the procedure.
      • common-name is the common name of the generated certificate. You can use opsmgr-services-tls-ca or substitute a different common name as long as you use the value consistently in the rest of the procedure.
      • duration is set to 1825, which is a recommended value of five years. If you do not specify a duration, the default value is 365 or one year.
      • ca is the path to the root CA for the new certificate. Replace PATH-TO-ROOT-CA with the appropriate path for the root CA.

Add Certificates to Ops Manager and PAS

To add your new and current Services TLS CA certificate to Ops Manager:

  1. Log in to CredHub.

  2. Get the current Services TLS CA by running:

    credhub get --name=/services/tls_ca -k ca
    
  3. Get the new CA, either from a pre-existing file, or from your new CredHub location by running:

    credhub get --name=/services/new_ca -k ca
    
  4. In Ops Manager, paste both CA values into the BOSH Director > Security > Trusted Certificates field.

  5. Click Save.

  6. In the PAS and Pivotal Isolation Segment tiles, paste both CA values into the Networking > Certificate Authorities trusted by the Gorouter and Certificate Authorities trusted by the HAProxy fields.

  7. Click Save.

Apply Changes for the First Time

Warning: This procedure involves redeploying all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.

To apply these changes:

  1. Navigate back to the Installation Dashboard.
  2. Click Review Pending Changes.
  3. Ensure that all product tiles, including PAS, PASW, Isolation Segment, and partner tiles, are selected.
  4. Verify which on-demand service tiles are using the Services TLS CA certificate. Run:

    credhub curl -p /api/v1/certificates?name=%2Fservices%2Ftls_ca
    
  5. From the returned list, identify which on-demand service tiles use TLS in your deployment, such as MySQL for Pivotal Platform, Pivotal Cloud Cache, Redis for Pivotal Platform, and RabbitMQ for Pivotal Platform.

  6. For each on-demand service tile that uses TLS:

    1. Expand the errands.
    2. Enable the errand to upgrade all service instances.

      Note: The name of the upgrade all service instances errand may differ slightly between services.

  7. Click Apply Changes.

Set New Services TLS CA Certificate

To set the new Services TLS CA certificate:

  1. Do one of the following:

    • If you have an existing certificate: Obtain the CA certificate and private key file corresponding to the CA that you applied in the previous step. Then, run:

      credhub set \
      --name="/services/tls_ca" \
      --type="certificate" \
      --certificate=PEM-PATH/root.pem \
      --private=CERT-KEY
      

      Where:

      • certificate is the location of the root.pem file for the certificate. Replace PEM-PATH with the path of your certificate’s root.pem file.
      • private is the private key for the new certificate. Replace CERT-KEY with the value of the private key for your certificate.
    • If you have created a new self-signed or intermediary certificate: Set the /services/new_ca that you generated in the previous step as the Services TLS CA:

      credhub get -n /services/new_ca -k ca > new_ca.ca
      credhub get -n /services/new_ca -k certificate > new_ca.certificate
      credhub get -n /services/new_ca -k private_key > new_ca.private_key
      credhub set -n /services/tls_ca \
      --type=certificate \
      --root=new_ca.ca \
      --certificate=new_ca.certificate \
      --private=new_ca.private_key
      

Apply Changes for the Second Time

Warning: This procedure involves redeploying all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.

In this step, you apply changes to ensure that all on-demand service instances generate new leaf certificates from the new Services TLS CA.

To apply these changes:

  1. Navigate back to the Installation Dashboard.
  2. Click Review Pending Changes.
  3. Ensure that all product tiles, including PAS, PASW, Isolation Segment, and partner tiles, are deselected in order to reduce deployment time.
  4. Select only the on-demand services tiles that use TLS, such as MySQL for Pivotal Platform, Pivotal Cloud Cache, Redis for Pivotal Platform, and RabbitMQ for Pivotal Platform.
  5. For each on-demand service tile that use TLS:
    1. Expand the errands for each tile.
    2. Enable the errand to upgrade all service instances.

      Note: The name of the upgrade all service instances errand may differ slightly between services.

  6. Click Apply Changes.

Remove the Old Services TLS CA Certificate

After your apps have reconnected to service instances with certificates generated by the new CA, remove the old CA certificate:

  1. Navigate to the Installation Dashboard in Ops Manager and click the BOSH Director tile.

  2. Click Security.

  3. Delete the old CA certificate in Trusted Certificates.

  4. Click Save.

  5. Navigate to the Installation Dashboard in Ops Manager and click the PAS tile.

  6. Click Networking.

  7. Delete the old CA certificate in the Certificate Authorities trusted by the Gorouter and Certificate Authorities trusted by the HAProxy fields.

  8. Click Save.

Apply Changes for the Third Time

As a security best practice, you should remove outdated certificates as soon as possible from your deployment. You can schedule this step to a convenient time because for most deployments you will not lose any deployment functionality if you do not perform this step immediately.

For some deployments, you may encounter an error if you create a service instance in a deployment that contains an expired Services TLS CA certificate. For more information, see this Knowledgebase Article.

Warning: This procedure involves redeploying all of the VMs in your Ops Manager deployment to apply a CA certificate. The operation can take a long time to complete.

To apply these changes:

  1. Navigate back to the Installation Dashboard.
  2. Click Review Pending Changes.
  3. Ensure that all product tiles, including PAS, PASW, Isolation Segment, and partner tiles, are unchecked.
  4. Select only the on-demand services tiles that use TLS, such as MySQL for Pivotal Platform, Pivotal Cloud Cache, Redis for Pivotal Platform, and RabbitMQ for Pivotal Platform.
  5. For each on-demand service tile that uses TLS:
    1. Expand the errands for each tile.
    2. Enable the errand to upgrade all service instances.

      Note: The names of upgrade all service instances errand may differ slightly between services.

  6. Click Apply Changes.