Rotating the Cloud Controller Database Encryption Key

Page last updated:

This topic describes how to rotate the Cloud Controller database (CCDB) encryption key.

Overview

The Cloud Controller is the primary API of Pivotal Application Service (PAS). It has its own database, the CCDB, in which it stores information about objects in PAS such as apps. The CCDB encryption key is used to encrypt sensitive data at rest in the CCDB such as app environment variables.

The CCDB encryption key is automatically generated when you first deploy PAS. You can rotate the CCDB encryption key as described in this topic. For example, you may want to perform this procedure if your existing key is leaked.

Rotate the CCDB Encryption Key

Warning: Do not modify the Cloud Controller database encryption key field in the PAS tile. This field is only for use when restoring PAS from a backup. If you modify this value at a time other than restore, commands such as cf push may fail. Instead, change keys using the Encryption key ledger as described below.

To rotate the CCDB encryption key:

  1. Navigate to the PAS tile in Ops Manager and select the Cloud Controller pane.

  2. Review the Encryption key ledger field. If you previously added a key and enabled the Primary checkbox, disable the checkbox. The PAS deploy fails if there is more than one key with the Primary checkbox enabled.

  3. In the Encryption key ledger field:

    Note: Do not remove any existing keys until after you complete the last step of this procedure. This causes the PAS deploy to fail.

    1. Click Add.
    2. Enter values for Encryption key label and Encryption key.
    3. Enable the checkbox to mark this as your Primary key.
  4. Click Save.

  5. Select the Errands pane.

  6. Set the Rotate CC Database Key errand to On.

  7. Click Save.

  8. Redeploy PAS. Once PAS deploys and the errand runs successfully, you can remove any previously specified CCDB encryption keys in the ledger.

Troubleshoot Failed Rotation

This section describes how to troubleshoot errors that you may encounter while rotating the CCDB encryption key.

Error: Encryption key(s) have had their values changed

Error The PAS deploy fails with a Cloud Controller error. The Cloud Controller logs include the following message: Encryption key(s) mykey have had their values changed. Label and value pairs should not change, rather a new label and value pair should be added.
Cause PAS does not support editing key names or values in the ledger. A previously existing key was edited in the PAS UI. For example, a key with the name key-name previously had the value key-value-1 and now it is key-value-2.
Solution Replace the value for the key with its original value and follow the rotation procedure to add a new key.
You may need to find the original value of the key using the steps in View Encryption Keys.

Error: Encryption key(s) are still in use but not present in ‘cc.database_encryption.keys’

Error The PAS deploy fails with a Cloud Controller error. The Cloud Controller logs include the following message: Encryption key(s) mykey are still in use but not present in 'cc.database_encryption.keys'
Cause A previous key was removed from the ledger in the PAS UI before the key rotation errand ran.
Solution Add the previously used key back to the ledger and redeploy PAS.
You may need to find the value of the previously used key using the steps in View Encryption Keys.

View Encryption Keys

The section describes how to retrieve the value of an existing or previous encryption key. You may need to do this when troubleshooting a failed key rotation.

Ops Manager actual-installation.yml

The Ops Manager actual-installation.yml file shows properties from the most recent successful deploy. To view the encryption keys listed in this file:

  1. Follow the steps in the Modify the Installation Files section of the Modifying Your Ops Manager Installation and Product Template Files topic to decrypt the actual-installation.yml file.

  2. Open the actual-installation.yml file and view the cloud_controller_encryption_keys property.

CredHub

To view current and previous encryption keys using CredHub:

  1. Follow the steps in How to access CredHub with the CredHub CLI in the Pivotal Knowledge Base to log in to the CredHub CLI.

  2. Run the following command to view the five most recent encryption keys:

    credhub get -n /opsmgr/PAS-GUID/cloud_controller/INDEX/encryption_keys --versions=5
    Where:

    • PAS-GUID is the GUID of your PAS BOSH deployment.
    • INDEX is the zero-based index of the key. This corresponds to the order of the Encryption key ledger in the Ops Manager UI.