Backing Up Pivotal Cloud Foundry with BBR

Page last updated:

This topic describes the procedure for backing up your critical backend Pivotal Cloud Foundry (PCF) components with BOSH Backup and Restore (BBR), a command-line tool for backing up and restoring BOSH deployments. To restore your backup, see the Restoring Pivotal Cloud Foundry from Backup with BBR topic.

To view the BBR release notes, see the BOSH Backup and Restore Release Notes.

During the backup, BBR stops the Cloud Controller API and the Cloud Controller workers to create a consistent backup. Only the API functionality, like pushing applications or using the Cloud Foundry Command Line Interface (cf CLI) are affected. The deployed applications do not experience downtime.

Note: You can only use BBR to back up PCF v1.11 and later. To back up earlier versions of PCF, perform the manual procedures.

WARNING
  • Backup artifacts can contain secrets. Secure backup artifacts by using encryption or other means.
  • The restore is a destructive operation. BBR is designed to restore PCF after a disaster. If it fails, the environment may be left in an unusable state and require reprovisioning. For more information, see Restoring Pivotal Cloud Foundry from Backup with BBR.
  • The Cloud Controller API will stop sending and receiving calls for the duration of PCF restoration.
  • BBR does not back up any service data.
  • Using an external MySQL database results in an unusable restore even if the restore appears to run successfully.
  • BBR requires an add-on to back up and restore external blobstores. For more information, see Enabling External Blobstore Backups.
  • For PCF v2.0.0, BBR only supports backup and restore of environments with zero or one CredHub instances. PAS v2.0.17 resolves this issue.

Recommendations

Pivotal recommends the following:

  • Follow the full procedure documented in this topic when creating a backup. This ensures that you always have a consistent backup of Ops Manager and PAS to restore from.
  • Back up frequently, especially before making any changes to your PCF deployment, such as the configuration of any tiles in Ops Manager.

Compatibility of Restore

When using a backup to restore, you must ensure that the restore environment is compatible. For more information, see Compatibility of Restore.

Supported Components

BBR is a binary that can back up and restore BOSH deployments and BOSH Directors. BBR requires that the backup targets supply scripts that implement the backup and restore functions.

BBR backs up the following PCF components:

  • PAS: PAS must be configured with an internal MySQL database and a WebDAV/NFS blobstore to be backed up and restored with BBR. BBR does not support PAS with an external blobstore or an external MySQL database.
      Warning: The following guidelines apply when you are using BBR to back up PCF and have configured an external blobstore and/or external database in PAS:
      • If you configured an internal database and an external blobstore in PAS, then follow the PCF backup process, and copy the external blobstore via the IaaS. There may be inconsistencies between the blobstore and database, which may result in some apps not coming up during a restore. Those apps can be repushed.
      • If you configured an external database and an external blobstore in PAS, then follow the PCF backup process, but skip the backup of PAS. Copy the external database and blobstore via the IaaS. There may be inconsistencies between the blobstore and database, which may result in some apps not coming up during a restore. Those apps can be repushed.
Note:
  • Backup artifacts of external, S3-compatible, versioned blobstores do not contain the physical blobs. BBR requires that the original buckets still exist to be restored.
  • To protect yourself from losing a bucket, see Enable Replication on Your External Blobstore in Backup and Restore with External Blobstores in the Cloud Foundry documentation.

Unsupported Blobstores and Databases in PAS

If you configured an unsupported external blobstore or an unsupported external database in PAS, see the following guidelines to perform backup successfully:

Scenario Action
You configured a supported database and an unsupported external blobstore in PAS. Follow the PCF backup process, and copy the external blobstore by using the IaaS-specific tool.
You configured an unsupported external database and a supported blobstore in PAS. Follow the PCF backup process, and copy the external database by using the IaaS-specific tool.
You configured both an unsupported external database and an unsupported external blobstore in PAS. Follow the PCF backup process, but skip Step 10: Back Up Your PAS Deployment. Copy the external database and blobstore by using the IaaS-specific tool.

WARNING: You may encounter inconsistencies between the blobstore and database. If apps do not appear during a restore, repush those apps.

Backing Up Services

WARNING: BBR does not currently back up any service data.

Keep in mind the following when backing up services:

  • You can back up and restore brokered services with the procedures documented in this topic and in the Restoring Pivotal Cloud Foundry from Backup with BBR topic. BBR backs up and restores the virtual machines (VMs) and the service instances, but not the service data.
  • You can redeploy on-demand service instances manually during restore, but the data in the instance is not backed up.
  • BBR does not back up managed services or their data.

Workflow

Operators download the BBR binary and transfer it to a jumpbox. Then they run BBR from the jumpbox, specifying the name of the BOSH deployment to back up.

BBR examines the jobs in the BOSH deployment, and triggers the scripts in the following stages:

  1. Pre-backup lock: The pre-backup lock scripts locks the job so backups are consistent across the cluster.
  2. Backup: The backup script backs up the release.
  3. Post-backup unlock: The post-backup unlock script unlocks the job after the backup is complete.

Scripts in the same stage are all triggered together. For instance, BBR triggers all pre-backup lock scripts before any backup scripts. Scripts within a stage may be triggered in any order.

The backup artifacts are drained to the jumpbox, where the operator can transfer them to storage and use them to restore PCF.

The following diagram shows a sample backup flow.

Backup flow

Before using BBR, follow the instructions in the Setting Up Your Jumpbox for BBR section.

Prepare to Create Your Backup

Step 1: Set Up Your Jumpbox

Prepare your jumpbox for BBR by following the steps in the Setting Up Your Jumpbox for BBR topic.

Note: Pivotal recommends that you regularly update the BBR binary on your jumpbox to the latest version. See Transfer BBR Binary to Your Jumpbox in Setting Up Your Jumpbox for BBR for more information.

Step 2: Record the Cloud Controller Database Encryption Credentials

Perform the following steps to retrieve the Cloud Controller Database encryption credentials from the PAS tile:

  1. Navigate to Ops Manager in a browser and log in to the Ops Manager Installation Dashboard.

  2. Select PAS Credentials and locate the Cloud Controller section.

  3. Record the Db Encryption Credentials. You must provide these credentials when you contact Pivotal Support for assistance restoring your installation.

    Ccdb encrypt creds

Step 3: Retrieve BOSH Director Address and Credentials

Perform the following steps to retrieve the IP address of your BOSH Director and the credentials for logging in from the Ops Manager Director tile:

  1. If you are not using the Ops Manager VM as your jumpbox, install the latest BOSH CLI on your jumpbox.
  2. From the Installation Dashboard in Ops Manager, select BOSH Director > Status and record the IP address listed for the Director. You access the BOSH Director using this IP address.

  3. Click Credentials and record the Director credentials.
  4. From the command line, log into the BOSH Director using the IP address and credentials that you recorded:
    $ bosh -e DIRECTOR_IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE log-in
    Email (): director
    Password (): *******************
    Successfully authenticated with UAA
    Succeeded
    

Step 4: Check your BOSH Director

Perform the following steps to back up your BOSH Director:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the Ops Manager tile.

  3. Click the Credentials tab.

  4. Locate Bbr Ssh Credentials and click Link to Credential next to it.

    You can also retrieve the credentials using the Ops Manager API with a GET request to the following endpoint: /api/v0/deployed/director/credentials/bbr_ssh_credentials. For more information, see the Using the Ops Manager API topic.

  5. Copy the value for private_key_pem.

  6. SSH into your jumpbox:

    $ ssh -i LOCAL-PATH-TO-JUMPBOX-PRIVATE-KEY JUMPBOX-USER@JUMPBOX-ADDRESS
    

  7. Run the following command to reformat the key and save it to a file named PRIVATE-KEY in the current directory, pasting in the contents of your private key for YOUR-PRIVATE-KEY:

    $ printf -- "YOUR-PRIVATE-KEY" > PRIVATE-KEY
    

  8. Run the BBR pre-backup-check command from your jumpbox:

    $ bbr director \
      --private-key-path PRIVATE-KEY \
      --username bbr \
      --host HOST \
      pre-backup-check
    
    Replace the placeholder values as follows:

    • PRIVATE-KEY: The path to the private key file you created above
    • HOST: The address of the BOSH Director. If the BOSH Director is public, this is a URL, such as https://my-bosh.xxx.cf-app.com. Otherwise, this is the BOSH-DIRECTOR-IP, which you retrieved in the Step 3: Retrieve BOSH Director Address and Credentials section of this topic.

      Note: Use the optional --debug flag to enable debug logs. See the Logging section of this topic for more information.

  9. If the pre-backup check succeeds, continue to Step 5: Make Sure the Backup Prepare Node is Deployed below.

    If the pre-backup check fails, the Director might not have the correct backup scripts, or the connection to the BOSH Director might have failed.

Step 5: Make Sure the Backup Prepare Node is Deployed

When BBR backs up PAS, it needs a Backup Prepare node. This procedure makes sure that the Backup Prepare node is deployed.

  1. In Ops Manager, open the Pivotal Application Service tile.

  2. In the Resource Config pane find the Backup Prepare Node job and check if the Instances dropdown menu is set to 1. Backup prepare node

  3. If the Instances dropdown menu is not set to 1:

    1. Set the Instances dropdown menu to 1.
    2. Click Save.
    3. Navigate back to the Installation Dashboard and click Apply Changes to redeploy.

Step 6: Identify Your Deployment

After logging in to your BOSH Director, run the following command to identify the name of the BOSH deployment that contains PCF:

$ bosh -e BOSH-DIRECTOR-IP --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE deployments

Name                     Release(s)
cf-example               push-apps-manager-release/661.1.24
                         cf-backup-and-restore/0.0.1
                         binary-buildpack/1.0.11
                         capi/1.28.0
                         cf-autoscaling/91
                         cf-mysql/35
                         ...

In the above example, the name of the BOSH deployment that contains PCF is cf-example.

Step 7: Check Your Deployment

Perform the following steps to check that your BOSH Director is reachable and has a deployment that can be backed up:

  1. From your jumpbox, run the BBR pre-backup check:

    $ BOSH_CLIENT_SECRET=BOSH-PASSWORD \
      bbr deployment \
      --target BOSH-DIRECTOR-IP \
      --username BOSH-CLIENT \
      --deployment DEPLOYMENT-NAME \
      --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
      pre-backup-check
    
    Replace the placeholder values as follows:

    • BOSH-CLIENT, BOSH-PASSWORD: From the Ops Manager Installation Dashboard, click Ops Manager Director, navigate to the Credentials tab, and click Uaa Bbr Client Credentials to retrieve the BOSH UAA credentials.

      You can also retrieve the credentials using the Ops Manager API with a GET request to the following endpoint: /api/v0/deployed/director/credentials/uaa_bbr_client_credentials. For more information, see the Using the Ops Manager API topic.

    • BOSH-DIRECTOR-IP: You retrieved this value in the Step 3: Retrieve BOSH Director Address and Credentials section.
    • DEPLOYMENT-NAME: You retrieved this value in the Step 6: Identify Your Deployment section.
    • PATH-TO-BOSH-SERVER-CERTIFICATE: This is the path to the Certificate Authority (CA) certificate for the BOSH Director. For more information, see Ensure BOSH Director Certificate Availability.
  2. If the pre-backup check succeeds, continue to Create Your Backup below.

    If the pre-backup check fails, the deployment you selected might not have the correct backup scripts, or the connection to the BOSH Director might have failed.

Create Your Backup

Step 8: Export Installation Settings

Pivotal recommends that you back up your Ops Manager installation settings by exporting frequently.

When exporting your installation settings, keep in mind the following:

  • This option is only available after you have deployed at least once.

  • Always export your installation settings before following the steps in the Deploy Ops Manager and Import Installation Settings section of the Restoring Pivotal Cloud Foundry from Backup with BBR topic.

  • Exporting your installation settings only backs up the settings you configured in Ops Manager. It does not back up your VMs or any external MySQL databases.

  • Your Ops Manager settings are encrypted. Make sure you keep track of your Decryption Passphrase because this is needed to restore the Ops Manager settings.

Note: When exporting installation settings for Ops Manager v1.12 and later, releases are not included in the output file. Releases are now included in the BOSH director backup, therefore, the output file is much smaller than in previous Ops Manager versions.

There are two ways to export Ops Manager installation settings:

Option 1: Use the Ops Manager UI

To export your installation settings using the Ops Manager UI, do the following:

  1. From the Installation Dashboard in the Ops Manager interface, click your user name at the top right navigation.

  2. Select Settings.

  3. Select Export Installation Settings.

  4. Click the Export Installation Settings button. Settings

Option 2: Use the Ops Manager API

If you want to automate the back up process, you can use the Ops Manager API to export your installation settings.

To export your installation settings using the Ops Manager API, run the following command:

curl OPSMAN-URL/api/v0/installation_asset_collection \
  -H "Authorization: Bearer UAA-ACCESS-TOKEN" > installation.zip

For instructions about authenticating and retrieving the UAA-ACCESS-TOKEN, see Using the Ops Manager API.

Step 9: Back Up Your BOSH Director

Notes:
  • The BOSH Director backup can take more than 10 minutes.
  • Because the BBR backup command can take a long time to complete, Pivotal recommends you run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails.
    Use nohup, a screen, or a tmux session.

Run the BBR backup command from your jumpbox to back up your BOSH Director:

$ bbr director \
  --private-key-path PRIVATE-KEY \
  --username bbr \
  --host HOST \
  backup

Replace the placeholder values as follows:

If your backup terminates early or fails, clean up by running backup-cleanup:

If your backup terminates early or fails, you can clean up by running backup-cleanup:

$ bbr director \
  --private-key-path PRIVATE-KEY \
  --username bbr \
  --host HOST \
  backup-cleanup

Step 10: Back Up Your PAS Deployment

Notes:
  • Backing up PAS can take more than 5 minutes, and can take considerably longer with larger blobstores or slow network connections. The backup also incurs Cloud Controller downtime during which users are unable to push, scale, or delete apps. Your apps will not be affected.
  • Because the BBR backup command can take a long time to complete, Pivotal recommends you run it independently of the SSH session, so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup but you could also run the command in a screen or tmux session.
  1. (Optional) If you are not using an internal blobstore or a versioned S3-compatible external blobstore, create a copy of the blobstore with your IaaS specific tool. Your blobstore backup may be slightly inconsistent with your PAS backup depending on the duration of time between performing the backups.

  2. Run the BBR backup command from your jumpbox to back up your PAS deployment:

    $ BOSH_CLIENT_SECRET=BOSH-PASSWORD \
    bbr deployment \
    --target BOSH-DIRECTOR-IP \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    backup
    

    • Use the optional --debug flag to enable debug logs. For more information, see Logging below.
    • Use the optional --with-manifest flag to ensure that BBR downloads your current deployment manifest when backing up. These manifests are included in the Ops Manager export, but are useful for reference. For example:
      $ BOSH_CLIENT_SECRET=BOSH-PASSWORD \
      bbr deployment \
      --target BOSH-DIRECTOR-IP \
      --username BOSH-CLIENT \
      --deployment DEPLOYMENT-NAME \
      --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
      backup --with-manifest

If the command fails, do the following:

  1. Run backup-cleanup:

    $ BOSH_CLIENT_SECRET=BOSH-PASSWORD \
     bbr deployment \
    --target BOSH-DIRECTOR-IP \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    backup-cleanup

  2. Ensure all the parameters in the command are set.

  3. Ensure the BOSH Director credentials are valid.

  4. Ensure the specified deployment exists.

After Taking Your Backup

Step 11: Transfer Backup Artifacts

After creating your backup successfully, perform the following steps:

  1. Move the backup artifacts off the jumpbox to your preferred storage space. The backup created by BBR consists of a folder with the backup artifacts and metadata files. Pivotal recommends compressing and encrypting the files.

  2. Make redundant copies of your backup artifacts and store them in multiple locations to minimize the risk of losing backups in the event of a disaster.

  3. Attempt to restore each backup.

  4. To validate your artifacts, proceed to Step 12: Validate Your Backup.

Note: Backup artifacts may contain data covered by the European Union’s General Data Protection Regulation (GDPR).

Step 12: (Optional) Validate Your Backup

If you want to validate your backup, follow the instructions below:

WARNING:
  • The VMs and disks from the backed-up BOSH Director should not be visible to the new BOSH Director when validating your backup. As a result, Pivotal recommends that you deploy the new BOSH Director to a different IaaS network and account than the VMs and disks of the backed-up BOSH Director.
  • Data services outside of PCF may produce unexpected side effects during restoration. The services will try to connect to those data services when you restore to a new environment. For example, consider an app that processes mail queues and connects to an external database. When you validate your backup in a test environment, the app may start processing the queue, and this work may be lost.

After backing up PCF, you may want to validate your backup by restoring it to a similar environment and checking the applications. Because BBR is designed for disaster recovery, its backups are intended to be restored to an environment deployed with the same configuration.

To spin up a second environment that matches the original in order to test a restore, do the following:

  1. Export your Ops Manager installation by performing the steps in the Step 8: Export Installation Settings section.

  2. Create a new Ops Manager VM in a different network to the original. Ensure that the Ops Manager VM has enough persistent disk to accommodate the files exported in the previous step. Consult the topic specific to your IaaS:

After you deploy the second environment, follow the instructions in the Restoring Pivotal Cloud Foundry from Backup with BBR topic.

Validate Your PAS Backup Only

For a sandbox or other non-production environment, you can optionally perform an in-place restore of PAS only. In this case, you restore the PAS backup to the same PCF environment that the backup was created from. Follow the procedures in Restoring PCF from Backup with BBR.

Logging

BBR outputs logs to stdout. By default, BBR logs:

  • The backup and restore scripts that it finds
  • When it starts or finishes a stage, such as pre-backup scripts or backup scripts
  • When the process is complete
  • When any error occurs

If more logging is needed, use the optional --debug flag to print the following information:

  • Logs about the API requests made to the BOSH server
  • All commands executed on remote instances
  • All commands executed on local environment
  • Standard in and standard out streams for the backup and restore scripts when they are executed

Cancelling a Backup

If you need to cancel a backup, perform the following steps:

  1. Enter CTRL-C to terminate the BBR process, then enter yes to confirm.

  2. Log in to your BOSH Director with the BOSH CLI by performing the procedures in the Step 3: Retrieve BOSH Director Address and Credentials section above.

  3. Perform the following steps for each cloud_controller VM in your deployment:

    1. List the VMs in your deployment:
      $ bosh -e BOSH-DIRECTOR-IP \
      --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
      -d DEPLOYMENT-NAME \
      ssh
    2. Select the VM you want to SSH into.
    3. Run the following command from the VM:
      $ sudo /var/vcap/jobs/cloud-controller-backup/bin/bbr/post-backup-unlock
  4. Run the BBR pre-backup check from your jumpbox by following the steps in the Step 7: Check Your Deployment section above. If the command reports that it cannot back up the deployment, SSH onto each VM mentioned in the error using the BOSH CLI and remove the /var/vcap/store/bbr-backup directory if present.

European Union’s General Data Protection Regulation (GDPR)

Backup artifacts may contain data covered by the European Union’s General Data Protection Regulation (GDPR). Please be mindful that any backup artifacts may contain personal data. For example, a backup of PAS could contain user email addresses. As such, backup artifacts should be handled in accordance with organizational policies and applicable laws as they pertain to security, confidentiality, and privacy.

Create a pull request or raise an issue on the source for this page in GitHub