Restoring PCF from Backup with BBR

Page last updated:

This topic describes the procedure for restoring your critical back end Pivotal Cloud Foundry (PCF) components with BOSH Backup and Restore (BBR), a command-line tool for backing up and restoring BOSH deployments. To perform the procedures in this topic, you must have backed up PCF by following the steps in the Backing Up Pivotal Cloud Foundry with BBR topic.

For BBR release notes, see BOSH Backup and Restore Release Notes in the Cloud Foundry documentation.

Overview

To perform the procedures in this topic, you must have backed up PCF by following the steps in Backing Up PCF with BBR.

Warning: Restoring PCF with BBR is a destructive operation. If the restore fails, the new environment may be left in an unusable state and require re-provisioning. Only perform the procedures in this topic for the purpose of disaster recovery, such as recreating PCF after a storage-area network (SAN) corruption.

Warning: When validating your backup, the VMs and disks from the backed-up BOSH Director should not be visible to the new BOSH Director. As a result, Pivotal recommends that you deploy the new BOSH Director to a different IaaS network and account than the VMs and disks of the backed up BOSH Director.

Note: If the BOSH Director you are restoring had any deployments that were deployed manually rather than through an Ops Manager tile, you must restore them manually at the end of the process. For more information, see (Optional) Step 16: Restore Non-Tile Deployments.

Compatibility of Restore

This section describes the restrictions for a backup artifact to be restorable to another environment. This section is for guidance only, and Pivotal recommends that you validate your backups by using the backup artifacts in a restore.

For a backup artifact to be restorable, the following restrictions apply:

  • CIDR ranges: BBR requires the IP address ranges to be the same in the restore environment as in the backup environment.

  • Topology: BBR requires the BOSH topology of a deployment to be the same in the restore environment as it was in the backup environment.

  • Naming of instance groups and jobs: For any deployment that implements the backup and restore scripts, the instance groups and jobs must have the same names.

  • Number of instance groups and jobs: For instance groups and jobs that have backup and restore scripts, there must be the same number of instances.

  • Limited validation: BBR puts the backed-up data into the corresponding instance groups and jobs in the restored environment, but cannot validate the restore beyond that. For example, if the MySQL encryption key is different in the restore environment, the BBR restore might succeed, but the restored MySQL database is unusable.

  • PCF version: BBR can restore to the same version of PCF that was backed up. BBR does not support restoring to other major, minor, or patch releases.

Note: A change in VM size or underlying hardware will not affect BBR’s ability to restore data, as long as there is adequate storage space to restore the data.

Restore Workflow

The diagram below shows the flow of the PCF restore process in a series of steps performed by the PCF operator. The following steps are covered in more detail throughout this topic. Click the diagram to see the full-size image.

Flow chart shows the PCF Operator through a series of steps interacting with the BOSH CLI, Ops Manager VM, Director VM, and the PAS tile. Details on these steps are described below.

  1. Launch new Ops Manager: Perform the procedures for your IaaS to deploy Ops Manager. For more information, see part one of Deploy Ops Manager and Import Installation Settings.

  2. Import settings: You can import settings either with the Ops Manager UI or API. For more information, see part two of Deploy Ops Manager and Import Installation Settings.

  3. Remove bosh-state.json: SSH into your Ops Manager VM and delete the bosh-state.json file. For more information, see Remove BOSH State File.

  4. Apply Changes (Director only): Use the Ops Manager API to only deploy the BOSH Director. For more information, see Deploy the BOSH Director.

  5. bbr restore <director>: Run the BBR restore command from your jumpbox to restore the BOSH Director. For more information, see Restore the BOSH Director.

  6. Use BOSH cck to fix the stale cloud ID references in the BOSH database: For each deployment in the BOSH Director, you must run a bosh cloud-check command. For more information, see Remove Stale Cloud IDs for All Deployments.

  7. Apply Changes: On the Ops Manager Installation Dashboard, click Review Pending Changes, review your changes, and then click Apply Changes. For more information, see Reviewing Pending Product Changes.

  8. bbr restore <PAS>: Run the BBR restore command from your jumpbox to restore PAS. For more information, see Restore PAS.

Prepare to Restore

This section describes the procedure you must perform before restoring your PCF backup with BBR.

Step 1: (Optional) Prepare Your Environment

In the event of a disaster, you can lose not only your VMs and disks, but your IaaS resources as well, such as networks and load balancers.

If you need to recreate your IaaS resources, prepare your environment for PCF by following the instructions specific to your IaaS in Installing Pivotal Cloud Foundry.

Note: The instructions for installing PCF on Amazon Web Services (AWS) and OpenStack combine the procedures for preparing your environment and deploying Ops Manager into a single topic. The instructions for the other supported IaaSes split these procedures into two separate topics.

If you re-create your IaaS resources, you must also add those resources to Ops Manager by performing the procedures in Step 3: (Optional) Configure Ops Manager for New Resources.

Step 2: Deploy Ops Manager and Import Installation Settings

  1. Perform the procedures for your IaaS to deploy Ops Manager:

  2. Import your installation settings. This can be done in two ways:

    1. Using the Ops Manager UI:

      1. Access your new Ops Manager by navigating to YOUR-OPS-MAN-FQDN in a browser.
      2. On the Welcome to Ops Manager page, click Import Existing Installation.

        Welcome

      3. In the import panel:

        • Enter the Decryption Passphrase in use when you exported the installation settings from Ops Manager.
        • Click Choose File and browse to the installation zip file that you exported in the Step 9: Export Installation Settings section of the Backing Up PCF with BBR topic.

        Decryption passphrase

      4. Click Import.

        Note: Some browsers do not provide feedback on the status of the import process, and may appear to hang. The import process takes at least 10 minutes, and takes longer the more tiles that were present on the backed-up Ops Manager.

      5. A Successfully imported installation message appears upon completion.

        Success

    2. Using the Ops Manager API:

      1. Run the following command:

        curl "https://OPS-MAN-FQDN/api/v1/installation_asset_collection \
          -X POST \
          -H "Authorization: Bearer UAA-ACCESS-TOKEN" \
          -F 'installation[file]=@installation.zip' \
          -F 'passphrase=DECRYPTION-PASSPHRASE'
        

        Where:

        • OPS-MAN-FQDN is the fully qualified domain name (FQDN) for your Ops Manager deployment.
        • UAA-ACCESS-TOKEN is the UAA access token. For more information about how to retrieve this token, see Using the Ops Manager API.
        • DECRYPTION-PASSPHRASE is the decryption passphrase in use when you exported the installation settings from Ops Manager.

Warning: Do not click Apply Changes in Ops Manager until the instruction in Step 14: Redeploy PAS.

Step 3: (Optional) Configure Ops Manager for New Resources

If you re-created IaaS resources such as networks and load balancers by following the procedure in Step 1: (Optional) Prepare Your Environment, you must update Ops Manager with your new resources.’

To update Ops Manager:

  1. Enable the advanced mode of Ops Manager. For more information, see How to Enable Advanced Mode in the Ops Manager in the Pivotal Knowledge Base.

    Note: In advanced mode, Ops Manager allows you to make changes that are normally disabled. You might see warning messages when you save changes.

  2. Navigate to the Ops Manager Installation Dashboard and click the BOSH Director tile.

  3. If you are using Google Cloud Platform (GCP), click Google Config and update:

    1. Project ID to reflect the GCP project ID.
    2. Default Deployment Tag to reflect the environment name.
    3. AuthJSON to reflect the service account.
  4. Click Create Networks and update the network names to reflect the network names for the new environment.

  5. If your BOSH Director had an external hostname, you must change it in Director Config > Director Hostname to ensure it does not conflict with the hostname of the backed up Director.

  6. Return to the Ops Manager Installation Dashboard and click the Pivotal Application Service (PAS) tile.

  7. Click Resource Config. If necessary for your IaaS, enter the name of your new load balancers in the Load Balancers column.

  8. If necessary, click Networking and update the load balancer SSL certificate and private key under Certificates and Private Keys for HAProxy and Router.

  9. If your environment has a new DNS address, update the old environment DNS entries to point to the new load balancer addresses. For more information, see the Step 4: Configure Networking section of the Using Your Own Load Balancer topic and follow the link to the instructions for your IaaS.

  10. If your PAS uses an external blobstore, ensure that the PAS tile is configured to use a different blobstore. Otherwise, it attempts to connect to the blobstore that the backed-up PAS is using.

  11. Ensure your System Domain and Apps Domain under PAS Domains are updated to refer to the new environment’s domains.

  12. Ensure that there are no outstanding warning messages in the BOSH Director tile, then disable Ops Manager advanced mode. For more information, see How to Enable Advanced Mode in the Ops Manager in the Pivotal Knowledge Base.

Step 4: Remove BOSH State File

To remove the BOSH state file:

  1. SSH into your Ops Manager VM. For more information, see the SSH into Ops Manager section of the Advanced Troubleshooting with the BOSH CLI topic.

  2. To delete the /var/tempest/workspaces/default/deployments/bosh-state.json file, run the following command on the Ops Manager VM:

    sudo rm /var/tempest/workspaces/default/deployments/bosh-state.json
    
  3. In a browser, navigate to Ops Manager’s FQDN.

  4. Log in to Ops Manager.

Step 5: Deploy the BOSH Director

Use the Ops Manager API or the checkbox on the Review Pending Changes page to deploy the BOSH Director by itself.

Step 6: Transfer Artifacts to Jumpbox

In the Step 9: Back Up Your PAS Deployment section of the Backing Up PCF with BBR topic, in the After Taking the Backups section, you moved the TAR and metadata files of the backup artifacts off your jumpbox to your preferred storage space. Now you must transfer those files back to your jumpbox.

Restore Your Backup

This section describes the procedure you must perform to restore your PCF backup with BBR.

Step 7: Retrieve BOSH Director Credentials

To use BBR, you must retrieve and record the following credentials:

  • Bosh Director Credentials
  • Bbr Ssh Credentials
  • Uaa Bbr Client Credentials

There are two ways to retrieve BOSH Director credentials:

  • Ops Manager Installation Dashboard
  • Ops Manager API

Option 1: Ops Manager Installation Dashboard

To retrieve your BOSH Director credentials using the Ops Manager Installation Dashboard, perform the following steps:

  1. Navigate to the Ops Manager Installation Dashboard.
  2. Click the BOSH Director tile.
  3. Click the Credentials tab.
  4. Locate Director Credentials.

    1. Click Link to Credentials next to it.
    2. Verify the value of the identity field. It should be director.
    3. Copy and record the value of the password field.
  5. Locate Bbr Ssh Credentials.

    1. Click Link to Credentials next to it.
    2. Copy the value of the private_key_pem field.
    3. Run the following command to reformat the copied value, and save it in the current directory to a file named PRIVATE-KEY-FILE:

      printf -- "YOUR-PRIVATE-KEY" > PRIVATE-KEY-FILE
      

      Where:

      • YOUR-PRIVATE-KEY is the text of your private key.
      • PRIVATE-KEY-FILE is the path to the private key file you are creating.

      For example:

      $ printf --  "-----BEGIN RSA PRIVATE KEY----- MIIEkeycontents ----END RSA PRIVATE KEY-----" > bbr_key.pem
      
  6. Locate the Uaa Bbr Client Credentials

    1. Click Link to Credentials next to it.
    2. Verify the value of the identity field. It should be bbr_client.
    3. Record the value of the password field.

Option 2: Ops Manager API

To retrieve BOSH Director credentials using the Ops Manager API, perform the following steps:

  1. Obtain your UAA access token. For more information, see Access the API.
  2. Retrieve the Director Credentials by performing the following steps:

    1. Run the following command:

      curl "https://OPS-MAN-FQDN/api/v0/deployed/director/credentials/director_credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MAN-FQDN is the fully-qualified domain name (FQDN) for your Ops Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.
    2. Verify the value of the identity field. It should be director.

    3. Record the value of the password field.

  3. Retrieve the Bbr Ssh Credentials by performing the following steps:

    1. Run the following command:

      curl "https://OPS-MAN-FQDN/api/v0/deployed/director/credentials/bbr_ssh_credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MAN-FQDN is the fully-qualified domain name (FQDN) for your Ops Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.
    2. Copy the value of the private_key_pem field.

    3. Run the following command to reformat the copied value, and save it in the current directory to a file named PRIVATE-KEY-FILE:

      printf -- "YOUR-PRIVATE-KEY" > PRIVATE-KEY-FILE
      

      Where:

      • YOUR-PRIVATE-KEY is the text of your private key.
      • PRIVATE-KEY-FILE is the path to the private key file you are creating.

      For example:

      $ printf --  "-----BEGIN RSA PRIVATE KEY----- MIIEkeycontents ----END RSA PRIVATE KEY-----" > bbr_key.pem
      
  4. Retrieve the Uaa Bbr Client Credentials by performing the following steps:

    1. Run the following command:

      curl "https://OPS-MAN-FQDN/api/v0/deployed/director/credentials/uaa_bbr_client_credentials" \
      -X GET \
      -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      

      Where:

      • OPS-MAN-FQDN is the fully-qualified domain name (FQDN) for your Ops Manager deployment.
      • UAA-ACCESS-TOKEN is your UAA access token.
    2. Verify the value of the identity field. It should be bbr_client.

    3. Record the value of the password field.

For more information about using the Ops Manager API, see the Using the Ops Manager API topic.

Step 8: Retrieve BOSH Director Address

To retrieve the IP address of your BOSH Director from the BOSH Director tile:

  1. If you are not using the Ops Manager VM as your jumpbox, install the latest BOSH CLI on your jumpbox.
  2. From the Installation Dashboard in Ops Manager, select BOSH Director > Status and record the IP address listed for the Director. You access the BOSH Director using this IP address.

  3. From the command line, log into the BOSH Director, using the IP address that you recorded above, by running the following command:

    bosh -e DIRECTOR-IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE log-in
    

    Where:

    • DIRECTOR-IP is the BOSH Director IP address recorded above.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the root Certificate Authority (CA) certificate as outlined in Step 1: Set Up your Jumpbox, above.
  4. When prompted for Email (), specify director.

  5. When promted for Password (), enter the Director Credentials that you obtained in Retrieve BOSH Director Credentials.

    For example:

    $ bosh -e 10.0.0.3 \
    --ca-cert /var/tempest/workspaces/default/root_ca_certificate log-in
    Email (): director
    Password (): *******************
    Successfully authenticated with UAA
    Succeeded
    

Step 9: Restore BOSH Director

Notes:
  • The BBR BOSH Director restore command can take at least 15 minutes to complete.
  • Pivotal recommends that you run it independently of the SSH session, so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup but you could also run the command in a screen or tmux session.

To restore BOSH Director:

  1. SSH into your jumpbox. If you are using the Ops Manager VM as your jumpbox, see the Log In to the Ops Manager VM with SSH section of Advanced Troubleshooting with the BOSH CLI for procedures on how to use SSH to connect to the Ops Manager VM.

  2. Ensure the BOSH Director backup artifact is in the folder you from which you will run BBR.

  3. Run the BBR restore command from your jumpbox to restore the BOSH Director:

    bbr director \
    --private-key-path PRIVATE-KEY-FILE \
    --username bbr \
    --host HOST \
    restore \
    --artifact-path PATH-TO-DIRECTOR-BACKUP
    

    Where:

    • PATH-TO-DIRECTOR-BACKUP is the path to the Director backup you want to restore.
    • PRIVATE-KEY-FILEis the path to the private key file you created in Step 7: Retrieve BOSH Director Credentials.
    • HOSTis the address of the BOSH Director.
      • If the BOSH Director is public, HOST is a URL, such as https://my-bosh.xxx.cf-app.com.
      • If the BOSH Director is not public, HOST is the BOSH-DIRECTOR-IP retrieved in Step 8: Retrieve BOSH Director Address.
    • Use the optional --debug flag if you want to enable debug logs. For more information, see the Logging section of the Backing Up PCF with BBR topic.
  4. After the command succeeds, continue to Step 10: Identify Your Deployment. If the command fails:

    1. Run:

      bbr director \
      --private-key-path PRIVATE-KEY-FILE \
      --username bbr \
      --host HOST \
      restore-cleanup
      

      Where:

    2. Run the BBR restore command again after ensuring that:

      • All the parameters in the command are set.
      • The BOSH Director credentials are valid.
      • The specified deployment exists.
      • The source deployment is compatible with the target deployment.
      • That the jumpbox can reach the BOSH Director.

Step 10: Identify Your Deployment

To identify the name of the BOSH deployment that contains PCF:

  1. Log in to your BOSH Director.

  2. Run:

    bosh -e BOSH-DIRECTOR-IP --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE deployments
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.

    For example:

    $ bosh -e BOSH-DIRECTOR-IP --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE deployments
    
    Name Release(s) cf-example push-apps-manager-release/661.1.24 cf-backup-and-restore/0.0.1 binary-buildpack/1.0.11 capi/1.28.0 cf-autoscaling/91 cf-mysql/35 ...

    In the above example, the name of the BOSH deployment that contains PCF is cf-example. PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the root Certificate Authority (CA) certificate for the BOSH Director. If you are using the Ops Manager VM as your jumpbox, the path is /var/tempest/workspaces/default/root_ca_certificate.

Step 11: Remove Stale Cloud IDs for All Deployments

To remove stale cloud IDs for all deployments:

  1. Review the deployments listed when performing Step 10: Identify Your Deployment.

  2. To reconcile the BOSH Director’s internal state with the state in the IaaS, run:

    bosh -e BOSH-DIRECTOR-IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    -d DEPLOYMENT-NAME -n cck \
    --resolution delete_disk_reference \
    --resolution delete_vm_reference
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority. (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine‚Äôs certificate chain.
    • DEPLOYMENT-NAME is the deployment name retrieved in Step 10: Identify Your Deployment.
      Run this command for each deployment.
  3. To delete disk references, run:

    bosh cloud-check
    

    If the bosh cloud-check command does not successfully delete disk references, and you see a message similar to the following, perform the additional procedures in Remove Unused Disks.

    Scanning 19 persistent disks: 19 OK, 0 missing ...
    

Step 12: Redeploy PAS

This section describes the steps to redeploy PAS.

Determine the Required Stemcell

Perform either of the following procedures to determine which stemcell is used by PAS:

  • Review the Stemcell Librbary:

    1. Go to Stemcell Library.
    2. Record the PAS stemcell release number from the Staged column.
  • Review a Stemcell List Using BOSH CLI:

    1. To retrieve the stemcell release using the BOSH CLI, run:

      bosh -e BOSH-DIRECTOR-IP deployments
      

      Where BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.

      For example:

      $ bosh -e BOSH-DIRECTOR-IP deployments
      Using environment '10.0.0.5' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

      Name Release(s) Stemcell(s) Team(s) Cloud Config cf-9cb6995b7d746cd77438 push-apps-manager-release/661.1.24 bosh-google-kvm-ubuntu-trusty-go_agent/3421.9 - latest ...

For more information about stemcells in Ops Manager, see Importing and Managing Stemcells.

> Upload the Stemcell

To upload the required stemcell:

  1. Download the stemcell from Pivotal Network.

  2. Upload the stemcell used by PAS by running:

    bosh -e BOSH-DIRECTOR-IP \
    -d DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    upload-stemcell \
    --fix PATH-TO-STEMCELL
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • DEPLOYMENT-NAME is the deployment name retrieved in Step 10: Identify Your Deployment.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.
    • PATH-TO-STEMCELL is the path to your tile’s stemcell.
  3. To ensure the stemcells for all of your other installed tiles have been uploaded, repeat the last step, running bosh upload-stemcell --fix PATH-TO-STEMCELL, for each stemcell that is different from the already uploaded PAS stemcell.

Redeploy PAS

To redeploy PAS:

  1. Go to the Ops Manager Installation Dashboard and click the PAS tile.

  2. Select Resource Config.

  3. Ensure the number of instances for MySQL Server is set to 1.

    Warning: Restore fails if there is not exactly one MySQL Server instance deployed.

  4. Ensure that all errands needed by your system are set to run.

  5. Return to the Ops Manager Installation Dashboard.

  6. Click Review Pending Changes.

  7. Review your changes and ensure the PAS tile is selected. Other tiles are optional. For more information, see Reviewing Pending Product Changes.

  8. Click Apply Changes to redeploy.

Step 13: (Optional) Restore Service Data

WARNING: BBR does not back up or restore any service data.

For this step, you must restore data to pre-provisioned service tiles.

The procedures for restoring service data vary. For more information, see the documentation for your service tile.

For example, if you are using Redis for PCF v1.14, see Using BOSH Backup and Restore (BBR).

Step 14: Restore PAS

Notes:
  • The BBR PAS restore command can take at least 15 minutes to complete.
  • Pivotal recommends that you run it independently of the SSH session, so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup but you could also run the command in a screen or tmux session.

To restore your PAS backup:

  1. See the table in the External Blobstore Support section of the Enabling External Blobstore Backups topic. If external blobstore support is not included in the version of Ops Manager and PAS you are using, restore the external blobstore with your IaaS-specific tools before restoring PAS.

  2. From your jumpbox, run:

    bbr deployment \
    --target BOSH-DIRECTOR-IP \
    --username BOSH-CLIENT \
    --password BOSH-PASSWORD \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    restore \
    --artifact-path PATH-TO-PAS-BACKUP
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • BOSH-CLIENT, BOSH-PASSWORD are the Uaa Bbr Client Credentials, identity and password, that you retrieved in Step 7: Retrieve BOSH Director Credentials.
    • DEPLOYMENT-NAME is the deployment name retrieved in Step 10: Identify Your Deployment.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.
    • PATH-TO-PAS-BACKUP is the path to the PAS backup you want to restore.
  3. If desired, scale the MySQL Server job back up to its previous number of instances by navigating to the Resource Config section of the PAS tile. After scaling the job, return to the Ops Manager Installation Dashboard.

  4. Click Review Pending Changes.

  5. Review your changes. For more information, see Reviewing Pending Product Changes.

  6. Click Apply Changes to deploy.

Step 15: (Optional) Restore On-Demand Service Instances

If you have on-demand service instances provisioned by an on-demand service broker, you can restore them after successfully restoring PAS.

To restore your on-demand service instances:

  1. Navigate to an on-demand service tile in the Installation Dashboard.

  2. Select the Errands tab.

  3. Ensure the Upgrade All Service Instances errand is On.

  4. Repeat for all on-demand service tiles.

  5. Return to the Installation Dashboard.

  6. Click Review Pending Changes, review your changes, and then click Apply Changes. This includes running the Upgrade All Service Instances errand for the on-demand service, which redeploys the on-demand service instances.

  7. (Optional) Restore service data to every on-demand service instance.

  8. Any app on PAS bound to an on-demand service instance may need to be restarted to start consuming the restored on-demand service instances.

Step 16: (Optional) Restore Non-Tile Deployments

If you have any deployments that were deployed manually with the BOSH Director rather than through an Ops Manager tile, you must restore their VMs.

To restore the VMs:

  1. To obtain a list of all deployments on your BOSH Director, run:

    bosh -e BOSH-DIRECTOR-IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    deployments
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.
  2. Identify the names of the deployments that you need to restore. Do not include the deployments from Ops Manager tiles.

  3. For each deployment you need to restore, run:

    bosh -n -e BOSH-DIRECTOR-IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    -d DEPLOYMENT-NAME \
    cck --resolution=recreate_vm
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.
    • DEPLOYMENT-NAME is the deployment name retrieved in Step 10: Identify Your Deployment.
  4. To verify the status of the VMs in each deployment, run:

    bosh -e BOSH-DIRECTOR-IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE \
    -d DEPLOYMENT-NAME \
    vms
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
    • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.
    • DEPLOYMENT-NAME is the deployment name retrieved in Step 10: Identify Your Deployment.

The process state for all VMs should show as running.

After Restoring Your Backup

This section describes the procedures you must perform after restoring your PCF backup with BBR.

Step 17: Remove Unused Disks

WARNING: This is a very destructive operation.

Disks from a previous deployment will prevent recreated deployments from working.

Use BOSH to Clean Up Disks

To clean up disk references:

  1. Run:

    bosh cloud-check
    

Manually Clean Up Disks

If bosh cloud-check does not clean up all disk references, you must manually delete the remaining disks.

To delete the remaining disks, you can perform one of two procedures:

  • Use the BOSH CLI to delete the disks:

    1. Target the redeployed BOSH Director using the BOSH CLI by performing the procedures in Step 8: Retrieve BOSH Director Address.
    2. List the deployments by running:

        bosh -e BOSH-DIRECTOR-IP \
        --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE deployments
      

      Where:

      • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
      • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.
    3. Delete each deployment by running:

        bosh -d DEPLOYMENT-NAME delete-deployment
      

      Where:

  • Log in to your IaaS account and delete the disks manually.

    1. To retrieve a list of disk IDs, run:

        bosh -e BOSH-DIRECTOR-IP \
        --ca-cert PATH-TO-BOSH-SERVER-CERTIFICATE instances -i
      

      Where:

      • BOSH-DIRECTOR-IP is the BOSH Director IP retrieved in Step 8: Retrieve BOSH Director Address.
      • PATH-TO-BOSH-SERVER-CERTIFICATE is the path to the Certificate Authority (CA) certificate for the BOSH Director, if the certificate is not verifiable by the local machine’s certificate chain.

After the disks are deleted, continue with Step 11: Remove Stale Cloud IDs for All Deployments.