Restoring an ERT Backup In-place

Page last updated:

This document describes how to run an in-place restore of an ERT backup created with BBR. An in-place restore differs from a normal restore in that you do not tear down and redeploy your environment before the restore. Instead, you restore the ERT backup to the same PCF environment that the backup was created from.

Warning: An in-place restore erases any changes made to ERT since the backup was created. Follow the procedures in this topic only for sandbox/dev environments or when recovering from a disaster that affected ERT data.

Step 1: Retrieve BOSH Director Address and Credentials

Perform the following steps to retrieve the IP address of your BOSH Director and the credentials for logging in from the Ops Manager Director tile:

  1. Install the BOSH v2 CLI on a machine outside of your PCF deployment. You can use the jumpbox for this task.
  2. From the Installation Dashboard in Ops Manager, select Ops Manager Director > Status and record the IP address listed for the Director. You access the BOSH Director using this IP address.

  3. Click Credentials and record the Director credentials.
  4. From the command line, log into the BOSH Director using the IP address and credentials that you recorded:
    $ bosh -e DIRECTOR_IP \
    --ca-cert PATH-TO-BOSH-SERVER-CERT log-in
    Email (): director
    Password (): *******************
    Successfully authenticated with UAA
    Succeeded
    

Step 2: Identify Your Deployment

After logging in to your BOSH Director, run bosh deployments to identify the name of the BOSH deployment that contains PCF:

$ bosh -e DIRECTOR_IP --ca-cert /var/tempest/workspaces/default/root_ca_certificate deployments

Name                     Release(s)
cf-example               push-apps-manager-release/661.1.24
                         cf-backup-and-restore/0.0.1
                         binary-buildpack/1.0.11
                         capi/1.28.0
                         cf-autoscaling/91
                         cf-mysql/35
                         ...

In the above example, the name of the BOSH deployment that contains PCF is cf-example.

Step 3: Restore Elastic Runtime

Note: If your apps must not be running after a restore run bosh stop on each diego_cell VM in the deployment.

  1. If you use an external blobstore and copied it during the backup, restore the external blobstore with your IAAS specific tools before running Elastic Runtime restore.

  2. Run the BBR restore command from your jumpbox to restore Elastic Runtime:

    $ BOSH_CLIENT_SECRET=BOSH_PASSWORD \
      bbr deployment \
        --target BOSH_DIRECTOR_IP \
        --username BOSH_CLIENT \
        --deployment DEPLOYMENT_NAME \
        --ca-cert PATH_TO_BOSH_SERVER_CERT \
        restore \
          --artifact-path PATH_TO_ERT_BACKUP
    

    Replace the placeholder values as follows:

    • BOSH_CLIENT, BOSH_PASSWORD: Use the BOSH UAA user provided in Pivotal Ops Manager > Credentials > Uaa Bbr Client Credentials.

      You can also retrieve the credentials using the Ops Manager API with a GET request to the following endpoint: /api/v0/deployed/director/credentials/uaa_bbr_client_credentials. For more information, see the Using the Ops Manager API topic.

    • BOSH_DIRECTOR_IP: You retrieved this value in the Step 1: Retrieve BOSH Director Address and Credentials section.
    • DEPLOYMENT-NAME: You retrieved this value in the Step 2: Identify Your Deployment section.
    • PATH_TO_BOSH_SERVER_CERT: This is the path to the BOSH Director’s Certificate Authority (CA) certificate, if the certificate is not verifiable by the local machine’s certificate chain.
    • PATH_TO_ERT_BACKUP: This is the path to the Elastic Runtime backup you want to restore.

    Validation: If you ran bosh stop on each diego_cell before running bbr restore, you can now run cf stop on all apps and then run bosh start on each diego_cell. After this, all apps will be deployed in a stopped state.

  3. Perform the following steps after restoring Elastic Runtime:

    1. Retrieve the MySQL admin credentials from CredHub using the Ops Manager API:
      1. Perform the procedures in the Using the Ops Manager API topic to authenticate and access the Ops Manager API.
      2. Use the GET /api/v0/deployed/products endpoint to retrieve a list of deployed products, replacing UAA-ACCESS-TOKEN with the access token recorded in the previous step:
        $ curl "https://OPS-MAN-FQDN/api/v0/deployed/products" \
            -X GET \
            -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      3. In the response to the above request, locate the product with an installation_name starting with cf- and copy its guid.
      4. Run the following curl command, replacing PRODUCT-GUID with the value of guid from the previous step:
        $ curl "https://OPS-MAN-FQDN/api/v0/deployed/products/PRODUCT-GUID/variables?name=mysql-admin-credentials" \
            -X GET \
            -H "Authorization: Bearer UAA-ACCESS-TOKEN"
      5. Record the MySQL admin credentials from the response to the above request.
    2. List the VMs in your deployment:
      $ bosh -e DIRECTOR_IP --ca-cert /var/tempest/workspaces/default/root_ca_certificate \
      -d DEPLOYMENT_NAME \
      ssh
    3. Select the mysql VM to SSH into.
    4. From the mysql VM, run the following command:
      $ sudo /var/vcap/packages/mariadb/bin/mysql -u root -p
      When prompted, enter the MySQL admin password.

    5. At the MySQL prompt, run the following command:
      mysql> use silk; drop table subnets; drop table gorp_migrations;
    6. Exit MySQL:
      mysql> exit
    7. Exit the mysql VM:
      $ exit
    8. List the VMs in your deployment:
      $ bosh -e DIRECTOR_IP --ca-cert /var/tempest/workspaces/default/root_ca_certificate \
      -d DEPLOYMENT_NAME \
      ssh
    9. SSH onto each diego_database VM and run the following command:
      $ sudo monit restart silk-controller

    Restored apps will begin to start. The amount of time it takes for all apps to start depends on the number of app instances, the resources available to the underlying infrastructure, and the value of the Max Inflight Container Starts field in the Elastic Runtime tile.

  4. If desired, scale the MySQL Server job back up to its previous number of instances by navigating to the Resource Config section of the Elastic Runtime tile. After scaling the job, return to the Ops Manager Installation Dashboard and click Apply Changes to deploy.

Create a pull request or raise an issue on the source for this page in GitHub