Recovering from an Ops Manager and PAS Upgrade Failure

Page last updated:

This topic explains how to use BOSH Backup and Restore (BBR) to restore Ops Manager and Pivotal Application Service (PAS) back to a healthy state after a failed upgrade between PAS minor versions.

Note: BBR can only restore across the same PAS minor versions. See metadata/cf-VERSION.yml in the backup artifact to verify which version of PAS was used to create the backup artifact.

Evaluate Your Deployment

You can restore Ops Manager and PAS to a healthy state after a failed upgrade between PAS minor versions if your backups, BOSH Director, and PAS deployment meet certain conditions. Follow the procedures in this section to confirm that your deployment can be restored by BBR.

Review Your Backup

Before you perform a restore, you must do the following:

  1. Run the following bbr commands from your Ops Manager VM:

    • Back up your deployments by running the following command:

      bbr deployment --deployment DEPLOYMENT_NAME backup
      
    • Back up your BOSH Director by running the following command:

      bbr director backup
      

      The above bbr commands produce the following backup artifacts:

      • DEPLOYMENT_NAME_TIMESTAMP/
      • DIRECTOR_IP_TIMESTAMP/
      • installation.zip
  2. Export your Ops Manager settings. For more information, see the Settings Page section of Using the Ops Manager Interface.

Validate Your Environment for Restore

Before you can restore after a failed upgrade between PAS minor versions, you must confirm that your BOSH Director and PAS deployment are healthy.

Confirm That Your BOSH Director Is Healthy

To check that the BOSH Director is in a healthy state for a bbr restore, SSH to the Ops Manager VM and ensure that all jobs are running.

To make sure that all jobs are running, do the following:

  1. SSH into your BOSH Director. For more information, see Log in to the BOSH Director VM with SSH in Advanced Troubleshooting with the BOSH CLI.

  2. View all deployment job instances by running the following command:

    monit summary
    

    Monit should report that all jobs are running.

  3. Ensure that the BOSH API is responsive by running the following command:

    bosh configs
    

    If this command runs successfully, then the BOSH Director API is in a healthy and responsive state and can be restored by BBR.

If your BOSH Director is unhealthy, see Restoring an Unhealthy BOSH Director below.

Confirm That Your PAS Deployment Is Healthy

To confirm that PAS is in a healthy state, run the following command:

bosh -d PAS-DEPLOYMENT-NAME instances

Where PAS-DEPLOYMENT-NAME is the name of your PAS BOSH deployment.

If your PAS deployment is healthy, then the output states that all of your instances are in the running state. If all instances are running, PAS can be restored by BBR.

If your PAS deployment is unhealthy, see Restoring an Unhealthy PAS Deployment.

Restore Your Deployment

After you confirm that your BOSH Director is healthy with its data intact, and all instances in your PAS deployment are responsive, you can restore PAS with BBR.

Restoring PAS

If your BOSH Director is healthy, you can restore across the same PAS minor version by following the procedures in Step 13: Restore PAS in Restoring PCF from Backup with BBR.

Recover to a Healthy State

If either the BOSH Director or PAS are not in a healthy state, they must be recovered before you can run bbr restore. If you need guidance on the following subsections, contact Pivotal Support.

Restoring an Unhealthy BOSH Director

If the BOSH Director is not in a healthy state, Pivotal recommends following our full restore workflow to regain your Ops Manager, BOSH Director, and BOSH deployments.

See Restore Workflow in Restoring PCF from Backup with BBR.

Restoring an Unhealthy PAS Deployment

If your BOSH Director is healthy, you can restore your previous BOSH Director state by following Step 8: Restore the BOSH Director in Restoring PCF from Backup with BBR.

After your BOSH Director is restored, you can then restore your deployments by following Step 13: Restore PAS in Restoring PCF from Backup with BBR.