Restoring Lost BOSH Director Persistent Disk

Page last updated:

This guide explains how to use BOSH Backup and Restore (BBR) to restore a BOSH Director that has lost its persistent disk.

Prerequisite

Restoring a BOSH Director persistent disk with BBR is only possible if you have previously taken a backup using the following command:

bbr director backup

This command produces a backup artifact in the format DIRECTOR_IP_TIMESTAMP/.

Warning: BOSH Director backups require identical references to all BOSH-deployed VMs. If BOSH re-creates your deployment VMs for any reason, such as changes to stemcells, networks, or availability zones (AZs), your BBR backup artifact is no longer compatible for performing a restore.

Prepare Your Environment

Your BOSH Director must be in a healthy state before you can perform a BOSH Director restore. The BOSH Director is in a healthy state when all jobs are running.

If your BOSH Director has lost its persistent disk, create a new disk to return your BOSH Director to a healthy state.

Create a New BOSH Director Persistent Disk

To create a new persistent disk for the BOSH Director, do the following:

  1. Create a new persistent disk using your IaaS console.

  2. Log in to the Pivotal Operations Manager VM. See the Log in to the Ops Manager VM with SSH section for your IaaS in Advanced Troubleshooting with the BOSH CLI.

  3. Open the bosh-state.json file in a text editor. Edit the disks.cid value to match the new persistent disk you created in the first step.

  4. Update the persistent disk size in Ops Manager and then click Apply Changes. This creates a new BOSH Director VM and attaches the new persistent disk to it.

Perform the Restore

Warning: BBR restore is a destructive process which removes any current data in your deployment. Performing a restore overwrites all new data since you created your most recent backup artifact.

Perform a BOSH Director restore by following the BBR instructions in Step 8: Restore the BOSH Director of Restoring Pivotal Platform From Backup with BBR.

Consolidate Your Environment

After you successfully restore your BOSH Director, the BOSH Director uses the VM references that were stored in the backup artifact. If you clicked Apply Changes in Ops Manager between when the backup and the restore took place, check if any BOSH-deployed VMs were deleted or added.

If changes have occurred, ensure that the BOSH Director database is consistent with the current state of your IaaS after the restore. If VMs were deleted, the BOSH Director still has references to these deleted VMs after the restore. If VMs were added, the BOSH Director has no knowledge of the new VMs.

To ensure that the BOSH Director database matches your IaaS console, do the following:

  1. Run the following command to find and delete outdated VM references for an environment:

    bosh --deployment DEPLOYMENT-NAME cloud-check
    

    Where DEPLOYMENT-NAME is the name of your BOSH deployment.

  2. Log in to your IaaS console and delete any orphaned VMs that your BOSH Director does not reference.

Your BOSH Director should now be in a healthy state with the new persistent disk attached. Your BOSH deployment VMs should be aligned with the current state of your IaaS.