Restoring Lost BOSH Director Persistent Disk
This guide explains how to use BOSH Backup and Restore (BBR) to restore a BOSH Director that has lost its persistent disk.
Restoring a BOSH Director persistent disk with BBR is only possible if you have previously taken a backup using the following command:
bbr director backup
This command produces a backup artifact in the format
Warning: BOSH Director backups require identical references to all BOSH-deployed VMs. If BOSH recreates your deployment VMs for any reason, such as changes to stemcells, networks, or availability zones (AZs), your BBR backup artifact is no longer compatible for performing a restore.
Your BOSH Director must be in a healthy state before you can perform a BOSH Director restore. The BOSH Director is in a healthy state when all jobs are running.
If your BOSH Director has lost its persistent disk, create a new disk to return your BOSH Director to a healthy state.
To create a new persistent disk for the BOSH Director, do the following:
Create a new persistent disk using your IaaS console.
Log in to the Ops Manager VM. See the Log in to the Ops Manager VM with SSH section for your IaaS in Advanced Troubleshooting with the BOSH CLI.
bosh-state.jsonfile in a text editor. Edit the
disks.cidvalue to match the new persistent disk you created in the first step.
Update the persistent disk size in Ops Manager and then click Apply Changes. This creates a new BOSH Director VM and attaches the new persistent disk to it.
Warning: BBR restore is a destructive process which removes any current data in your deployment. Performing a restore overwrites all new data since you created your most recent backup artifact.
Perform a BOSH Director restore by following the BBR instructions in Step 8: Restore the BOSH Director of Restoring PCF From Backup with BBR.
After you successfully restore your BOSH Director, the BOSH Director uses the VM references that were stored in the backup artifact. If you clicked Apply Changes in Ops Manager between when the backup and the restore took place, check if any BOSH-deployed VMs were deleted or added.
If changes have occurred, ensure that the BOSH Director database is consistent with the current state of your IaaS after the restore. If VMs were deleted, the BOSH Director still has references to these deleted VMs after the restore. If VMs were added, the BOSH Director has no knowledge of the new VMs.
To ensure that the BOSH Director database matches your IaaS console, do the following:
Run the following command to find and delete outdated VM references for an environment:
bosh --deployment DEPLOYMENT-NAME cloud-check
DEPLOYMENT-NAMEis the name of your BOSH deployment.
Log in to your IaaS console and delete any orphaned VMs that your BOSH Director does not reference.
Your BOSH Director should now be in a healthy state with the new persistent disk attached. Your BOSH deployment VMs should be aligned with the current state of your IaaS.