Troubleshooting BBR

Page last updated:

This topic lists common troubleshooting scenarios and their solutions when using BOSH Backup and Restore (BBR) to back up and restore Pivotal Platform.

Troubleshooting During a Restore

This section provides solutions for errors that occur during a restore.

Restore Fails with a MySQL Monit Start Timeout

Symptom

While running the BBR restore command, restoring the job mysql-restore fails with the following error:

1 error occurred:

* restore script for job mysql-restore failed on mysql/0.
...
Monit start failed: Timed out waiting for monit: 2m0s

Explanation

This happens when the MySQL job fails to start within the timeout period. It ends up in an Execution Failed state and monit never tries to start it again.

Solution

Ensure that your MySQL Server cluster has only one instance. If there is more than one instances of MySQL Server, the restore fails with a monit start timeout. Scale down to one instance and retry.

If your MySQL Server cluster is already scaled down to one node, it might have taken longer than normal to restart the cluster. To manually verify and retry:

  1. To list the VMs in your deployment, run:

    bosh -e BOSH-DIRECTOR-IP --ca-cert /var/tempest/workspaces/default/root_ca_certificate \
    -d DEPLOYMENT-NAME \
    ssh
    

    Where:

    • BOSH-DIRECTOR-IP is the BOSH Director IP. For how to retrieve this IP address, see the Retrieve BOSH Director Address section of the Restoring Pivotal Platform from Backup with BBR topic.
    • DEPLOYMENT-NAME is the deployment name. For how to retrieve the deployment name, see the Identify Your Deployment section of the Restoring Pivotal Platform from Backup with BBR topic.
  2. Locate the mysql VM.

  3. SSH into the mysql VM.

  4. To check that the MySQL job process is running, run:

    ps aux | grep galera-init
    

    For more information, see Configuring PAS.

  5. Check that monit reports that the MySQL job process is in an Execution Failed state by running:

    sudo monit summary
    
  6. If so, run the following command from the mysql VM to disable monitoring:

    monit unmonitor
    
  7. Enable monitoring by running:

    monit monitor
    
  8. After a few minutes, run:

    monit summary
    

    The command should report that all the processes are running.

  9. Re-attempt the restore with BBR.

Deployment Does Not Match the Structure of the Backup

Symptom

The following error displays:

Deployment 'deployment-name' does not match the structure of the provided backup

Explanation

The instance groups with the restore scripts in the destination environment do not match the backup metadata. For example, they might have the wrong number of instances of a particular instance group, or the metadata names an instance group that does not exist in the destination environment.

Solution

BBR only supports restoring to an environment that matches your original environment. Pivotal recommends altering the destination environment to match the structure of the backup.

General Troubleshooting

This section provides solutions for general errors.

Connection Error

Symptom

BBR displays an error message containing “SSH Dial Error” or another connection error.

Explanation

The jumpbox and the VMs in the deployment are experiencing connection problems.

Solution

To resolve connection problems between the jumpbox and the VMs in the deployment:

  1. Ensure your deployment is healthy by running:

    bosh vms
    
  2. Clean up the data from the failed backup on the instances by running:

    bbr deployment backup-cleanup
    

    Note: This step is required. Otherwise, further BBR commands fail.

  3. Repeat the BBR operation.

Error Running Metadata Script

Symptom

BBR backup or restore fails with a metadata error:

1 error occurred:
error 1:
An error occurred while running metadata script for job redis-server on redis/0ce9f81f-1756-480b-8e3e-a4609b14b6a6: error from metadata

Explanation

There is a problem with your Pivotal Platform install.

Solution

Contact Pivotal Support