LATEST VERSION: v1.1 - RELEASE NOTES
Pivotal Container Service v1.1

Back Up the PKS Control Plane

Page last updated:

This topic describes how to use BOSH Backup and Restore (BBR) to back up the PKS control plane.

To perform a restore, see Restore the PKS Control Plane.

Prerequisites

If you want to use the result of the backup to restore to a destination environment, verify that the current environment and the destination environment are compatible. For more information, see Compatibility of Restore.

Before you begin backing up the PKS control plane, perform the following steps:

  1. Download the root CA certificate for your PKS deployment:

    1. From the Ops Manager Installation Dashboard, click your username in the top right corner.
    2. Navigate to Settings > Advanced.
    3. Click Download Root CA Cert.
  2. Locate your PKS BOSH deployment name:

    1. From the Ops Manager Installation Dashboard, click the Director tile.
    2. Click the Credentials tab.
    3. Navigate to Bosh Commandline Credentials and click Link to Credential.
    4. Copy the credential value.
    5. From the command line, run the following command to retrieve your PKS BOSH deployment name, replacing BOSH-CLI-CREDENTIALS with the credential value you copied in the previous step:
      BOSH-CLI-CREDENTIALS deployments | grep pivotal-container-service
      Your PKS BOSH deployment name begins with pivotal-container-service and includes a unique identifier.

Back Up the PKS Control Plane

Perform the following steps to check that your BOSH Director is reachable and has a deployment that can be backed up:

  1. SSH into your jumpbox.
    For general information about the jumpbox, see Install BOSH Backup and Restore.

  2. Run the BBR pre-backup check:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET bbr deployment --target BOSH-TARGET --username BOSH-CLIENT --deployment DEPLOYMENT-NAME --ca-cert PATH-TO-BOSH-SERVER-CERT pre-backup-check


    Where:

    Credential Location
    BOSH-CLIENT-SECRET In the BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT_SECRET.
    BOSH-TARGET In the BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_ENVIRONMENT. You must be able to reach the target address from the workstation where you run bbr commands.
    BOSH-CLIENT In the BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT.
    PATH-TO-BOSH-CA-CERT Use the path to the root CA certificate you downloaded in the Prerequisites section.
    DEPLOYMENT-NAME Use the PKS BOSH deployment name you located in the Prerequisites section.

    For example:

    $ BOSH_CLIENT_SECRET=p455w0rd \
    bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    pre-backup-check
  3. If the pre-backup check command fails, do the following:

    • Run the command again adding the --debug flag to enable debug logs.
      For more information, see Exit Codes and Logging.
    • Make the fix suggested in the output and run the pre-backup check again.
      For example, the deployment you selected might not have the correct backup scripts, or the connection to the BOSH Director failed.
  4. If the pre-backup check succeeds, run the BBR backup command from your jumpbox to back up the PKS control plane:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET nohup bbr deployment --target BOSH-DIRECTOR-IP --username BOSH-CLIENT --deployment DEPLOYMENT-NAME --ca-cert PATH-TO-BOSH-SERVER-CERT backup

    Replace the placeholder values with the same values as above.

    Note: If you want to include the manifest in the backup artifact, add the --with-manifest flag. However, be aware that the backup artifact then includes credentials that you must keep secret.

    For example,

    $ BOSH_CLIENT_SECRET=p455w0rd \
        nohup bbr deployment \
        --target bosh.example.com \
        --username admin \
        --deployment cf-acceptance-0 \
        --ca-cert bosh.ca.cert \
        backup

    Note: The BBR backup command can take a long time to complete. You can run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup, but you could also run the command in a screen or tmux session.

  5. If the command completes successfully, follow the steps in Good Practices for Managing Your Backup Artifact below.

  6. If the backup command fails, do the following:

Recovering from a Failing Command

If the backup fails, follow these steps:

  1. Ensure all the parameters in the command are set.
  2. Ensure the BOSH Director credentials are valid.
  3. Ensure the deployment you specify in the BBR command exists.
  4. Ensure that the jumpbox can reach the BOSH Director.
  5. Consult Exit Codes and Logging.
  6. If you see the error message Directory /var/vcap/store/bbr-backup already exists on instance, run the appropriate cleanup command. See Clean Up after a Failed Backup below.
  7. If the backup artifact is corrupted discard the failing artifacts and rerun the backup.

Cancel a Backup

Backups can take a long time. If you need to cancel a backup, for example if you realize that the backup is going to fail or that your developers need to push an app in a hurry, follow these steps:

  1. Terminate the BBR process by pressing Ctrl-C and typing yes to confirm.
  2. Because stopping a backup can leave the system in an unusable state and prevent additional backups, follow the procedures in Clean Up after a Failed Backup below.

Clean Up after a Failed Backup

If your backup process fails, it might leave the BBR backup folder on the instance, causing any subsequent attempts to backup to fail. In addition, BBR might not have run the post-backup scripts, leaving the instance in a locked state.

Follow the steps below to use the BBR cleanup script to tidy up after a failed backup attempt.

  1. If the PKS control plane backup failed, run the following command:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET bbr deployment --target BOSH-TARGET --username BOSH-CLIENT --deployment DEPLOYMENT-NAME --ca-cert PATH-TO-BOSH-CA-CERT backup-cleanup

    For example,

    $ BOSH_CLIENT_SECRET=p455w0rd \
    bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.crt \
    backup-cleanup
    
  2. If the cleanup script fails, consult the following table to match the exit codes to an error message.

    Value Error
    0 Success
    1 General failure
    8 The post-backup unlock failed. Your deployment might be in a bad state and require attention.
    16 The cleanup failed. This is a non-fatal error indicating that the utility has been unable to clean up open BOSH SSH connections to the deployment VMs. Manual cleanup might be required to clear any hanging BOSH users and connections.


    For more information about how to interpret the exit code, see Exit Codes.

Good Practices for Managing Your Backup Artifact

Keep your backup artifact safe by following these steps:

  1. Move the backup artifact off the jumpbox to your storage space.

    BBR stores each backup in a subdirectory named DEPLOYMENT-TIMESTAMP within the current working directory. The backup created by BBR consists of a folder with the backup artifacts and metadata files.

  2. Compress and encrypt the backup artifacts when storing them.

  3. Make redundant copies of your backup and store them in multiple locations.

    This minimizes the risk of losing your backups in the event of a disaster.

  4. Each time you redeploy PKS, test your backup artifact by following the procedures in Restore the PKS Control Plane.


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub