LATEST VERSION: v1.4 - RELEASE NOTES
Pivotal Container Service v1.3

Backing up the PKS Control Plane

Page last updated:

This topic describes how to use BOSH Backup and Restore (BBR) to back up the PKS control plane.

Each PKS deployment includes custom backup and restore scripts which encapsulate the correct procedure for backing up and restoring the deployment.

BBR supports backing up only deployments supplying these scripts.

BBR orchestrates running the backup and restore scripts and transferring the generated backup artifacts to and from a backup repository. If configured correctly, BBR can use TLS to communicate securely with backup targets.

To perform a restore, see Restoring the PKS Control Plane.

To view the BBR release notes, see the Cloud Foundry documentation, BOSH Backup and Restore Release Notes.

Supported Components

BBR can backup the following components:

  • PKS control plane UAA MySQL database
  • PKS control plane PKS API MySQL database
  • BOSH Director
  • External blobstores
  • Kubernetes cluster etcd database

Unsupported Components

BBR can not be used to back up the following components:

  • Harbor tile
  • Cloud provider objects:
    • Persistent volumes
    • Network resources

Prerequisites

If you want to use the result of the backup to restore to a destination environment, verify that the current environment and the destination environment are compatible. For more information, see the Compatibility of Restore section of Restoring the PKS Control Plane.

Restoring the PKS Control Plane.

Before you begin backing up the PKS control plane, download the root CA certificate. To download the root CA certificate for your PKS deployment, perform the following steps:

  1. On the Ops Manager Installation Dashboard, in the top right corner, click your username.
  2. Navigate to Settings > Advanced.
  3. Click Download Root CA Cert.

Connect to Your Jumpbox

You can establish a connection to your jumpbox in one of the following ways.

For general information about the jumpbox, see Installing BOSH Backup and Restore.

Connect with SSH

SSH into your jumpbox. If you connect to your jumpbox with SSH, you must run the BBR commands in the following sections from within your jumpbox.

Connect with BOSH_ALL_PROXY

Set and use BOSH_ALL_PROXY. Using BOSH_ALL_PROXY opens an SSH tunnel with SOCKS5 to the jumpbox. This tunnel enables you to forward requests to the BOSH Director through the jumpbox from your local machine.

Use one of the following methods to create the tunnel:

  • Tunnel created by BOSH CLI: To provide the BOSH CLI with the SSH credentials it needs to create the tunnel, run the following command:

     export BOSH_ALL_PROXY=ssh+socks5://jumpbox@jumpbox-ip:12345?private_key=jumpbox.key
    
  • Tunnel established separately:

  1. To establish the tunnel and make it available on a local port, run the following command:

    ssh -4 -D 12345 -fNC jumpbox@jumpbox-ip -i jumpbox.key
    
  2. To provide the BOSH CLI with access to the tunnel through use of the BOSH_ALL_PROXY environment variable, run the following command:

    export BOSH_ALL_PROXY=socks5://localhost:12345
    

Note: Using BOSH_ALL_PROXY can result in longer backup and restore times due to network performance degradation. Because all operations must pass through the proxy, moving backup artifacts can be significantly slower.

Locate and Record the PKS Deployment Name

Locate and record your PKS BOSH deployment name as follows:

  1. On the Ops Manager Installation Dashboard, click the BOSH Director tile.
  2. In the BOSH Director tile, click the Credentials tab.
  3. Navigate to Bosh Command line Credentials and click Link to Credential.
  4. Copy the credential value.
  5. Open an SSH connection to either your jumpbox, as described in the previous section, or the Ops Manager VM. For instructions on how to SSH into the Ops Manager VM, see Advanced Troubleshooting with the BOSH CLI.
  6. On the command line, run the following command to retrieve your PKS BOSH deployment name.

    BOSH-CLI-CREDENTIALS deployments | grep pivotal-container-service
    

    Where BOSH-CLI-CREDENTIALS includes the full value that you copied from the BOSH Director tile. For example:

    $ BOSH_CLIENT=ops_manager BOSH_CLIENT_SECRET=p455w0rd BOSH_CA_CERT=/var/tempest/workspaces/default/root_ca_certificate BOSH_ENVIRONMENT=10.0.0.5 bosh deployments | grep pivotal-container-service
    
    pivotal-container-service-51f08f6402aaa960f041           backup-and-restore-sdk/1.8.0    bosh-google-kvm-ubuntu-xenial-go_agent/250.25
    service-instance_4ffeb5b5-5182-4faa-9d92-696d97cc9ae1    bosh-dns/1.10.0                 bosh-google-kvm-ubuntu-xenial-go_agent/250.25
    pivotal-container-service-51f08f6402aaa960f041
    
  7. In the output, look for the PKS BOSH deployment name that begins with pivotal-container-service and includes a unique identifier. In the example output above, the BOSH deployment name is pivotal-container-service-51f08f6402aaa960f041.

Back up the PKS Control Plane

  1. Run the BBR pre-backup check to confirm that your BOSH Director is reachable and has a deployment that can be backed up:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
    bbr deployment \
    --target BOSH-TARGET \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERT \
    pre-backup-check
    

    Replace the placeholder text using the information in the following table.

    Placeholder Text Instructions
    BOSH-CLIENT-SECRET In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT_SECRET.
    BOSH-TARGET In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_ENVIRONMENT. You must be able to reach the target address from the workstation where you run bbr commands.
    BOSH-CLIENT In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT.
    DEPLOYMENT-NAME Use the PKS BOSH deployment name that you located in the Locate and Record the PKS Deployment Names section above.
    PATH-TO-BOSH-CA-CERT Use the path to the root CA certificate that you downloaded in the Prerequisites section above.

    For example:

    $ BOSH_CLIENT_SECRET=p455w0rd \
    bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    pre-backup-check
  2. If the pre-backup check command fails, perform the following actions:

    • Run the command again, adding the --debug flag to enable debug logs. For more information, see BBR Logging.
    • Make any correction suggested in the output and run the pre-backup check again. For example, the deployment that you selected might not have the correct backup scripts, or the connection to the BOSH Director failed.
  3. If the pre-backup check succeeds, run the BBR backup command from your jumpbox to back up the PKS control plane:

    BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
    nohup bbr deployment \
    --target BOSH-TARGET \
    --username BOSH-CLIENT \
    --deployment DEPLOYMENT-NAME \
    --ca-cert PATH-TO-BOSH-SERVER-CERT \
    backup [--with-manifest] [--artifact-path]
    

    Replace the placeholder text using the information in the following table. These are the same values as shown in the previous table.

    Placeholder Text Instructions
    BOSH-CLIENT-SECRET In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT_SECRET.
    BOSH-TARGET In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_ENVIRONMENT. You must be able to reach the target address from the workstation where you run bbr commands.
    BOSH-CLIENT In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT.
    DEPLOYMENT-NAME Use the PKS BOSH deployment name that you located in the Locate and Record the PKS Deployment Names section above.
    PATH-TO-BOSH-CA-CERT Use the path to the root CA certificate that you downloaded in the Prerequisites section above.

    Specify optional flags for the backup command:
    Flag Description
    --with-manifest This includes the manifest in the backup artifact. If you use this flag, the backup artifact then contains credentials that you should keep secret.
    --artifact-path This allows you to specify a path for the backup artifact.

    For example:

    $ BOSH_CLIENT_SECRET=p455w0rd \
    nohup bbr deployment \
    --target bosh.example.com \
    --username admin \
    --deployment cf-acceptance-0 \
    --ca-cert bosh.ca.cert \
    backup
    

    Note: The BBR backup command can take a long time to complete. You can run it independently of the SSH session so that the process can continue running even if your connection to the jumpbox fails. The command above uses nohup, but you can run the command in a screen or tmux session instead.

  4. If the command completes successfully, follow the steps in Manage Your Backup Artifact below.

  5. If the backup command fails, perform the following actions:

Recover from a Failing Command

If the backup fails, follow these steps:

  1. Ensure that you set all the parameters in the backup command.
  2. Ensure the BOSH Director credentials are valid.
  3. Ensure the deployment that you specify in the BBR command exists.
  4. Ensure that the jumpbox can reach the BOSH Director.
  5. Consult BBR Logging.
  6. If you see the error message Directory /var/vcap/store/bbr-backup already exists on instance, run the appropriate cleanup command. See Clean up After a Failed Backup below.
  7. If the backup artifact is corrupted, discard the failing artifacts and run the backup again.

Cancel a Backup

Backups can take a long time. If you need to cancel a backup, for example if you realize that the backup is going to fail or that your developers need to push an app in a hurry, follow these steps:

  1. Terminate the BBR process by pressing Ctrl-C and typing yes to confirm.
  2. Because stopping a backup can leave the system in an unusable state and prevent additional backups, follow the procedures in Clean up After a Failed Backup below.

Clean up After a Failed Backup

If your backup process fails, it might leave the BBR backup directory on the instance, causing any subsequent attempts to backup to fail. In addition, BBR might not have run the post-backup scripts, leaving the instance in a locked state.

If the PKS control plane backup failed, run the following command to use the BBR cleanup script to clean up:

BOSH_CLIENT_SECRET=BOSH-CLIENT-SECRET \
bbr deployment \
--target BOSH-TARGET \
--username BOSH-CLIENT \
--deployment DEPLOYMENT-NAME \
--ca-cert PATH-TO-BOSH-CA-CERT \
backup-cleanup

Replace the placeholder text using the information in the following table. These are the same values as shown in the previous table.

Placeholder Text Instructions
BOSH-CLIENT-SECRET In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT_SECRET.
BOSH-TARGET In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_ENVIRONMENT. You must be able to reach the target address from the workstation where you run bbr commands.
BOSH-CLIENT In your BOSH Director tile, navigate to Credentials > Bosh Commandline Credentials. Record the value for BOSH_CLIENT.
DEPLOYMENT-NAME Use the PKS BOSH deployment name that you located in the Locate and Record the PKS Deployment Names section above.
PATH-TO-BOSH-CA-CERT Use the path to the root CA certificate that you downloaded in the Prerequisites section above.

For example:

$ BOSH_CLIENT_SECRET=p455w0rd \
bbr deployment \
--target bosh.example.com \
--username admin \
--deployment cf-acceptance-0 \
--ca-cert bosh.ca.crt \
backup-cleanup

Manage Your Backup Artifact

Keep your backup artifact safe by following these steps:

  1. Move the backup artifact off the jumpbox to your storage space. BBR stores each backup in a subdirectory named DEPLOYMENT-TIMESTAMP within the current working directory. The backup created by BBR consists of a directory with the backup artifacts and metadata files.

  2. Compress and encrypt the backup artifacts when storing them.

  3. Make redundant copies of your backup and store them in multiple locations. This minimizes the risk of losing your backups in the event of a disaster.

  4. Each time you redeploy PKS, test your backup artifact by following the procedures in Restoring the PKS Control Plane.


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub