Backup and Restore Concourse with BOSH Backup and Restore (BBR)
Already backing up your Concourse?
If you already have a backup process implemented you should still verify that your backup process or job has created a recent backup, and validate that it is healthy before moving on. You can then skip this topic and move on to Upgrading your Concourse release.
Pivotal recommends that you perform backups with BOSH Backup and Restore (BBR). The process involves three parts:
- Deploying the BOSH Backup and Restore SDK, deployed as a release to your BOSH environment
- Installing the BOSH Backup and Restore CLI client
- Performing the backup
If you have not used BBR before, the next section has setup instructions.
If you have already configured your Concourse deployment with a
backup-and-restore-sdk release and already have the BBR client installed locally, you can skip to Performing a Backup below.
Tip: You can use the
bosh releases command to see what you have installed so far.
concourse-bosh-deployment Directory on Your Local Machine
To back up a deployed Concourse, you need to update that deployment with the BBR
backup-and-restore-sdk. If you are starting from scratch on a new workstation, the instructions in this section cover cloning the
concourse-bosh-deployment repository to a local directory, so that you can craft a new deployment with the appropriate release and configuration.
If you already have this repository cloned to a local directory, you can skip Step 1
Clone the concourse-bosh-deployment repository by running the following command:
git clone https://github.com/concourse/concourse-bosh-deployment.git
Move to the
All the paths used in this topic are relative to this directory.
Checkout the release that corresponds to your working version of Concourse. For example, if you are backing up Concourse 3.13.3:
git checkout v3.13.3
For a list of all Concourse releases, see concourse-bosh-deployment in GitHub.
Checking out a release rather than a branch means that git produces the following output:
1 2 3 4 5 6 7 8 9 10 11 12
Note: checking out 'RELEASE'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b <new-branch-name> HEAD is now at HASH... COMMIT-MESSAGE
Set Up Certificates, Log In, and Alias Your BOSH Environment
If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.
Get a CA certificate for your BOSH Director.
If you created your BOSH Director manually, retrieve the credentials that were created during setup.
If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:
OPS-MANAGER-DOMAINis the Ops Manager domain.
The JSON response you receive has a key-value pair. The CA certificate is the returned value, between the quotation marks, that starts with
-----BEGIN CERTIFICATE-----and ends with
Copy and paste the CA certificate into a file, with all instances of
\nreplaced by carriage returns.
Pasting the CA Certificate
In most shells in OSX, you can use the
pbpastecommand to access the contents of your clipboard. After copying the CA certificate to your clipboard, you can then run this command to format it correctly when pasting:
echo -e "$(pbpaste)" > ca-cert.yml
Alternatives for Pasting the CA Certificate
Two alternatives for pasting the CA certificate include:
- Using the command-line JSON processor
jqto translate the JSON when pasting
- Pasting the value into a file and replacing the newline characters with carriage returns using the find and replace operation in a text editor
- Using the command-line JSON processor
Ensure the file is formed correctly, with
-----BEGIN CERTIFICATE-----at the beginning and
-----END CERTIFICATE-----at the end, by running:
This certificate file can be used whenever a CLI command asks for a
--ca-certflag and value.
Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as
ca-cert.yml, run the following command:
bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.yml
BOSH-ENVIRONMENT-IPis your BOSH environment IP address.
Finding director credentials
If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.
If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:
OPS-MANAGER-DOMAINis your Ops Manager domain.
Give your environment an alias by running the following command:
bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.yml
BOSH-ENVIRONMENT-IPis your BOSH environment IP address
ALIASis the alias you're creating BOSH environment
Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.
You can use this alias whenever you target this environment by using the
-eflag in a BOSH command.
Set Up the BBR SDK on Your BOSH Environment
This section details the task of uploading the BOSH
backup-and-restore-sdk release to a Concourse environment.
backup-and-restore-sdkrelease by running the
1 2 3 4 5
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ upload-release \ --sha1 364838c384f2edec80866b4abf2397c4c5d15c62 \ https://bosh.io/d/github.com/cloudfoundry-incubator/backup-and-restore-sdk-release?v=1.15.1
An alternative approach is to download the latest release from the BOSH Backup and Restore Release Page and pass in the path to the file in the following format:
bosh -e BOSH-ENVIRONMENT-ALIAS upload-release PATH-TO-FILE
If you run the
bosh releasescommand again, you should see
backup-and-restore-sdkin the list now.
Create and edit a file called
./cluster/operations. For example:
Edit this file so that it has the following contents:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
- type: replace path: /releases/- value: name: backup-and-restore-sdk url: https://bosh.io/d/github.com/cloudfoundry-incubator/backup-and-restore-sdk-release?v=((bbr_sdk_version)) sha1: ((bbr_sdk_sha1)) version: ((bbr_sdk_version)) - type: replace path: /instance_groups/name=web/jobs/- value: name: bbr-atcdb properties: postgresql: database: atc role: name: ((postgres_role)) password: ((postgres_password)) postgresql_database: atc release: concourse - type: replace path: /instance_groups/name=web/jobs/- value: name: database-backup-restorer release: backup-and-restore-sdk
Save and close the
versions.ymlby adding lines for the backup-and-restore-sdk's version and SHA1. The two properties you need to define are
To deploy backup-and-restore-sdk 1.15.1, copy and paste the highlighted lines into your
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
# this file is partially maintained by CI; the concourse and garden-runc # versions and sha1s are automatically bumped, while the rest are preserved # as-is. # # this should make getting started easy while being easy enough to maintain # manually. feel free to PR sane defaults along with newly supported # infrastructures and such! --- concourse_version: '3.13.3' concourse_sha1: 'aaa4e4d42adb2293abc79422351ca71ed548f95c' garden_runc_version: '1.13.1' garden_runc_sha1: '54cbb89cae1be0708aa056185671665d7f4b2a4f' postgres_version: '28' postgres_sha1: 'c1fcec62cb9d2e95e3b191e3c91d238e2b9d23fa' virtualbox_cpi_version: '0.0.12' virtualbox_cpi_sha1: 537c77b087eeda859b1eb89553efa6f808214d50 virtualbox_stemcell_version: '3468.5' virtualbox_stemcell_sha1: 25301a41c15caad859b66efc0fd75bacdfc57dab bbr_sdk_version: '1.15.1' bbr_sdk_sha1: '364838c384f2edec80866b4abf2397c4c5d15c62'
If you want to use a different version of
backup-and-restore-sdk-release, see backup-and-restore-sdk Release to find the version and SHA1 for your release.
Concourse and BOSH Variables
Deploying Concourse with BOSH requires you to set some variables to fill in details in the manifest that are written as placeholders. These are in the format
((placeholder)). The currently deployed Concourse manifest contains the values that you will use to fill out each of the variables.
The preferred approach is to create a file to define a key/value pair for each required variable, and check it into version control so that it can be safely and easily maintained over time. You might already have a solution for this; if not, follow the steps below.
The following steps describe setting variables that are necessary for deployment in a yaml file. If you prefer to set them on the command line when you run
bosh deploy, you can pass them in with the
--var KEY=VALUE flag and syntax. For more information about
bosh deploy, see Deploy in the BOSH documentation.
Run the following command to fetch the manifest and put it in a
1 2 3 4
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ -d NAME-OF-CONCOURSE-DEPLOYMENT \ manifest > deployed-manifest.yml
Sending the response to a file is a security precaution that ensures that any included keys aren't echoed to your terminal session history. Remember to destroy this file once you're done with this backup process!
If you do not already have a variable file from a previous deployment, create a file to store your Concourse and BOSH-related environment variables. For example, use vim to create a file called
This file will hold all of the variables that BOSH needs to interpolate into references in the manifest file.
Why create a file for these variables?
The benefit of doing things this way is that you can check this file into your version control, track changes over time, and more easily document and share this information with others in your organization. This also saves you from looking up configuration details every time you need to change your Concourse deployment).
Ensure you have the correct key/value pairs in your variables file:
- If you already have a variables file: Ensure that you have each of the key/value pairs listed in the snippet below. If any are missing, add them in.
- If you have created a new file: Copy and paste the following snippet to use as a template. Replace each value placeholder in the snippet above as appropriate for your deployment and cloud configuration, using the properties in the
deployed-manifest.ymlfile that created in step 1 of this section.
1 2 3 4 5 6 7 8 9 10 11
--- deployment_name: DEPLOYMENT-NAME db_persistent_disk_type: PERSISTANT-DISK-TYPE-FOR-DATABASE db_vm_type: VM-TYPE-FOR-DATABASE-INSTANCE external_url: EXTERNAL-URL network_name: NETWORK-NAME postgres_password: PASSWORD web_ip: WEB-IP web_network_name: WEB-NETWORK-NAME web_vm_type: VM-TYPE-FOR-WEB-INSTANCE worker_vm_type: VM-TYPE-FOR-WORKER-INSTANCE
DEPLOYMENT-NAMEis your Concourse deployment name. You can find this by running:
bosh -e BOSH-ENVIRONMENT-ALIAS deployments
NETWORK-NAMEis the name subproperty of the network property in your deployed manifest.
PASSWORDis the password sub-property of the postgres property in your deployed manifest.
If your Concourse is deployed with something like CredHub, these variables might be automatically interpolated after instead of during deployment. If you suspect this is the case, one way to check is with the following command:
1 2 3 4
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ -d TARGET-CONCOURSE-DEPLOYMENT \ manifest | grep “(("
This searches through your deployed manifest for any placeholders that are being interpolated after deployment. If this returns any values, this could indicate that you have something like CredHub involved. Before you proceed, stop to investigate your current deployment, or get in touch with whomever has the necessary context.
Save and close the file.
Confirm Your Availability Zones and Stemcells
The Concourse manifest has some defaults set for availability zones (AZs). You need to modify these defaults to suit a given environment. To ensure these are set correctly, follow these steps:
Retrieve the values of the
azsproperties in your currently deployed manifest.
To find these, reference the
deployed-manifest.ymlfile created in the last section, or run the
bosh manifestcommand to re-create it:
1 2 3 4
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ -d CONCOURSE-DEPLOYMENT-NAME \ manifest > deployed-manifest.yml
concourse-bosh-deploymentrepository's cluster Concourse manifest file:
Search through this manifest file to find any
azsproperties and edit each one of these so that they match the availability zones (azs) in your
deployed-manifest.ymlfile. There should be three in total, one in each instance group.
For example, if your availability zone is
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
... instance_groups: - name: web ... azs: [default] ... - name: db ... azs: [default] ... - name: worker ... azs: [default] ... ...
Next, search for every
./cluster/concourse.ymland ensure the stemcell names match those you noted from your
Save and close the
Deploy Your Updated Concourse Deployment with BBR
Deploy the new setup to your BOSH environment. The
bosh deploy command uses a manifest that acts as a recipe for the deployment that you are creating on your cloud environment. When you run
bosh deploy, you need to incorporate the following:
- Your new concourse manifest, for example,
- Your versions file, for example,
- Your environment variables, for example,
- Any required operations files, including the
- A filename where BOSH can record and access cluster credentials, for example,
Operations files make substitutions in your manifest file to customize your deployment. The example below demonstrates a simple setup with basic authentication.
To change the authentication type, change the
-o ./cluster/operations/basic-auth.yml line.
Edit and run this snippet, substituting your environment alias and deployment name, along with any additional ops files you want to add.
1 2 3 4 5 6 7 8
bosh -e BOSH-ENVIRONMENT-ALIAS deploy -d CONCOURSE-DEPLOYMENT-NAME ./cluster/concourse.yml \ -l versions.yml \ -l variables.yml \ -o ./cluster/operations/bbr.yml \ -o ./cluster/operations/no-auth.yml \ -o ./cluster/operations/privileged-http.yml \ -o ./cluster/operations/static-web.yml \ --vars-store cluster-creds.yml
When prompted to review the deployment, you should see the addition of the backup-and-restore-sdk and its configuration. If you continue, BOSH prepares the deployment and starts compiling packages for you.
If you have CPI errors, try running the deployment script again. In rare cases the operation can time out while packages are being built, causing a CPI timeout. The deployment should be successful if you re-run it a second time, as by that point the packages are already built.
If you're curious about how this operations file affects the release by logging in to the web VM with SSH and checking the contents of the
1 2 3 4
bosh -d concourse ssh web/0 sudo su cd /var/vcap/jobs ls
You should see the database-backup-restorer in the directory.
Once you have verified the contents of the directory, run the command
exittwice to close your ssh session and return to your working directory.
Install the BOSH Backup and Restore CLI
To install the BBR CLI on your computer, do the following:
Check if BBR is already installed by running the following command. If this returns a version number instead of an error, you can skip to the next section.
Download the latest release's asset file that corresponds to your operating system from the bosh-backup-and-restore GitHub repository. For example, Apple OSX users should download the darwin-amd64 file.
There is no Windows release for the BBR CLI. If you are on Windows, you can SSH into a jumpbox where you can use a linux release of BBR. For instructions, see Installing BOSH Backup and Restore in the Cloud Foundry documentation. You can then continue to the Perform a Backup section below.
Move the file into your local /bin and rename it to
bbrby running the following command:
mv ~/Downloads/BBR-BINARY-FILE-NAME /usr/local/bin/bbr
Make the file executable by running the following command:
chmod +x /usr/local/bin/bbr
Performing a Backup
To run a backup with the
bbr command, do the following:
Retrieve the UAA BBR Client Credentials, username and password, from the following endpoint on your environment:
bbrwith the following command:
In the following command, use the environment IP address instead of using the environment alias you used with the
1 2 3 4 5 6 7
bbr deployment \ --target ENVIRONMENT-IP-ADDRESS \ --deployment TARGET-CONCOURSE-DEPLOYMENT-NAME \ --username UAA-BBR-USERNAME \ --password UAA-BBR-PASSWORD \ --ca-cert CA-CERTIFICATE-FILE \ backup
A successful backup results in the creation of a directory named after your deployment name and the date and time of the backup.
Move the backup directory to a secure location.
This backup might include sensitive information, such as passwords in builds, depending on how you have configured your pipelines.
When you are ready to continue with the upgrade process, proceed to the Upgrade Concourse guide.
If you need to restore a backup, see Performing a Restore.