Restoring from a BBR Backup

ivotal recommends that you create and restore Concourse backups with BOSH Backup and Restore (BBR). The process for restoring involves the following prerequisites:

A valid BBR backup of Concourse. If you have not used BBR before, see the Performing a backup with BBR guide to get started.
bbr, the BOSH Backup and Restore (BBR) CLI tool
fly, Concourse's CLI tool

With those conditions met, you can use this guide to perform a restore operation.

Reasons to restore Concourse from a backup include:

Something went wrong with your Concourse pipelines
You want to validate that a backup works
An upgrade/downgrade failed

Creating and then validating backups can even be automated into a Concourse pipeline.

Note about restoring Concourse

Currently only restoring Concourse to a fresh deployment is supported.

If there is existing data, errors can arise when restoring Concourse.

Restore Procedure

This guide assumes you have done a bbr backup on some Concourse instance and you want to restore the backup to a new deployment.

Go to install docs to create a new deployment and use the backup-atc.yml ops-file. This ops-file will ensure the bbr-atcdb and database-backup-restorer jobs are on the same VM instance as PostgreSQL.

At this point, you should have a BOSH deployment of Concourse and PostgreSQL that you can target.

If your PostgreSQL instance where the Concourse atc database is located is managed separatly from your Concourse BOSH deployment you will have to ensure that the bbr-atcdb and database-backup-restorer jobs are on the same VM instance as PostgreSQL.

Configure the database with `superuser` capabilities for Concourse to enable `pg_statements`

There are two methods that can be used to address this step. Please read both methods below and choose the one that seems most appropriate.

Method 1

Log into Concourse's postgres instance using psql or your preferred database client
1 2
# ssh onto the vm where postgres is running bosh -d DEPLOYMENT ssh INSTANCE_NAME
Where:
- DEPLOYMENT is the BOSH deployment where PostgreSQL is running
- INSTANCE_NAME is the VM instance where PostgreSQL is running
INSTANCE_NAME

To figure out what INSTANCE_NAME should be you can run bosh vms to see what instance names are part of your BOSH deployment. For example:
1
bosh -d DEPLOYMENT vms

Once you're ssh'd on the VM and you have access to a user (DB_USER) with the superuser permission, you can login using psql.

$ /var/vcap/packages/postgres-11.5/bin/psql -d atc -U DB_USER -W
Password: <input password>
psql (11.5)
Type "help" for help.

atc=>

Run ALTER USER <concourse_db_user> WITH SUPERUSER;

Method 2

Modify the PostgreSQL manifest to elevate the Concourse role to SUPERUSER. Fill out the databases.roles[n].permissions field in the postgres job.

For example:

  - name: postgres
    release: postgres
    properties:
      databases:
        databases:
        - name: atc
        port: 5432
        roles:
        - name: concourse
          password: ((concourse_db_user.password))
          permissions:
          - "SUPERUSER"

Deploy Concourse database instance

After updating the BOSH manifest to give the database user superuser permissions, the database will have to be redeployed.

If part of Concourse deployment, use BOSH to redeploy the db vm.
If separate from the Concourse deployment, follow infrastructure specific instructions.

`bbr restore`

Run the bbr deployment restore command:
1 2 3 4
bbr deployment \ --deployment TARGET-CONCOURSE \ restore \ --artifact-path PATH-TO-BACKUP
Where:
- TARGET-CONCOURSE is your target deployment, and
- PATH-TO-BACKUP is the path to the backup directory you wish to restore from.
Make sure that you are running the same version of Concourse that your backup was created from.

Verify your pipelines have been successfully restored

Run the following command to verify that the state of your pipelines has been restored:
1
fly -t TARGET-CONCOURSE pipelines
Where TARGET-CONCOURSE is your target deployment.

Unpause pipelines

Troubleshooting

Broken git Resources

Symptom: After a restore, git repository resources appear broken with an error like this:

1	`unknown handle: UUID`

Solution: Recreate the pipelines by running fly set-pipeline.

Cannot Drop Table

Symptom: There's an error when restoring Concourse like this:

pg_restore: dropping TABLE build_events
pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 201; 1259 19912 TABLE build_events concourse
pg_restore: [archiver (db)] could not execute query: ERROR:  cannot drop table public.build_events because other objects depend on it
DETAIL:  table public.pipeline_build_events_4 depends on table public.build_events
table public.pipeline_build_events_5 depends on table public.build_events
table public.pipeline_build_events_6 depends on table public.build_events
HINT:  Use DROP ... CASCADE to drop the dependent objects too.
    Command was: DROP TABLE IF EXISTS public.build_events;

You may need to delete the artifact-file that was created before re-running.
exit status 1 - exit code 1

Solution: Restore Concourse to a fresh deployment. This happens when restoring Concourse to a deployment with existing data.

Unregistered Redirect URI

Symptom: When logging in after BBR restoration, there's an error like this:

1 2	`Bad Request Unregistered redirect_uri ("(CONCOURSE_URL)/sky/callback").`

Solution: Stop and restart the Concourse web instance(s).

1	`bosh --deployment TARGET-CONCOURSE stop web`

1	`bosh --deployment TARGET-CONCOURSE start web`