Skip to content
Support Downloads Contact Us Sign In

Performing the Concourse Upgrade

Upgrading Concourse involves following these steps:

  • Download the latest supported Concourse release
  • Upload it to your BOSH environment
  • Create a new deployment manifest
  • Use BOSH to deploy this new manifest in place of your existing Concourse deployment

This step-by-step tutorial includes details for checking out sample manifest and supporting files for the desired deployment version from the concourse-bosh-deployment repository. These can be used as a template for customizing your release.

The guide below is written to provide necessary details for someone starting this process from scratch on a new workstation. You can skip certain steps if you have deployed from concourse-bosh-deployment before and already have a working directory, credentials, and/or cloud configuration variables.

If you have not backed up your current Concourse deployment, follow the steps in Backup Concourse before proceeding.

Set up the concourse-bosh-deployment Directory on Your Local Machine

Skip step 1 if you already have concourse-bosh-deployment cloned to a local directory.

  1. Clone the concourse-bosh-deployment repository by running the following command:

    1
    git clone https://github.com/concourse/concourse-bosh-deployment.git

  2. Navigate to the concourse-bosh-deployment directory by running the following command:

    1
    cd concourse-bosh-deployment

    All the paths used in this guide are relative to this directory.

    Check your Git Status

    Pivotal recommends checking your Git status whenever you enter the concourse-bosh-deployment directory. Do so by running the following command:

    1
    git status

    If you encounter unstaged files that can be overwritten by checking out the new release branch, git stash your changes, check out the release, and then git stash pop and resolve any merge conflicts.

  3. Check out the concourse-bosh-deployment Git release that corresponds to your desired upgrade version. For example:

    1
    git checkout v4.2.4

    Tip

    Run git tag to see a list of the different releases.

    Checking out a release, rather than a branch, produces the following output:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
    Note: checking out RELEASE.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, use `-b` with the checkout command again. For example:

  git checkout -b <new-branch-name>

HEAD is now at HASH COMMIT-MESSAGE

    Where:

    RELEASE is the release version number.

    HASH is the hash key of the most recent commit.

    COMMIT-MESSAGE is the most recent commit message.

Set Up Certificates, Log In, and Alias Your BOSH Environment

Note

If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.

  1. Get a CA certificate for your BOSH Director.

    1. If you created your BOSH Director manually, retrieve the credentials that were created during setup.

      If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:

      1
      https://OPS-MANAGER-DOMAIN/api/v0/security/root_ca_certificate

      Where OPS-MANAGER-DOMAIN is the Ops Manager domain.

      The JSON response you receive has a key-value pair. The CA certificate is the returned value, between the quotation marks, that starts with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----.

    2. Copy and paste the CA certificate into a file, with all instances of \n replaced by carriage returns.

      Pasting the CA Certificate

      In most shells in OSX, you can use the pbpaste command to access the contents of your clipboard. After copying the CA certificate to your clipboard, you can then run this command to format it correctly when pasting:

      1
       echo -e "$(pbpaste)" > ca-cert.yml
      Alternatives for Pasting the CA Certificate

      Two alternatives for pasting the CA certificate include:

      1. Using the command-line JSON processor jq to translate the JSON when pasting
      2. Pasting the value into a file and replacing the newline characters with carriage returns using the find and replace operation in a text editor

    3. Ensure the file is formed correctly, with -----BEGIN CERTIFICATE----- at the beginning and -----END CERTIFICATE----- at the end, by running:

      1
      cat ca-cert.yml

      Tip

      This certificate file can be used whenever a CLI command asks for a --ca-cert flag and value.

  2. Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as ca-cert.yml, run the following command:

    1
    bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.yml

    Where BOSH-ENVIRONMENT-IP is your BOSH environment IP address.

    Finding director credentials

    If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.

    If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:

    1
    http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials

    Where OPS-MANAGER-DOMAIN is your Ops Manager domain.

  3. Give your environment an alias by running the following command:

    1
    bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.yml

    Where:

    • BOSH-ENVIRONMENT-IP is your BOSH environment IP address
    • ALIAS is the alias you're creating BOSH environment

    Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.

    Tip

    You can use this alias whenever you target this environment by using the -e flag in a BOSH command.

Postgres Version

In the latest release for 4.2.x of Concourse, v4.2.4, the suggested postgres version is release 36. This works great for clean installations, but when upgrading from v3.x.x releases, it's a good idea to take a precautionary step:

  1. Open versions.yml in your favorite editor. Comment out the existing version and SHA1 values for Postgres, and add the following lines for version 30 in their place. The postgres section looks like this:

    1
2
3
4
    # postgres_version: '36'
# postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'
postgres_version: '30'
postgres_sha1: 'a798999d29b9f5aa12035cff907b26674b491200'

Why are we doing this?

Users upgrading from 3.13.x who currently have Postgres release 28 (Postgres version 9.6.8) might encounter an error during their deployment if they try to upgrade directly to Postgres release 32.

This error wastes a lot of time, and can be tricky to correct, so Pivotal recommends this additional step as a precaution. By first upgrading to release 30, users can subsequently upgrade to release 36 without errors:

  1. From Postgres Release 28 (9.6.8) to Postgres Release 30 (9.6.10), then
  2. From Postgres Release 30 (9.6.10) to Postgres Release 36 (11.2)

BOSH and Concourse Variables

Deploying with BOSH requires setting some variables to fill in details in your manifest. These are written as placeholders in the following format: ((placeholder)).

You can do this in a number of ways, but Pivotal recommends creating a file to define a key-value pair for each required variable, and checking it into version control so that it can be safely and easily maintained over time. Follow the steps below to do so.

Tip

The following steps describe setting variables that are necessary for deployment in a YAML file. If you prefer to set them on the command line when you run bosh deploy, you can pass them in at that time with the --var KEY=VALUE flag and syntax. For more information, see Deploy in the BOSH documentation.

  1. To find the values needed to fill out each of the variables in the next step, check the currently deployed Concourse manifest. Use the following command to tell BOSH to fetch the manifest and put it in a deployed-manifest.yml file:

    1
2
3
4
    bosh \
-e BOSH-ENVIRONMENT-ALIAS \
-d CONCOURSE-DEPLOYMENT-NAME \
manifest > deployed-manifest.yml

    Where BOSH-ENVIRONMENT-ALIAS is your BOSH environment alias and CONCOURSE-DEPLOYMENT-NAME is your Concourse deployment name.

    This command produces a file that contains a value for each of the variables that you define in the next step.

    Sending the response to a file is a security precaution that ensures that any included keys are not echoed to your terminal session history. Destroy this file after you are done with this upgrade process.

  2. Create a file to store your Concourse- and BOSH-related environment variables, if you do not already have one from a previous deployment. For example, the following command uses Vim to create a file called variables.yml:

    1
    vim variables.yml

    This file holds all the variables that BOSH needs to interpolate into references in the manifest file. The manifest file then serves as a recipe for the new deployment.

    Why create a file for these variables?

    The benefit to doing things this way is that you can check this file into your version control, track changes over time, and more easily document and share this information with others in your org. It also saves you the trouble of looking up configuration details every time you need to change your Concourse deployment.

  3. Add the key-value pairs shown in the snippet below into the file.

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
    ---
local_user:
  username: USERNAME
  password: PASSWORD

deployment_name: DEPLOYMENT-NAME
db_persistent_disk_type: PERSISTENT-DISK-TYPE
db_vm_type: VM-TYPE
external_url: EXTERNAL-URL
network_name: NETWORK-NAME
postgres_password: POSTGRESQL-PASSWORD
web_ip: WEB-IP
web_network_name: WEB-NETWORK-NAME
web_vm_type: WEB-VM-TYPE
worker_vm_type: WORKER-VM-TYPE

    Where:

    • DEPLOYMENT-NAME is your deployment name

    • PERSISTENT-DISK-TYPE is the persistent disk type for the database

    • VM-TYPE is the VM type for the database instance

    • EXTERNAL-URL is the external URL

    • NETWORK-NAME is the name subproperty of the network property in your deployed manifest

    • POSTGRESQL-PASSWORD is the password subproperty of the PostgreSQL property in your deployed manifest

    • WEB-IP is your web IP

    • WEB-NETWORK-NAME is your web network name

    • WEB-VM-TYPE is the VM type for the web instance

    • WORKER-VM-TYPE is the VM type for the worker instance

  4. Replace each value placeholder with the appropriate properties in the deployed-manifest.yml file that you created earlier.

    Optional Tip:

    If Concourse is deployed with a component such as CredHub, these variables might be automatically interpolated. You can check if this is the case by running the following command:

    1
2
3
4
    bosh \
-e BOSH-ENVIRONMENT-ALIAS \
-d TARGET-CONCOURSE-DEPLOYMENT \
manifest | grep “(("

    If the search returns any values, CredHub or a similar component might be involved. In this case, investigate your current deployment, or get in touch with whoever has the necessary context, before proceeding further.

  5. Save and close your variables file.

Confirm your availability zones and stemcells

Your deployed manifest, located at ./cluster/concourse.yml, has some defaults set for availability zones (AZs). Often these must be customized to suit a given environment. To ensure these are set correctly, follow these steps:

  1. Check the azs (availability zones) and the stemcell names in your currently deployed manifest.

    To find these, you can reference the deployed-manifest.yml file we created in the last section, or run the bosh manifest command to re-create it:

    1
2
3
4
    bosh \
-e BOSH-ENVIRONMENT-ALIAS \
-d CONCOURSE-DEPLOYMENT-NAME \
manifest > deployed-manifest.yml

  2. Open the Concourse manifest file of the concourse-bosh-deployment repository cluster by running the following command:

    1
    vim ./cluster/concourse.yml

  3. Change each of the three az properties in the manifest file so that they match the AZs in your deployed-manifest.yml file. For example, if your AZ is default, your manifest file looks like this:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    ...
instance_groups:
- name: web
  ...
  azs: [default]
  ...

- name: db
  ...
  azs: [default]
  ...

- name: worker
  ...
  azs: [default]
  ...
...

  4. Next, search for every stemcell mentioned in ./cluster/concourse.yml and ensure the stemcell names match those you noted from your deployed manifest in step 12.

    Find and Replace

    Once you've identified whether you need to change azs or stemcells, you can use a find-and-replace operation in your favorite editor to change all these values at once. For example, if the Concourse Manifest has the azs property set to [z1] and you need to change it to [default], in vim you could use the following command:

    1
    :s/z1/default/g

    If you need to change the stemcell from trusty to xenial, in vim you could use the following command:

    1
    :s/trusty/xenial/g

  5. Save and close the ./cluster/concourse.yml file.

bosh deploy Concourse to Your Cloud Environment

Tip for upgrading very large Concourses

If you have many pipelines with many containers, pause your pipelines before you start the upgrade. Then, after you finish the upgrade, wait for all of your workers to be ready and then turn on your pipelines one by one.

Pivotal recommends doing this to prevent your pipelines from overwhelming the first workers that come online, which could cause a crash.

To deploy your new Concourse version, you need the following information:

  • Your new concourse manifest, such as ./cluster/concourse.yml
  • Your versions file, such as -l versions.yml
  • Your environment variables, such as -l variables.yml
  • Any required operations files for your environment or Concourse setup
  • A --vars-store flag with the name of a file in which BOSH can store your cluster credentials, such as --vars-store cluster-creds.yml

About the cluster-creds.yml file

BOSH creates this file for you if it does not already exist. In this case, you just need to specify a filename. In the case that you already have a cluster-creds.yml file, specify it here so that BOSH does not re-create it.

If you specify a file that does not exist, and BOSH re-creates this file for you, it also re-creates things such as passwords and secrets that exist within the file.

  1. When you are ready, navigate to your terminal and run the following bosh deploy command:

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
    bosh deploy \
-e BOSH-ENVIRONMENT-ALIAS \
-d CONCOURSE-DEPLOYMENT-NAME ./cluster/concourse.yml \
-l versions.yml \
-l variables.yml \
-o ./cluster/operations/backup-atc.yml \
-o ./cluster/operations/basic-auth.yml \
-o ./cluster/operations/privileged-http.yml \
-o ./cluster/operations/static-web.yml \
--vars-store cluster-creds.yml

Different Concourse deployments require different environment variables and operations files. If you encounter an error, check the error message for clues about additional variables that need to be set. For more information, see the Concourse documentation.

Finalize Postgres Upgrade

Earlier, we upgraded the Postgres release from 28 to 30. Now that we've completed that, we can upgrade Postgres the rest of the way to the recommended release 36.

  1. Open versions.yml in your favorite editor. You should see these lines from earlier: 

    1
2
3
4
    # postgres_version: '36'
# postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'
postgres_version: '30'
postgres_sha1: 'a798999d29b9f5aa12035cff907b26674b491200'

  2. Remove the lines relating to Postgress verion 30, so that you're left with the commented lines for Postgres version 36:

    1
2
    # postgres_version: '36'
# postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'

  3. Uncomment the lines relating to Postgres version 36:

    1
2
    postgres_version: '36'   
postgres_sha1: '3dd10b417b21cfa3257f1cc891e9e46f02fefe16'

  4. Save and close the versions.yml file.

  5. Redeploy your Concourse deployment. This will deploy the specified Postgres release. Remember to always include:

    • Your new concourse manifest (./cluster/concourse.yml),
    • Your versions file (versions.yml),
    • Your environment variables (variables.yml),
    • Any required operations files, and
    • Your cluster credentials (cluster-creds.yml)

    Your deployment command may require different operations files, but should look something like this:

    1
2
3
4
5
6
7
8
    bosh -e <your bosh environment> deploy -d concourse ./cluster/concourse.yml \
-l versions.yml \
-l variables.yml \
-o ./cluster/operations/basic-auth.yml \
-o ./cluster/operations/bbr.yml \
-o ./cluster/operations/privileged-http.yml \
-o ./cluster/operations/static-web.yml \
--vars-store cluster-creds.yml

Upon successfully running your deploy script, Postgres will be updated to release 36, and your Concourse upgrade is complete.