Install Concourse with BOSH

Concourse for VMware Tanzu is available for installation as a BOSH release. If you haven't read the Prerequisites and Background Information page, please do so before continuing.

This guide describes a process for installing Pivotal Concourse as a BOSH release for CI/CD usage. If you're interested in setting up Platform Automation, navigate to the Installing Pivotal Concourse for Platform Automation instead.

Set Up Certificates, Log In, and Alias Your BOSH Environment

Note

If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.

Get a CA certificate for your BOSH Director.
1. If you created your BOSH Director manually, retrieve the credentials that were created during setup.
  
  If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:
  1
  https://OPS-MANAGER-DOMAIN/download_root_ca_cert
  Where OPS-MANAGER-DOMAIN is the Ops Manager domain.
  
  This will create a file in your downloads folder called root_ca_certificate.
2. Copy and rename the root_ca_certificate file into the working directory. For example, on OSX:
  1
  mv ~/Downloads/root_ca_certificate ./ca-cert.crt
  Tip
  This certificate file can be used whenever a CLI command asks for a --ca-cert flag and value.
Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as ca-cert.crt, run the following command:
1
bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.crt
Where BOSH-ENVIRONMENT-IP is your BOSH environment IP address.
Finding director credentials

If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.

If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:
1
http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials
Where OPS-MANAGER-DOMAIN is your Ops Manager domain.
Give your environment an alias by running the following command:
1
bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.crt
Where:
- BOSH-ENVIRONMENT-IP is your BOSH environment IP address
- ALIAS is the alias you're creating BOSH environment
Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.

Tip

You can use this alias whenever you target this environment by using the -e flag in a BOSH command.

Setup `concourse-bosh-deployment` directory on your local machine

The concourse-bosh-deployment repository has a sample BOSH manifest, versions.yml file, and a selection of deployment-modifying operations files. Using these sample files makes it much faster and easier to get started.

Clone the concourse-bosh-deployment repo by running the following snippet on the command line:
1
git clone https://github.com/concourse/concourse-bosh-deployment.git
Move to the concourse-bosh-deployment directory:
1
cd concourse-bosh-deployment
All the paths used in this tutorial are relative to this directory.

Checkout the release that corresponds to the version of Concourse you want to install. For example, if you're installing the latest release, Concourse v5.5.8:

1	`git checkout v5.5.8`

For a list of all Concourse releases, see concourse-bosh-deployment in GitHub.

Success

Checking out a release rather than a branch means that git produces the following output:

Note: checking out 'RELEASE'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

    git checkout -b <new-branch-name>

HEAD is now at HASH... COMMIT-MESSAGE

Download & Upload Concourse Release

If you haven't already, download the appropriate Concourse release from VMware Tanzu Network. For example, if you were to use the Concourse v5.5.8 release, you should end up with a file called concourse-bosh-release-5.5.8.tgz in your Downloads directory.
Use the bosh upload-release command to upload the Concourse tarball that you downloaded from VMware Tanzu Network.

For example, with the latest release, Concourse v5.5.8:
1
bosh -e BOSH-ENVIRONMENT upload-release ~/Downloads/concourse-bosh-release-5.5.8.tgz
Click here for more information about uploading releases.

BOSH and Concourse variables

Installing Concourse requires setting certain configuration variables that depend on infrastructure. In this section we'll use the bosh cloud-config command to look up our available options, and create a file to store the aforementioned configuration variables. We'll include this file in our final deployment.

Run the following command to fetch the cloud configuration details of your IaaS and put them in a cloud-config.yml file to reference later:
1 2 3
bosh \ -e BOSH-ENVIRONMENT-ALIAS \ cloud-config > cloud-config.yml
Tip

The following steps describe setting variables that are necessary for deployment in a yaml file. If you'd prefer to set them on the command line when you run bosh deploy, you could pass them in at that time with the --var KEY=VALUE flag and syntax. Click here to learn more about bosh deploy.
Create a file called variables.yml file to store your Concourse- and BOSH-related environment variables. You can do this using vim or your favorite editor:
1
vim variables.yml

Fill in the following variables:

---
local_user:
  username: USERNAME
  password: PASSWORD

deployment_name: DEPLOYMENT-NAME
db_persistent_disk_type: PERSISTENT-DISK-TYPE
db_vm_type: VM-TYPE
external_url: EXTERNAL-URL
network_name: NETWORK-NAME
postgres_password: POSTGRES-PASSWORD
web_ip: WEB-IP
web_vm_type: VM-TYPE
worker_vm_type: VM-TYPE

Where:

DEPLOYMENT-NAME is the name of your choice for your Concourse deployment
EXTERNAL-URL is a url associated with the web_ip property (eg, http://WEB-IP:8080)
local_user.username your choice of username, used to log in to Concourse
local_user.password is your choice of password, used to log in to Concourse
NETWORK-NAME is the name of the networks property in cloud-config.yml
PERSISTENT-DISK-TYPE is the name of one of the disk types in cloud-config.yml
POSTGRES-PASSWORD is the password of your choice for the PostgreSQL db used by Concourse
VM-TYPE is the name of one of the VM types in your cloud-config.yml file
WEB-IP is a non-reserved IP address for your deployment (eg, 1.1.1.1).

Reserved IP Ranges

In some cases, you might need to modify the reserved IP ranges of your network on Ops Manager before choosing a web_ip for Concourse. For example, the reserved IP range is: 10.0.5.1-10.0.05.10, and you wish to set web_ip as 10.0.5.8, the new reserved IP range to set on Ops Manager would be: 10.0.5.1-10.0.5.7, 10.0.5.9-10.0.5.10

External URL

Concourse uses an external URL to facilitate communication between workers and ATCs.

When deploying for the first time you might yet not know the final external URL - for example, you may later want to add a load balancer, or change your configuration as you go.

For now, choose any available URL to get started. You can always change this value later if you run into trouble or change your mind. To do so at any time, update the value in your variable files and redeploy.

Save and close the variables.yml file.

Set Up a Deployment

To configure Concourse for VMware Tanzu, do the following:

Prepare the Concourse deployment manifest.

You can write your own manifest or modify a template with your unique configuration. The concourse-bosh-deployment repository that we cloned earlier on has an example manifest template in the concourse.yml file located at ./cluster/concourse.yml.

For more information about Cluster Concourse deployment, see the README.md files at concourse-bosh-deployment in GitHub.
Configure authentication in your deployment manifest by providing tls_cert and tls_key values.

For instructions, see the guides on Authentication.
Prepare a cloud-config.yml file and upload it to your BOSH Director with bosh update-cloud-config.

Deploy Concourse with `bosh deploy`

To deploy your new Concourse, use the following information:

Your edited concourse manifest (eg, ./cluster/concourse.yml),
The versions file that comes with the concourse-bosh-deployment (eg, -l versions.yml),
Your environment variables (eg, -l variables.yml),
Any required operations files for your environment or Concourse setup.

The snippet below has a basic set of operations files, but you can add whatever options your unique deployment might require.
A --vars-store flag and a filename where BOSH can store your cluster credentials (eg, --vars-store cluster-creds.yml)

About the cluster-creds.yml file

BOSH creates this file for you if it does not already exist. If this is the case, you still need to specify a filename for BOSH to use. In the case that you already have an existing cluster-creds.yml file, specify it here so that BOSH does not re-create it.

Keep in mind that if you specify a filename that does not exist, and BOSH re-creates this file for you, it also re-creates things such as passwords or secrets that go inside the file.

When you are ready, move to your terminal and run the following bosh deploy commands:

bosh deploy \
-e BOSH-ENVIRONMENT-ALIAS \
-d CONCOURSE-DEPLOYMENT-NAME \
./cluster/concourse.yml \
-l versions.yml \
-l variables.yml \
-o ./cluster/operations/backup-atc.yml \
-o ./cluster/operations/basic-auth.yml \
-o ./cluster/operations/privileged-http.yml \
-o ./cluster/operations/static-web.yml \
--vars-store cluster-creds.yml

Make sure all your file paths are correct. For more information, see Deploying in the BOSH documentation.

Different Concourse deployments require different environment variables and operations files. If you get an error, check the error message for clues about additional variables that need to be set. Check out the open-source Concourse documentation for additional information.

Upon successfully running your deploy script, a success message is displayed, and Concourse has been installed.

Use the external URL that you specified in your variables.yml file to target your concourse with fly and view the Concourse web UI with command fly -t NEW-TARGET-NAME login -c http://WEB-IP:8080 -u USERNAME -p PASSWORD.