Skip to content

Install Concourse

Pivotal Concourse is installed as a BOSH release. If you haven't read the Prerequisites and Background Information page, please do so before continuing.


Set Up Certificates, Log In, and Alias Your BOSH Environment

Note

If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.

  1. Get a CA certificate for your BOSH Director.

    1. If you created your BOSH Director manually, retrieve the credentials that were created during setup.

      If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:

      1
      https://OPS-MANAGER-DOMAIN/api/v0/security/root_ca_certificate
      

      Where OPS-MANAGER-DOMAIN is the Ops Manager domain.

      The JSON response you receive has a key-value pair. The CA certificate is the returned value, between the quotation marks, that starts with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----.

    2. Copy and paste the CA certificate into a file, with all instances of \n replaced by carriage returns.

      Pasting the CA Certificate

      In most shells in OSX, you can use the pbpaste command to access the contents of your clipboard. After copying the CA certificate to your clipboard, you can then run this command to format it correctly when pasting:

      1
       echo -e "$(pbpaste)" > ca-cert.yml
      
      Alternatives for Pasting the CA Certificate

      Two alternatives for pasting the CA certificate include:

      1. Using the command-line JSON processor jq to translate the JSON when pasting
      2. Pasting the value into a file and replacing the newline characters with carriage returns using the find and replace operation in a text editor
    3. Ensure the file is formed correctly, with -----BEGIN CERTIFICATE----- at the beginning and -----END CERTIFICATE----- at the end, by running:

      1
      cat ca-cert.yml
      

      Tip

      This certificate file can be used whenever a CLI command asks for a --ca-cert flag and value.

  2. Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as ca-cert.yml, run the following command:

    1
    bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.yml
    

    Where BOSH-ENVIRONMENT-IP is your BOSH environment IP address.

    Finding director credentials

    If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.

    If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:

    1
    http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials
    

    Where OPS-MANAGER-DOMAIN is your Ops Manager domain.

  3. Give your environment an alias by running the following command:

    1
    bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.yml
    

    Where:

    • BOSH-ENVIRONMENT-IP is your BOSH environment IP address
    • ALIAS is the alias you're creating BOSH environment

    Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.

    Tip

    You can use this alias whenever you target this environment by using the -e flag in a BOSH command.


Setup concourse-bosh-deployment directory on your local machine

The concourse-bosh-deployment repository has a sample BOSH manifest, versions.yml file, and a selection of deployment-modifying operations files. Using these sample files makes it much faster and easier to get started.

  1. Clone the concourse-bosh-deployment repo by running the following snippet on the command line:

    1
    git clone https://github.com/concourse/concourse-bosh-deployment.git
    
  2. Move to the concourse-bosh-deployment directory:

    1
    cd concourse-bosh-deployment
    

    All the paths used in this tutorial are relative to this directory.

  3. Checkout the release that corresponds to the version of Concourse you want to install. For example, if you're installing Concourse 4.2.4:

    1
    git checkout v4.2.4
    

    For a list of all Concourse releases, see concourse-bosh-deployment in GitHub.

    Success

    Checking out a release rather than a branch means that git produces the following output:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    Note: checking out 'RELEASE'.
    
    You are in 'detached HEAD' state. You can look around, make experimental
    changes and commit them, and you can discard any commits you make in this
    state without impacting any branches by performing another checkout.
    
    If you want to create a new branch to retain commits you create, you may
    do so (now or later) by using -b with the checkout command again. Example:
    
        git checkout -b <new-branch-name>
    
    HEAD is now at HASH... COMMIT-MESSAGE
    

Download Concourse Release

  1. If you haven't already, download the Concourse 4.2.4 release from Pivotal Network.

    You should end up with a file called concourse-4.2.4.tgz in your Downloads directory.

  2. Use the bosh upload-release command to upload the Concourse tarball that you downloaded from Pivotal Network.

    For example, with Concourse 4.2.4:

    1
    bosh -e BOSH-ENVIRONMENT upload-release ~/Downloads/concourse-4.2.4.tgz
    

    Click here for more information about uploading releases.


BOSH and Concourse variables

Installing Concourse requires setting certain configuration variables that depend on infrastructure. In this section we'll use the bosh cloud-config command to look up our available options, and create a file to store the aforementioned configuration variables. We'll include this file in our final deployment.

  1. Run the following command to fetch the cloud configuration details of your IaaS and put them in a cloud-config.yml file to reference later:

    1
    2
    3
    4
    bosh \
    -e BOSH-ENVIRONMENT-ALIAS \
    -d NAME-OF-CONCOURSE-DEPLOYMENT \
    cloud-config > cloud-config.yml
    

    Tip

    The following steps describe setting variables that are necessary for deployment in a yaml file. If you'd prefer to set them on the command line when you run bosh deploy, you could pass them in at that time with the --var <key>=<value> flag and syntax. Click here to learn more about bosh deploy.

  2. Create a file called variables.yml file to store your Concourse- and BOSH-related environment variables. You can do this using vim or your favorite editor:

    1
    vim variables.yml
    
  3. Fill in the following variables:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    ---
    local_user:
      username: USERNAME
      password: PASSWORD
    
    deployment_name: DEPLOYMENT-NAME
    db_persistent_disk_type: PERSISTENT-DISK-TYPE
    db_vm_type: VM-TYPE
    external_url: EXTERNAL-URL
    network_name: NETWORK-NAME
    postgres_password: POSTGRES-PASSWORD
    web_ip: WEB-IP
    web_vm_type: VM-TYPE
    worker_vm_type: VM-TYPE
    

    Where:

    • DEPLOYMENT-NAME is the name of your choice for your Concourse deployment
    • EXTERNAL-URL is a url associated with the web_ip property (eg, http://WEB-IP:8080)
    • local_user.username your choice of username, used to log in to Concourse
    • local_user.password is your choice of password, used to log in to Concourse
    • NETWORK-NAME is the name of the networks property in cloud-config.yml
    • PERSISTENT-DISK-TYPE is the name of one of the disk types in cloud-config.yml
    • POSTGRES-PASSWORD is the password of your choice for the PostgreSQL db used by Concourse
    • VM-TYPE is the name of one of the VM types in your cloud-config.yml file
    • WEB-IP is a non-reserved IP address for your deployment (eg, 1.1.1.1).

    Reserved IP Ranges

    In some cases, you might need to modify the reserved IP ranges of your network before choosing a web_ip for Concourse on OpsManager. For example, the reserved IP range is: 10.0.5.1-10.0.05.10, we set web_ip as 10.0.5.8, the new reserved IP range would be: 10.0.5.1-10.0.5.7, 10.0.5.9-10.0.5.10

    External URL

    Concourse uses an external URL to facilitate communication between workers and ATCs.

    When deploying it for the first time you might yet not know the final external URL - for example, you may later want to add a load balancer, or change your configuration as you go.

    For now, choose any available URL to get started - you can always change this value later if you run into trouble or change your mind, and redeploy to update it.

  4. Save and close the variables.yml file.


Set Up a Deployment

To configure Pivotal Concourse, do the following:

  1. Prepare the Concourse deployment manifest.

    You can write your own manifest or modify a template with your unique configuration. The concourse-bosh-deployment repository that we cloned earlier on has an example manifest template in the concourse.yml file located at ./cluster/concourse.yml.

    For more information about Cluster Concourse deployment, see the README.md files at concourse-bosh-deployment in GitHub.

  2. Configure authentication in your deployment manifest by providing tls_cert and tls_key values.

    For instructions, see Authentication.

  3. Prepare a cloud-config.yml file and upload it to your BOSH Director with bosh update-cloud-config.


Deploy Concourse with bosh deploy

To deploy your new Concourse, use the following information:

  • Your edited concourse manifest (eg, ./cluster/concourse.yml),
  • The versions file that comes with the concourse-bosh-deployment (eg, -l versions.yml),
  • Your environment variables (eg, -l variables.yml),
  • Any required operations files for your environment or Concourse setup (the snippet below has a basic set of simple operations files, but you can add whatever options your unique deployment might require) and
  • A --vars-store flag and a filename where BOSH can store your cluster credentials (eg, --vars-store cluster-creds.yml)

About the cluster-creds.yml file

BOSH creates this file for you if it does not already exist. In this case, you just need to specify a filename. In the case that you already have an existing cluster-creds.yml file, specify it here so that BOSH does not re-create it.

Keep in mind that if you specify a filename that does not exist, and BOSH re-creates this file for you, it also re-creates things such as passwords or secrets that go inside the file.

When you are ready, move to your terminal and run the following bosh deploy commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
bosh deploy \
-e <bosh environment alias> \
-d <concourse deployment name> ./cluster/concourse.yml \
-l versions.yml \
-l variables.yml \
-o ./cluster/operations/backup-atc.yml \
-o ./cluster/operations/basic-auth.yml \
-o ./cluster/operations/privileged-http.yml \
-o ./cluster/operations/static-web.yml \
--vars-store cluster-creds.yml

Make sure all your file paths are correct. For more information, see Deploying in the BOSH documentation.

Different Concourse deployments require different environment variables and operations files. If you get an error, check the error message for clues about additional variables that need to be set. Check out the open-source Concourse documentation for additional information.

Upon successfully running your deploy script, a success message is displayed, and Concourse has been installed.

Use the external URL that you specified in your variables.yml file to target your concourse with fly and view the Concourse web UI with command fly -t <choose a target name> login -c http://<web_ip>:8080 -u <username> -p <password>.