Integrating Credhub and UAA with Concourse BOSH Release

This guide details the steps involved in installing Concourse for VMware Tanzu as a BOSH release, and then integrating Credhub for secrets management with UAA for user account and authentication management.

Install Concourse First

The process of integrating these tools begins with installing Concourse with BOSH. If you don't have an existing BOSH-deployed Concourse to work with, head to the installation guides before continuing: Install Concourse with BOSH

Set Up Certificates, Log In, and Alias Your BOSH Environment

Note

If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.

Get a CA certificate for your BOSH Director.
1. If you created your BOSH Director manually, retrieve the credentials that were created during setup.
  
  If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:
  1
  https://OPS-MANAGER-DOMAIN/download_root_ca_cert
  Where OPS-MANAGER-DOMAIN is the Ops Manager domain.
  
  This will create a file in your downloads folder called root_ca_certificate.
2. Copy and rename the root_ca_certificate file into the working directory. For example, on OSX:
  1
  mv ~/Downloads/root_ca_certificate ./ca-cert.crt
  Tip
  This certificate file can be used whenever a CLI command asks for a --ca-cert flag and value.
Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as ca-cert.crt, run the following command:
1
bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.crt
Where BOSH-ENVIRONMENT-IP is your BOSH environment IP address.
Finding director credentials

If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.

If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:
1
http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials
Where OPS-MANAGER-DOMAIN is your Ops Manager domain.
Give your environment an alias by running the following command:
1
bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.crt
Where:
- BOSH-ENVIRONMENT-IP is your BOSH environment IP address
- ALIAS is the alias you're creating BOSH environment
Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.

Tip

You can use this alias whenever you target this environment by using the -e flag in a BOSH command.

Prerequisites

The following elements must be in place before proceeding:

BOSH deployed Concourse
1. If you need a Concourse, follow our guide for Installing Concouse with BOSH.
Credhub CLI
1. Download the latest Credhub CLI from the Credhub CLI Releases page
2. Install the binary to your PATH. For example, on linux or OSX:
  1 2 3
  tar xzf credhub*.tgz mv ./credhub /usr/local/bin/credhub chmod +x /usr/local/bin/credhub
UAA CLI
1. Download and install the UAA CLI by following the instructions in the CF UAA CLI GitHub repo

Deploy Credhub & UAA

Introduction

CredHub is a component designed for centralized credential management in CF. It is a single component that can address several scenarios in the CF ecosystem. At the highest level, CredHub centralizes and secures credential generation, storage, lifecycle management, and access. Concourse with its design of credential managers is able to work with multiple credential products including credhub. credhub can be deployed either independant or colocated. This doc focuses on how to deploy colocated credhub.

Install Credhub & UAA With BOSH

Select a local static IP address for Credhub & UAA (single VM).

Credhub and UAA requires a local static IP address to be assigned for your VM where Credhub and UAA will be deployed. Look at the networks section of your BOSH cloud config to determine the range of static IPs you can pick from. For example, if the networks section looks like this, select a static IP address within the range 10.0.0.5-10.0.0.20.

networks:
- name: default
  type: manual
  subnets:
  - range: 10.0.0.0/24
    gateway: 10.0.0.1
    static: [10.0.0.5-10.0.0.20]
    azs: [z1,z2,z3]
    dns: [8.8.8.8]
    cloud_properties:
      name: net-10-0-0-0

Make a note of your intended IP address so that it can be used in a later step.

Tip

You can verify that the IP address you've specified is not already in use by running bosh -e ENVIRONMENT-ALIAS instances. Make sure that your specified IP address isn't already assigned to another instance.

(Optional) Assign a domain for Credhub & UAA.

In some cases, it might be beneficial to assign a domain for Credhub and UAA - this makes it more convenient to change IP addresses without needing to redeploy Concourse.

If this convenience is desired, create a domain in your domain provider and point it to the static IP you selected in the previous step. Otherwise, feel free to skip this step and move on.

Domain Configuration

For more information on domain and IP configuration in BOSH, explore the networking section in this guide.
Download and Upload Credhub, UAA, postgres, and BPM bosh releases:
Warning

You may have some of these uploaded already. You can check by running bosh releases.

Upload each BOSH release that you just downloaded:

bosh -e ALIAS upload-release credhub-release-2.5.7.tgz
bosh -e ALIAS upload-release uaa-release.74.9.0.tgz
bosh -e ALIAS upload-release postgres-release-39.tgz
bosh -e ALIAS upload-release bpm-release-1.1.5.tgz

Download the latest 623.x stemcell from VMware Tanzu Network.
Run the command below to upload the stemcell to your BOSH environment:
1
bosh -e ALIAS upload-stemcell PATH_TO_STEMCELL
Populate a variables file to store variables to be used by the Credhub deployment. For example, with vim:
1
vim credhub-vars.yml

Copy and paste the following YAML into your editor:

---
deployment-network: NETWORK-NAME
external-ip-address: "EXTERNAL-IP-ADDRESS"
internal-ip-address: "STATIC-IP-ADDRESS"
db_host: localhost
db_port: 5432
uaa_external_url: "https://EXTERNAL-IP-ADDRESS:8443"
uaa_internal_url: "https://STATIC-IP-ADDRESS:8443"
uaa_version: "74.9.0"
uaa_sha1: "9647fff0fcb249e71ba2290849b4cdbbf7550165"
credhub_version: "2.5.7"
credhub_sha1: "9647fff0fcb249e71ba2290849b4cdbbf7550165"
postgres_version: "39"
postgres_sha1: "8ff395540e77a461322a01c41aa68973c10f1ffb"
bpm_version: "1.1.5"
bpm_sha1: "e612e88543012ae5d376dd3746159d5abe748076"

Where:

NETWORK-NAME: This is a network name taken from your BOSH cloud config. You can find the name of available networks by running bosh cloud-config and reading the output. You could pick one of the networks, or create a new network through bosh -e ALIAS update-cloud-config YOUR-OWN-CLOUD-CONFIG. You can read more about BOSH networking at https://bosh.io/docs/networks/

STATIC-IP-ADDRESS: Based on the network you chose for NETWORK-NAME. Pick an IP address that isn't being used by any other deployments and falls within the static range of IPs, highlighted below in the following sample/example cloud-config:

networks:
- name: default
  subnets:
  - azs:
    - z1
    cloud_properties:
      ...
    gateway: 10.0.0.1
    range: 10.0.0.0/16
    reserved:
    - 10.0.0.1-10.0.0.255
    static:
    - 10.0.255.0-10.0.255.254
  type: manual
- name: private
  subnets:
  - azs:
    - z1
    cloud_properties:
      ...
    gateway: 10.0.0.1
    range: 10.0.0.0/16
    reserved:
    - 10.0.0.1-10.0.0.255
    static:
    - 10.0.255.0-10.0.255.254
  type: manual

EXTERNAL-IP-ADDRESS: Can be the same as STATIC-IP-ADDRESS if you don't plan to assign an external IP to this deployment.

Create a new manifest. For example, with vim:

1	`vim credhub-uaa-manifest.yml`

Copy and paste the contents of the following code block in to your new manifest:

---
name: credhub-uaa
instance_groups:
- name: credhub-uaa
  azs:
  - z1
  instances: 1
  vm_type: default
  persistent_disk_type: default
  stemcell: xenial
  networks:
  - name: ((deployment-network))
    static_ips:
    - ((internal-ip-address))
  jobs:
  - name: uaa
    release: uaa
    properties:
      encryption:
        active_key_label: 'key-1'
        encryption_keys:
          - label: 'key-1'
            passphrase: "((uaa_encryption_key))"
      login:
        saml:
          serviceProviderCertificate: ((uaa_login_saml.certificate))
          serviceProviderKey: ((uaa_login_saml.private_key))
      uaa:
        clients:
          admin:
            authorized-grant-types: client_credentials
            scope: uaa.none
            authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin
            secret: "((uaa_admin_client_secret))"
          credhub_cli:
            authorized-grant-types: password,refresh_token
            authorities: uaa.none
            scope: credhub.read,credhub.write
            secret: "" # credhub expects this to be empty
            access-token-validity: 120
            refresh-token-validity: 1800
            override: true
          concourse_client:
            authorized-grant-types: client_credentials
            authorities: credhub.read,credhub.write
            scope: credhub.read,credhub.write
            secret: "((concourse_credhub_client_secret))"
            access-token-validity: 120
            refresh-token-validity: 1800
            override: true
        jwt:
          policy:
            active_key_id: key-1
            keys:
              key-1:
                signingKey: ((uaa_jwt_signing_key.private_key))
        scim:
          users:
            - name: admin
              password: "((cf_admin_password))"
              groups:
                - uaa.admin
            - name: credhub
              password: "((credhub_user_password))"
              groups:
                - credhub.read
                - credhub.write
        sslCertificate: ((uaa_ssl.certificate))
        sslPrivateKey: ((uaa_ssl.private_key))
        url: "((uaa_external_url))"
      uaadb:
        address: ((db_host))
        tls: disabled
        databases:
        - name: uaadb
          tag: uaa
        db_scheme: postgres
        port: ((db_port))
        roles:
        - name: uaaadmin
          password: "((uaa_db_user_password))"
          tag: admin
  - name: credhub
    release: credhub
    properties:
      credhub:
        port: 8844
        tls:
          certificate: ((credhub_ssl.certificate))
          private_key: ((credhub_ssl.private_key))
        data_storage:
          type: postgres
          username: credhubadmin
          password: ((credhub_db_user_password))
          host: ((db_host))
          port: ((db_port))
          database: credhub
          require_tls: false
        authentication:
          uaa:
            ca_certs:
              - ((credhub-ca.certificate))
            enabled: true
            url: "((uaa_external_url))"
            wait_for_start: true
        encryption:
          keys:
            - provider_name: internal-provider
              key_properties:
                encryption_password: "((credhub_encryption_key))"
              active: true
          providers:
            - name: internal-provider
              type: internal
        authorization:
          acls:
            enabled: true
          permissions:
            - path: /*
              actors:
                - uaa-client:concourse_client
                - uaa-client:credhub_cli
              operations:
                - read
                - write
                - delete
                - read_acl
                - write_acl
  - name: bpm
    release: bpm
  - name: postgres
    release: postgres
    properties:
      databases:
        address: 127.0.0.1
        port: 5432
        databases:
        - name: credhub
        - name: uaadb
        roles:
        - name: admin
          password: ((database-admin))
        - name: uaaadmin
          password: ((uaa_db_user_password))
        - name: credhubadmin
          password: ((credhub_db_user_password))
        tls:
          ca: ((database-tls.ca))
          certificate: ((database-tls.certificate))
          private_key: ((database-tls.private_key))

releases:
- name: uaa
  version: "((uaa_version))"

- name: credhub
  version: "((credhub_version))"

- name: postgres
  version: "((postgres_version))"

- name: bpm
  version: "((bpm_version))"

variables:
- name: uaa_encryption_key
  type: password

- name: uaa_admin_client_secret
  type: password

- name: cf_admin_password
  type: password

- name: uaa_jwt_signing_key
  type: rsa

- name: concourse_credhub_client_secret
  type: password

- name: credhub_client_secret
  type: password

- name: credhub_user_password
  type: password

- name: credhub_encryption_key
  type: password

- name: credhub_db_user_password
  type: password

- name: uaa_db_user_password
  type: password

- name: database-admin
  type: password

- name: credhub-ca
  type: certificate
  options:
    is_ca: true
    common_name: CredHub CA

- name: database-tls
  type: certificate
  options:
    ca: credhub-ca
    common_name: ((external-ip-address))
    alternative_names:
    - ((external-ip-address))
    - ((internal-ip-address))
    - 127.0.0.1
    - localhost

- name: uaa_ssl
  type: certificate
  options:
    ca: credhub-ca
    common_name: ((external-ip-address))
    alternative_names:
    - ((external-ip-address))
    - ((internal-ip-address))
    - 127.0.0.1
    - localhost

- name: credhub_ssl
  type: certificate
  options:
    ca: credhub-ca
    common_name: ((external-ip-address))
    alternative_names:
    - ((external-ip-address))
    - ((internal-ip-address))
    - 127.0.0.1
    - localhost

- name: uaa_login_saml
  type: certificate
  options:
    ca: credhub-ca
    common_name: ((external-ip-address))
    alternative_names:
    - ((external-ip-address))
    - ((internal-ip-address))
    - 127.0.0.1
    - localhost

stemcells:
- alias: xenial
  os: ubuntu-xenial
  version: latest

update:
  canaries: 1
  canary_watch_time: 10000-200000
  max_in_flight: 3
  serial: false
  update_watch_time: 10000-200000

Deploy the new manifest:

1
2
3

bosh -e ALIAS deploy -d credhub-uaa credhub-uaa-manifest.yml \
  --vars-file credhub-vars.yml \
  --vars-store credhub-vars-store.yml

Integrate Concourse with Credhub

In this section, you'll find steps for modifying your existing Concourse deployment to integrate with Credhub. This step assumes you use concourse-bosh-deployment to deploy your Concourse. See the Prerequisites section for more information.

Cluster Credentials

When you deploy Concourse for the first time, BOSH creates a file to store your cluster credentials. If you've followed the Concourse for VMware Tanzu Installation Guide, this should be named cluster-creds.yml. When you deploy in the following steps, use this exact same file to ensure BOSH doesn't recreate these values.

To achieve the intended configuration, Concourse must be deployed with an ops-file called credhub.yml in the cluster/operations/ directory located in the concourse-bosh-deployment repo. This ops file contains variables that must be populated before we deploy.

Update the cluster-creds.yml vars-store with the appropriate variables. For example:

...
credhub_ca_cert: |
  -----BEGIN CERTIFICATE-----
  ...
  -----END CERTIFICATE-----
credhub_url: "https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844"
credhub_client_id: "concourse_client"
credhub_client_secret: "CLIENT-SECRET"

Where:

credhub_ca_cert is the credhub-ca/ca value in the vars-store file credhub-vars-store.yml:

credhub-ca:
ca: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
certificate: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----

EXTERNAL-IP-OR-URL-OF-CREDHUB: The value of external-ip-address in the vars-file credhub-vars.yml from the Credhub deployment.
CLIENT-SECRET: The value of concourse_credhub_client_secret in the vars-store file credhub-vars-store.yml from the Credhub deployment.

Redeploy Concourse.

The following is a sample BOSH deploy command. You may have to adjust it to match your deployment:

bosh deploy \
  -e ENVIRONMENT-ALIAS \
  -d DEPLOYMENT-NAME ./cluster/concourse.yml \
  -l versions.yml \
  -l variables.yml \
  -o ./cluster/operations/backup-atc.yml \
  -o ./cluster/operations/basic-auth.yml \
  -o ./cluster/operations/privileged-http.yml \
  -o ./cluster/operations/static-web.yml \
  -o ./cluster/operations/credhub.yml \
  --vars-store cluster-creds.yml

Where:

ENVIRONMENT-ALIAS is the BOSH environment alias
DEPLOYMENT-NAME is the name of your choice for your Concourse deployment

BOSH will show you the delta in the manifest you're applying. Your delta should look like this, although your version numbers may vary:

Using deployment 'concourse-credhub'

Release 'concourse/5.5.8' already exists.

Release 'bpm/1.1.5' already exists.

Release 'postgres/39' already exists.

  instance_groups:
  - name: web
    jobs:
    - name: web
      properties:
+       credhub:
+         client_id: "<redacted>"
+         client_secret: "<redacted>"
+         tls:
+           ca_cert:
+             certificate: "<redacted>"
+         url: "<redacted>"

Verify Integration

Now that Credhub and UAA have been deployed, you can verify that everything works by setting a secret using the Credhub CLI, and then fetching the secret from a Concourse pipeline.

Extract Credhub CA Certificate into a certificate file called credhub-ca.crt. You can get the value of the CA certificate from the vars-store file credhub-vars-store.yml.

Copy the value of credhub-ca/ca into credhub-ca.crt.
1 2 3 4 5
credhub-ca: ca: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
The contents of credhub-ca.crt should look like this:
1 2 3
-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

Using the Credhub CLI, log into the Credhub server.

$ credhub api \
--server https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844 \
--ca-cert=credhub-ca.crt

Setting the target url: https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844

$ credhub login \
--client-name concourse_client \
--client-secret "CLIENT-SECRET"

Login Successful

Where:

CLIENT-SECRET is the value of concourse_credhub_client_secret in the Credhub deployment's vars-store file credhub-vars-store.yml

Create a secret in Credhub that you'll use a pipeline in Concourse to retrieve later. The next step assumes you login to Concourse with the main team:

$ credhub set \
-n /concourse/main/test \
--type value \
--value "a secret"

id: 56caaffc-64f2-4333-a4a5-78f8e78efa3b
name: /concourse/main/test
type: value
value: <redacted>
version_created_at: "2019-11-20T21:28:14Z"

Create a file called test-pipeline.yml. For example, with vim:
1
vim test-pipeline.yml

Copy and paste the contents of the following code block into test-pipeline.yml:

---
jobs:
  - name: job
    public: true
    plan:
      - task: simple-task
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: { repository: alpine }
          run:
            path: echo
            args: ["This is ((test))"]

Set and run the pipeline:

$ fly -t <target> login -c http://<web_ip>:8080 -u <username> -p <password> -n main
logging in to team 'main'
target saved

$ fly -t <target> sp -p test -c test-pipeline.yml
jobs:
  job job has been added:
+ name: job
+ plan:
+ - config:
+     container_limits: {}
+     image_resource:
+       source:
+         repository: alpine
+       type: registry-image
+     platform: linux
+     run:
+       args:
+       - This is ((test))
+       path: echo
+   task: simple-task
+ public: true

$ fly -t <target> up -p test
unpaused 'test'

$ fly -t <target> tj -j test/job
started test/job #1

initializing
running echo This is a secret
This is a secret
succeeded

Success

The output of the job should contain this line somewhere:

1	`This is a secret`

Tip

To learn more about how Concourse integrates with Credhub, check out the OSS Credhub documentation: https://concourse-ci.org/credhub-credential-manager.html

Setting up Credhub for multiple teams (ACL)

To enable Concourse users to set their secrets in Credhub you need to create accounts for them in UAA, and then set their path permissions in Credhub, referencing their newly created UAA accounts.

To create a UAA account that a Credhub user can use, run these commands:

1
2
3

uaac create-user USER-NAME --email EMAIL-ADDRESS --password PASSWORD --familyName FAMILY-NAME --givenName GIVEN-NAME
uaac add-member credhub.read USER-NAME
uaac add-member credhub.write USER-NAME

Next, get the UAA ID of the user you just created.
1
uaac get-user USER-NAME

Then grant access to the path in Credhub for the user you just created:

1	`credhub set-permission --actor uaa-user:USER-UAA-ID --operations read,write,delete --path "/concourse/TEAM-NAME/*"`