Integrating Credhub and UAA with Concourse BOSH Release
This guide details the steps involved in installing Concourse for VMware Tanzu as a BOSH release, and then integrating Credhub for secrets management with UAA for user account and authentication management.
Install Concourse First
The process of integrating these tools begins with installing Concourse with BOSH. If you don't have an existing BOSH-deployed Concourse to work with, head to the installation guides before continuing: Install Concourse with BOSH
Set Up Certificates, Log In, and Alias Your BOSH Environment
Note
If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.
-
Get a CA certificate for your BOSH Director.
-
If you created your BOSH Director manually, retrieve the credentials that were created during setup.
If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:
1
https://OPS-MANAGER-DOMAIN/download_root_ca_cert
Where
OPS-MANAGER-DOMAIN
is the Ops Manager domain.This will create a file in your downloads folder called
root_ca_certificate
. -
Copy and rename the
root_ca_certificate
file into the working directory. For example, on OSX:1
mv ~/Downloads/root_ca_certificate ./ca-cert.crt
Tip
This certificate file can be used whenever a CLI command asks for a
--ca-cert
flag and value.
-
-
Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as
ca-cert.crt
, run the following command:1
bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.crt
Where
BOSH-ENVIRONMENT-IP
is your BOSH environment IP address.Finding director credentials
If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.
If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:
1
http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials
Where
OPS-MANAGER-DOMAIN
is your Ops Manager domain. -
Give your environment an alias by running the following command:
1
bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.crt
Where:
BOSH-ENVIRONMENT-IP
is your BOSH environment IP addressALIAS
is the alias you're creating BOSH environment
Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.
Tip
You can use this alias whenever you target this environment by using the
-e
flag in a BOSH command.
Prerequisites
The following elements must be in place before proceeding:
- BOSH deployed Concourse
- If you need a Concourse, follow our guide for Installing Concouse with BOSH.
-
Credhub CLI
- Download the latest Credhub CLI from the Credhub CLI Releases page
-
Install the binary to your
PATH
. For example, on linux or OSX:1 2 3
tar xzf credhub*.tgz mv ./credhub /usr/local/bin/credhub chmod +x /usr/local/bin/credhub
-
UAA CLI
- Download and install the UAA CLI by following the instructions in the CF UAA CLI GitHub repo
Deploy Credhub & UAA
Introduction
CredHub is a component designed for centralized credential management in CF. It is a single component that can address several scenarios in the CF ecosystem. At the highest level, CredHub centralizes and secures credential generation, storage, lifecycle management, and access. Concourse with its design of credential managers is able to work with multiple credential products including credhub. credhub can be deployed either independant or colocated. This doc focuses on how to deploy colocated credhub.
Install Credhub & UAA With BOSH
-
Select a local static IP address for Credhub & UAA (single VM).
Credhub and UAA requires a local static IP address to be assigned for your VM where Credhub and UAA will be deployed. Look at the
networks
section of your BOSH cloud config to determine the range of static IPs you can pick from. For example, if thenetworks
section looks like this, select a static IP address within the range10.0.0.5-10.0.0.20
.1 2 3 4 5 6 7 8 9 10 11
networks: - name: default type: manual subnets: - range: 10.0.0.0/24 gateway: 10.0.0.1 static: [10.0.0.5-10.0.0.20] azs: [z1,z2,z3] dns: [8.8.8.8] cloud_properties: name: net-10-0-0-0
Make a note of your intended IP address so that it can be used in a later step.
Tip
You can verify that the IP address you've specified is not already in use by running
bosh -e ENVIRONMENT-ALIAS instances
. Make sure that your specified IP address isn't already assigned to another instance. -
(Optional) Assign a domain for Credhub & UAA.
In some cases, it might be beneficial to assign a domain for Credhub and UAA - this makes it more convenient to change IP addresses without needing to redeploy Concourse.
If this convenience is desired, create a domain in your domain provider and point it to the static IP you selected in the previous step. Otherwise, feel free to skip this step and move on.
Domain Configuration
For more information on domain and IP configuration in BOSH, explore the networking section in this guide.
-
Download and Upload Credhub, UAA, postgres, and BPM bosh releases:
Warning
You may have some of these uploaded already. You can check by running
bosh releases
. -
Upload each BOSH release that you just downloaded:
1 2 3 4
bosh -e ALIAS upload-release credhub-release-2.5.7.tgz bosh -e ALIAS upload-release uaa-release.74.9.0.tgz bosh -e ALIAS upload-release postgres-release-39.tgz bosh -e ALIAS upload-release bpm-release-1.1.5.tgz
-
Download the latest
623.x
stemcell from VMware Tanzu Network. -
Run the command below to upload the stemcell to your BOSH environment:
1
bosh -e ALIAS upload-stemcell PATH_TO_STEMCELL
-
Populate a variables file to store variables to be used by the Credhub deployment. For example, with vim:
1
vim credhub-vars.yml
-
Copy and paste the following YAML into your editor:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
--- deployment-network: NETWORK-NAME external-ip-address: "EXTERNAL-IP-ADDRESS" internal-ip-address: "STATIC-IP-ADDRESS" db_host: localhost db_port: 5432 uaa_external_url: "https://EXTERNAL-IP-ADDRESS:8443" uaa_internal_url: "https://STATIC-IP-ADDRESS:8443" uaa_version: "74.9.0" uaa_sha1: "9647fff0fcb249e71ba2290849b4cdbbf7550165" credhub_version: "2.5.7" credhub_sha1: "9647fff0fcb249e71ba2290849b4cdbbf7550165" postgres_version: "39" postgres_sha1: "8ff395540e77a461322a01c41aa68973c10f1ffb" bpm_version: "1.1.5" bpm_sha1: "e612e88543012ae5d376dd3746159d5abe748076"
Where:
NETWORK-NAME
: This is a network name taken from your BOSH cloud config. You can find the name of available networks by runningbosh cloud-config
and reading the output. You could pick one of the networks, or create a new network throughbosh -e ALIAS update-cloud-config YOUR-OWN-CLOUD-CONFIG
. You can read more about BOSH networking at https://bosh.io/docs/networks/-
STATIC-IP-ADDRESS
: Based on the network you chose forNETWORK-NAME
. Pick an IP address that isn't being used by any other deployments and falls within thestatic
range of IPs, highlighted below in the following sample/example cloud-config:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
networks: - name: default subnets: - azs: - z1 cloud_properties: ... gateway: 10.0.0.1 range: 10.0.0.0/16 reserved: - 10.0.0.1-10.0.0.255 static: - 10.0.255.0-10.0.255.254 type: manual - name: private subnets: - azs: - z1 cloud_properties: ... gateway: 10.0.0.1 range: 10.0.0.0/16 reserved: - 10.0.0.1-10.0.0.255 static: - 10.0.255.0-10.0.255.254 type: manual
-
EXTERNAL-IP-ADDRESS
: Can be the same asSTATIC-IP-ADDRESS
if you don't plan to assign an external IP to this deployment.
-
Create a new manifest. For example, with vim:
1
vim credhub-uaa-manifest.yml
Copy and paste the contents of the following code block in to your new manifest:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259
--- name: credhub-uaa instance_groups: - name: credhub-uaa azs: - z1 instances: 1 vm_type: default persistent_disk_type: default stemcell: xenial networks: - name: ((deployment-network)) static_ips: - ((internal-ip-address)) jobs: - name: uaa release: uaa properties: encryption: active_key_label: 'key-1' encryption_keys: - label: 'key-1' passphrase: "((uaa_encryption_key))" login: saml: serviceProviderCertificate: ((uaa_login_saml.certificate)) serviceProviderKey: ((uaa_login_saml.private_key)) uaa: clients: admin: authorized-grant-types: client_credentials scope: uaa.none authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin secret: "((uaa_admin_client_secret))" credhub_cli: authorized-grant-types: password,refresh_token authorities: uaa.none scope: credhub.read,credhub.write secret: "" # credhub expects this to be empty access-token-validity: 120 refresh-token-validity: 1800 override: true concourse_client: authorized-grant-types: client_credentials authorities: credhub.read,credhub.write scope: credhub.read,credhub.write secret: "((concourse_credhub_client_secret))" access-token-validity: 120 refresh-token-validity: 1800 override: true jwt: policy: active_key_id: key-1 keys: key-1: signingKey: ((uaa_jwt_signing_key.private_key)) scim: users: - name: admin password: "((cf_admin_password))" groups: - uaa.admin - name: credhub password: "((credhub_user_password))" groups: - credhub.read - credhub.write sslCertificate: ((uaa_ssl.certificate)) sslPrivateKey: ((uaa_ssl.private_key)) url: "((uaa_external_url))" uaadb: address: ((db_host)) tls: disabled databases: - name: uaadb tag: uaa db_scheme: postgres port: ((db_port)) roles: - name: uaaadmin password: "((uaa_db_user_password))" tag: admin - name: credhub release: credhub properties: credhub: port: 8844 tls: certificate: ((credhub_ssl.certificate)) private_key: ((credhub_ssl.private_key)) data_storage: type: postgres username: credhubadmin password: ((credhub_db_user_password)) host: ((db_host)) port: ((db_port)) database: credhub require_tls: false authentication: uaa: ca_certs: - ((credhub-ca.certificate)) enabled: true url: "((uaa_external_url))" wait_for_start: true encryption: keys: - provider_name: internal-provider key_properties: encryption_password: "((credhub_encryption_key))" active: true providers: - name: internal-provider type: internal authorization: acls: enabled: true permissions: - path: /* actors: - uaa-client:concourse_client - uaa-client:credhub_cli operations: - read - write - delete - read_acl - write_acl - name: bpm release: bpm - name: postgres release: postgres properties: databases: address: 127.0.0.1 port: 5432 databases: - name: credhub - name: uaadb roles: - name: admin password: ((database-admin)) - name: uaaadmin password: ((uaa_db_user_password)) - name: credhubadmin password: ((credhub_db_user_password)) tls: ca: ((database-tls.ca)) certificate: ((database-tls.certificate)) private_key: ((database-tls.private_key)) releases: - name: uaa version: "((uaa_version))" - name: credhub version: "((credhub_version))" - name: postgres version: "((postgres_version))" - name: bpm version: "((bpm_version))" variables: - name: uaa_encryption_key type: password - name: uaa_admin_client_secret type: password - name: cf_admin_password type: password - name: uaa_jwt_signing_key type: rsa - name: concourse_credhub_client_secret type: password - name: credhub_client_secret type: password - name: credhub_user_password type: password - name: credhub_encryption_key type: password - name: credhub_db_user_password type: password - name: uaa_db_user_password type: password - name: database-admin type: password - name: credhub-ca type: certificate options: is_ca: true common_name: CredHub CA - name: database-tls type: certificate options: ca: credhub-ca common_name: ((external-ip-address)) alternative_names: - ((external-ip-address)) - ((internal-ip-address)) - 127.0.0.1 - localhost - name: uaa_ssl type: certificate options: ca: credhub-ca common_name: ((external-ip-address)) alternative_names: - ((external-ip-address)) - ((internal-ip-address)) - 127.0.0.1 - localhost - name: credhub_ssl type: certificate options: ca: credhub-ca common_name: ((external-ip-address)) alternative_names: - ((external-ip-address)) - ((internal-ip-address)) - 127.0.0.1 - localhost - name: uaa_login_saml type: certificate options: ca: credhub-ca common_name: ((external-ip-address)) alternative_names: - ((external-ip-address)) - ((internal-ip-address)) - 127.0.0.1 - localhost stemcells: - alias: xenial os: ubuntu-xenial version: latest update: canaries: 1 canary_watch_time: 10000-200000 max_in_flight: 3 serial: false update_watch_time: 10000-200000
-
Deploy the new manifest:
1 2 3
bosh -e ALIAS deploy -d credhub-uaa credhub-uaa-manifest.yml \ --vars-file credhub-vars.yml \ --vars-store credhub-vars-store.yml
Integrate Concourse with Credhub
In this section, you'll find steps for modifying your existing Concourse deployment to integrate with Credhub. This step assumes you use concourse-bosh-deployment
to deploy your Concourse. See the Prerequisites section for more information.
Cluster Credentials
When you deploy Concourse for the first time, BOSH creates a file to store your cluster credentials. If you've followed the Concourse for VMware Tanzu Installation Guide, this should be named cluster-creds.yml
. When you deploy in the following steps, use this exact same file to ensure BOSH doesn't recreate these values.
To achieve the intended configuration, Concourse must be deployed with an ops-file called credhub.yml
in the cluster/operations/
directory located in the concourse-bosh-deployment
repo. This ops file contains variables that must be populated before we deploy.
-
Update the
cluster-creds.yml
vars-store with the appropriate variables. For example:1 2 3 4 5 6 7 8
... credhub_ca_cert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- credhub_url: "https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844" credhub_client_id: "concourse_client" credhub_client_secret: "CLIENT-SECRET"
Where:
-
credhub_ca_cert
is thecredhub-ca/ca
value in the vars-store filecredhub-vars-store.yml
:1 2 3 4 5 6 7 8 9 10 11 12 13
credhub-ca: ca: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- certificate: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- private_key: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY-----
EXTERNAL-IP-OR-URL-OF-CREDHUB
: The value ofexternal-ip-address
in the vars-filecredhub-vars.yml
from the Credhub deployment.CLIENT-SECRET
: The value ofconcourse_credhub_client_secret
in the vars-store filecredhub-vars-store.yml
from the Credhub deployment.
-
-
Redeploy Concourse.
The following is a sample BOSH deploy command. You may have to adjust it to match your deployment:
1 2 3 4 5 6 7 8 9 10 11
bosh deploy \ -e ENVIRONMENT-ALIAS \ -d DEPLOYMENT-NAME ./cluster/concourse.yml \ -l versions.yml \ -l variables.yml \ -o ./cluster/operations/backup-atc.yml \ -o ./cluster/operations/basic-auth.yml \ -o ./cluster/operations/privileged-http.yml \ -o ./cluster/operations/static-web.yml \ -o ./cluster/operations/credhub.yml \ --vars-store cluster-creds.yml
Where:
ENVIRONMENT-ALIAS
is the BOSH environment aliasDEPLOYMENT-NAME
is the name of your choice for your Concourse deployment
BOSH will show you the delta in the manifest you're applying. Your delta should look like this, although your version numbers may vary:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Using deployment 'concourse-credhub' Release 'concourse/5.5.8' already exists. Release 'bpm/1.1.5' already exists. Release 'postgres/39' already exists. instance_groups: - name: web jobs: - name: web properties: + credhub: + client_id: "<redacted>" + client_secret: "<redacted>" + tls: + ca_cert: + certificate: "<redacted>" + url: "<redacted>"
Verify Integration
Now that Credhub and UAA have been deployed, you can verify that everything works by setting a secret using the Credhub CLI, and then fetching the secret from a Concourse pipeline.
-
Extract Credhub CA Certificate into a certificate file called
credhub-ca.crt
. You can get the value of the CA certificate from the vars-store filecredhub-vars-store.yml
.Copy the value of
credhub-ca/ca
intocredhub-ca.crt
.1 2 3 4 5
credhub-ca: ca: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
The contents of
credhub-ca.crt
should look like this:1 2 3
-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
-
Using the Credhub CLI, log into the Credhub server.
1 2 3 4 5 6 7 8 9 10 11
$ credhub api \ --server https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844 \ --ca-cert=credhub-ca.crt Setting the target url: https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844 $ credhub login \ --client-name concourse_client \ --client-secret "CLIENT-SECRET" Login Successful
Where:
CLIENT-SECRET
is the value ofconcourse_credhub_client_secret
in the Credhub deployment's vars-store filecredhub-vars-store.yml
-
Create a secret in Credhub that you'll use a pipeline in Concourse to retrieve later. The next step assumes you login to Concourse with the
main
team:1 2 3 4 5 6 7 8 9 10
$ credhub set \ -n /concourse/main/test \ --type value \ --value "a secret" id: 56caaffc-64f2-4333-a4a5-78f8e78efa3b name: /concourse/main/test type: value value: <redacted> version_created_at: "2019-11-20T21:28:14Z"
-
Create a file called
test-pipeline.yml
. For example, with vim:1
vim test-pipeline.yml
-
Copy and paste the contents of the following code block into
test-pipeline.yml
:1 2 3 4 5 6 7 8 9 10 11 12 13 14
--- jobs: - name: job public: true plan: - task: simple-task config: platform: linux image_resource: type: registry-image source: { repository: alpine } run: path: echo args: ["This is ((test))"]
-
Set and run the pipeline:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
$ fly -t <target> login -c http://<web_ip>:8080 -u <username> -p <password> -n main logging in to team 'main' target saved $ fly -t <target> sp -p test -c test-pipeline.yml jobs: job job has been added: + name: job + plan: + - config: + container_limits: {} + image_resource: + source: + repository: alpine + type: registry-image + platform: linux + run: + args: + - This is ((test)) + path: echo + task: simple-task + public: true $ fly -t <target> up -p test unpaused 'test' $ fly -t <target> tj -j test/job started test/job #1 initializing running echo This is a secret This is a secret succeeded
Success
The output of the job should contain this line somewhere:
1
This is a secret
Tip
To learn more about how Concourse integrates with Credhub, check out the OSS Credhub documentation: https://concourse-ci.org/credhub-credential-manager.html
Setting up Credhub for multiple teams (ACL)
To enable Concourse users to set their secrets in Credhub you need to create accounts for them in UAA, and then set their path permissions in Credhub, referencing their newly created UAA accounts.
-
To create a UAA account that a Credhub user can use, run these commands:
1 2 3
uaac create-user USER-NAME --email EMAIL-ADDRESS --password PASSWORD --familyName FAMILY-NAME --givenName GIVEN-NAME uaac add-member credhub.read USER-NAME uaac add-member credhub.write USER-NAME
-
Next, get the UAA ID of the user you just created.
1
uaac get-user USER-NAME
-
Then grant access to the path in Credhub for the user you just created:
1
credhub set-permission --actor uaa-user:USER-UAA-ID --operations read,write,delete --path "/concourse/TEAM-NAME/*"