Skip to content

Integrating Credhub and UAA with Concourse BOSH Release

This guide details the steps involved in installing Concourse for VMware Tanzu as a BOSH release, and then integrating Credhub for secrets management with UAA for user account and authentication management.

Install Concourse First

The process of integrating these tools begins with installing Concourse with BOSH. If you don't have an existing BOSH-deployed Concourse to work with, head to the installation guides before continuing: Install Concourse with BOSH


Set Up Certificates, Log In, and Alias Your BOSH Environment

Note

If you already have your CA certificate and have already logged in to and aliased your BOSH environment, you can skip this section.

  1. Get a CA certificate for your BOSH Director.

    1. If you created your BOSH Director manually, retrieve the credentials that were created during setup.

      If you are working with an Ops Manager-deployed BOSH Director, log in to Ops Manager and access the following endpoint in your Ops Manager domain:

      1
      https://OPS-MANAGER-DOMAIN/download_root_ca_cert
      

      Where OPS-MANAGER-DOMAIN is the Ops Manager domain.

      This will create a file in your downloads folder called root_ca_certificate.

    2. Copy and rename the root_ca_certificate file into the working directory. For example, on OSX:

      1
      mv ~/Downloads/root_ca_certificate ./ca-cert.crt
      

      Tip

      This certificate file can be used whenever a CLI command asks for a --ca-cert flag and value.

  2. Ensure you are logged in to your BOSH environment with the appropriate BOSH Director credentials. For example, if you saved your CA certificate as ca-cert.crt, run the following command:

    1
    bosh -e BOSH-ENVIRONMENT-IP login --ca-cert=ca-cert.crt
    

    Where BOSH-ENVIRONMENT-IP is your BOSH environment IP address.

    Finding director credentials

    If you set up a BOSH Director directly, the director credentials were returned as a file after the setup process finished.

    If you have an Ops Manager-deployed BOSH Director, you can find the director credentials in the Ops Manager credentials tab or at the following endpoint in your Ops Manager domain:

    1
    http://OPS-MANAGER-DOMAIN/api/v0/deployed/director/credentials/director_credentials
    

    Where OPS-MANAGER-DOMAIN is your Ops Manager domain.

  3. Give your environment an alias by running the following command:

    1
    bosh -e BOSH-ENVIRONMENT-IP alias-env ALIAS --ca-cert=ca-cert.crt
    

    Where:

    • BOSH-ENVIRONMENT-IP is your BOSH environment IP address
    • ALIAS is the alias you're creating BOSH environment

    Using an alias for your BOSH environment substantially reduces the keystrokes needed for commands in future.

    Tip

    You can use this alias whenever you target this environment by using the -e flag in a BOSH command.


Prerequisites

The following elements must be in place before proceeding:

  • BOSH deployed Concourse
    1. If you need a Concourse, follow our guide for Installing Concouse with BOSH.
  • Credhub CLI

    1. Download the latest Credhub CLI from the Credhub CLI Releases page
    2. Install the binary to your PATH. For example, on linux or OSX:

      1
      2
      3
      tar xzf credhub*.tgz
      mv ./credhub /usr/local/bin/credhub
      chmod +x /usr/local/bin/credhub
      
  • UAA CLI

    1. Download and install the UAA CLI by following the instructions in the CF UAA CLI GitHub repo

Deploy Credhub & UAA

Introduction

CredHub is a component designed for centralized credential management in CF. It is a single component that can address several scenarios in the CF ecosystem. At the highest level, CredHub centralizes and secures credential generation, storage, lifecycle management, and access. Concourse with its design of credential managers is able to work with multiple credential products including credhub. credhub can be deployed either independant or colocated. This doc focuses on how to deploy colocated credhub.

Install Credhub & UAA With BOSH

  1. Select a local static IP address for Credhub & UAA (single VM).

    Credhub and UAA requires a local static IP address to be assigned for your VM where Credhub and UAA will be deployed. Look at the networks section of your BOSH cloud config to determine the range of static IPs you can pick from. For example, if the networks section looks like this, select a static IP address within the range 10.0.0.5-10.0.0.20.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    networks:
    - name: default
      type: manual
      subnets:
      - range: 10.0.0.0/24
        gateway: 10.0.0.1
        static: [10.0.0.5-10.0.0.20]
        azs: [z1,z2,z3]
        dns: [8.8.8.8]
        cloud_properties:
          name: net-10-0-0-0
    

    Make a note of your intended IP address so that it can be used in a later step.

    Tip

    You can verify that the IP address you've specified is not already in use by running bosh -e ENVIRONMENT-ALIAS instances. Make sure that your specified IP address isn't already assigned to another instance.

  2. (Optional) Assign a domain for Credhub & UAA.

    In some cases, it might be beneficial to assign a domain for Credhub and UAA - this makes it more convenient to change IP addresses without needing to redeploy Concourse.

    If this convenience is desired, create a domain in your domain provider and point it to the static IP you selected in the previous step. Otherwise, feel free to skip this step and move on.

    Domain Configuration

    For more information on domain and IP configuration in BOSH, explore the networking section in this guide.

  3. Download and Upload Credhub, UAA, postgres, and BPM bosh releases:

    Warning

    You may have some of these uploaded already. You can check by running bosh releases.

  4. Upload each BOSH release that you just downloaded:

    1
    2
    3
    4
    bosh -e ALIAS upload-release credhub-release-2.5.7.tgz
    bosh -e ALIAS upload-release uaa-release.74.9.0.tgz
    bosh -e ALIAS upload-release postgres-release-39.tgz
    bosh -e ALIAS upload-release bpm-release-1.1.5.tgz
    
  5. Download the latest 623.x stemcell from VMware Tanzu Network.

  6. Run the command below to upload the stemcell to your BOSH environment:

    1
    bosh -e ALIAS upload-stemcell PATH_TO_STEMCELL
    
  7. Populate a variables file to store variables to be used by the Credhub deployment. For example, with vim:

    1
    vim credhub-vars.yml
    
  8. Copy and paste the following YAML into your editor:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    ---
    deployment-network: NETWORK-NAME
    external-ip-address: "EXTERNAL-IP-ADDRESS"
    internal-ip-address: "STATIC-IP-ADDRESS"
    db_host: localhost
    db_port: 5432
    uaa_external_url: "https://EXTERNAL-IP-ADDRESS:8443"
    uaa_internal_url: "https://STATIC-IP-ADDRESS:8443"
    uaa_version: "74.9.0"
    uaa_sha1: "9647fff0fcb249e71ba2290849b4cdbbf7550165"
    credhub_version: "2.5.7"
    credhub_sha1: "9647fff0fcb249e71ba2290849b4cdbbf7550165"
    postgres_version: "39"
    postgres_sha1: "8ff395540e77a461322a01c41aa68973c10f1ffb"
    bpm_version: "1.1.5"
    bpm_sha1: "e612e88543012ae5d376dd3746159d5abe748076"
    

    Where:

    • NETWORK-NAME: This is a network name taken from your BOSH cloud config. You can find the name of available networks by running bosh cloud-config and reading the output. You could pick one of the networks, or create a new network through bosh -e ALIAS update-cloud-config YOUR-OWN-CLOUD-CONFIG. You can read more about BOSH networking at https://bosh.io/docs/networks/
    • STATIC-IP-ADDRESS: Based on the network you chose for NETWORK-NAME. Pick an IP address that isn't being used by any other deployments and falls within the static range of IPs, highlighted below in the following sample/example cloud-config:

       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      networks:
      - name: default
        subnets:
        - azs:
          - z1
          cloud_properties:
            ...
          gateway: 10.0.0.1
          range: 10.0.0.0/16
          reserved:
          - 10.0.0.1-10.0.0.255
          static:
          - 10.0.255.0-10.0.255.254
        type: manual
      - name: private
        subnets:
        - azs:
          - z1
          cloud_properties:
            ...
          gateway: 10.0.0.1
          range: 10.0.0.0/16
          reserved:
          - 10.0.0.1-10.0.0.255
          static:
          - 10.0.255.0-10.0.255.254
        type: manual
      
    • EXTERNAL-IP-ADDRESS: Can be the same as STATIC-IP-ADDRESS if you don't plan to assign an external IP to this deployment.

  9. Create a new manifest. For example, with vim:

    1
    vim credhub-uaa-manifest.yml
    

    Copy and paste the contents of the following code block in to your new manifest:

      1
      2
      3
      4
      5
      6
      7
      8
      9
     10
     11
     12
     13
     14
     15
     16
     17
     18
     19
     20
     21
     22
     23
     24
     25
     26
     27
     28
     29
     30
     31
     32
     33
     34
     35
     36
     37
     38
     39
     40
     41
     42
     43
     44
     45
     46
     47
     48
     49
     50
     51
     52
     53
     54
     55
     56
     57
     58
     59
     60
     61
     62
     63
     64
     65
     66
     67
     68
     69
     70
     71
     72
     73
     74
     75
     76
     77
     78
     79
     80
     81
     82
     83
     84
     85
     86
     87
     88
     89
     90
     91
     92
     93
     94
     95
     96
     97
     98
     99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    179
    180
    181
    182
    183
    184
    185
    186
    187
    188
    189
    190
    191
    192
    193
    194
    195
    196
    197
    198
    199
    200
    201
    202
    203
    204
    205
    206
    207
    208
    209
    210
    211
    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222
    223
    224
    225
    226
    227
    228
    229
    230
    231
    232
    233
    234
    235
    236
    237
    238
    239
    240
    241
    242
    243
    244
    245
    246
    247
    248
    249
    250
    251
    252
    253
    254
    255
    256
    257
    258
    259
    ---
    name: credhub-uaa
    instance_groups:
    - name: credhub-uaa
      azs:
      - z1
      instances: 1
      vm_type: default
      persistent_disk_type: default
      stemcell: xenial
      networks:
      - name: ((deployment-network))
        static_ips:
        - ((internal-ip-address))
      jobs:
      - name: uaa
        release: uaa
        properties:
          encryption:
            active_key_label: 'key-1'
            encryption_keys:
              - label: 'key-1'
                passphrase: "((uaa_encryption_key))"
          login:
            saml:
              serviceProviderCertificate: ((uaa_login_saml.certificate))
              serviceProviderKey: ((uaa_login_saml.private_key))
          uaa:
            clients:
              admin:
                authorized-grant-types: client_credentials
                scope: uaa.none
                authorities: uaa.admin,clients.read,clients.write,clients.secret,scim.read,scim.write,clients.admin
                secret: "((uaa_admin_client_secret))"
              credhub_cli:
                authorized-grant-types: password,refresh_token
                authorities: uaa.none
                scope: credhub.read,credhub.write
                secret: "" # credhub expects this to be empty
                access-token-validity: 120
                refresh-token-validity: 1800
                override: true
              concourse_client:
                authorized-grant-types: client_credentials
                authorities: credhub.read,credhub.write
                scope: credhub.read,credhub.write
                secret: "((concourse_credhub_client_secret))"
                access-token-validity: 120
                refresh-token-validity: 1800
                override: true
            jwt:
              policy:
                active_key_id: key-1
                keys:
                  key-1:
                    signingKey: ((uaa_jwt_signing_key.private_key))
            scim:
              users:
                - name: admin
                  password: "((cf_admin_password))"
                  groups:
                    - uaa.admin
                - name: credhub
                  password: "((credhub_user_password))"
                  groups:
                    - credhub.read
                    - credhub.write
            sslCertificate: ((uaa_ssl.certificate))
            sslPrivateKey: ((uaa_ssl.private_key))
            url: "((uaa_external_url))"
          uaadb:
            address: ((db_host))
            tls: disabled
            databases:
            - name: uaadb
              tag: uaa
            db_scheme: postgres
            port: ((db_port))
            roles:
            - name: uaaadmin
              password: "((uaa_db_user_password))"
              tag: admin
      - name: credhub
        release: credhub
        properties:
          credhub:
            port: 8844
            tls:
              certificate: ((credhub_ssl.certificate))
              private_key: ((credhub_ssl.private_key))
            data_storage:
              type: postgres
              username: credhubadmin
              password: ((credhub_db_user_password))
              host: ((db_host))
              port: ((db_port))
              database: credhub
              require_tls: false
            authentication:
              uaa:
                ca_certs:
                  - ((credhub-ca.certificate))
                enabled: true
                url: "((uaa_external_url))"
                wait_for_start: true
            encryption:
              keys:
                - provider_name: internal-provider
                  key_properties:
                    encryption_password: "((credhub_encryption_key))"
                  active: true
              providers:
                - name: internal-provider
                  type: internal
            authorization:
              acls:
                enabled: true
              permissions:
                - path: /*
                  actors:
                    - uaa-client:concourse_client
                    - uaa-client:credhub_cli
                  operations:
                    - read
                    - write
                    - delete
                    - read_acl
                    - write_acl
      - name: bpm
        release: bpm
      - name: postgres
        release: postgres
        properties:
          databases:
            address: 127.0.0.1
            port: 5432
            databases:
            - name: credhub
            - name: uaadb
            roles:
            - name: admin
              password: ((database-admin))
            - name: uaaadmin
              password: ((uaa_db_user_password))
            - name: credhubadmin
              password: ((credhub_db_user_password))
            tls:
              ca: ((database-tls.ca))
              certificate: ((database-tls.certificate))
              private_key: ((database-tls.private_key))
    
    releases:
    - name: uaa
      version: "((uaa_version))"
    
    - name: credhub
      version: "((credhub_version))"
    
    - name: postgres
      version: "((postgres_version))"
    
    - name: bpm
      version: "((bpm_version))"
    
    variables:
    - name: uaa_encryption_key
      type: password
    
    - name: uaa_admin_client_secret
      type: password
    
    - name: cf_admin_password
      type: password
    
    - name: uaa_jwt_signing_key
      type: rsa
    
    - name: concourse_credhub_client_secret
      type: password
    
    - name: credhub_client_secret
      type: password
    
    - name: credhub_user_password
      type: password
    
    - name: credhub_encryption_key
      type: password
    
    - name: credhub_db_user_password
      type: password
    
    - name: uaa_db_user_password
      type: password
    
    - name: database-admin
      type: password
    
    - name: credhub-ca
      type: certificate
      options:
        is_ca: true
        common_name: CredHub CA
    
    - name: database-tls
      type: certificate
      options:
        ca: credhub-ca
        common_name: ((external-ip-address))
        alternative_names:
        - ((external-ip-address))
        - ((internal-ip-address))
        - 127.0.0.1
        - localhost
    
    - name: uaa_ssl
      type: certificate
      options:
        ca: credhub-ca
        common_name: ((external-ip-address))
        alternative_names:
        - ((external-ip-address))
        - ((internal-ip-address))
        - 127.0.0.1
        - localhost
    
    - name: credhub_ssl
      type: certificate
      options:
        ca: credhub-ca
        common_name: ((external-ip-address))
        alternative_names:
        - ((external-ip-address))
        - ((internal-ip-address))
        - 127.0.0.1
        - localhost
    
    - name: uaa_login_saml
      type: certificate
      options:
        ca: credhub-ca
        common_name: ((external-ip-address))
        alternative_names:
        - ((external-ip-address))
        - ((internal-ip-address))
        - 127.0.0.1
        - localhost
    
    stemcells:
    - alias: xenial
      os: ubuntu-xenial
      version: latest
    
    update:
      canaries: 1
      canary_watch_time: 10000-200000
      max_in_flight: 3
      serial: false
      update_watch_time: 10000-200000
    
  10. Deploy the new manifest:

    1
    2
    3
    bosh -e ALIAS deploy -d credhub-uaa credhub-uaa-manifest.yml \
      --vars-file credhub-vars.yml \
      --vars-store credhub-vars-store.yml
    

Integrate Concourse with Credhub

In this section, you'll find steps for modifying your existing Concourse deployment to integrate with Credhub. This step assumes you use concourse-bosh-deployment to deploy your Concourse. See the Prerequisites section for more information.

Cluster Credentials

When you deploy Concourse for the first time, BOSH creates a file to store your cluster credentials. If you've followed the Concourse for VMware Tanzu Installation Guide, this should be named cluster-creds.yml. When you deploy in the following steps, use this exact same file to ensure BOSH doesn't recreate these values.

To achieve the intended configuration, Concourse must be deployed with an ops-file called credhub.yml in the cluster/operations/ directory located in the concourse-bosh-deployment repo. This ops file contains variables that must be populated before we deploy.

  1. Update the cluster-creds.yml vars-store with the appropriate variables. For example:

    1
    2
    3
    4
    5
    6
    7
    8
    ...
    credhub_ca_cert: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----
    credhub_url: "https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844"
    credhub_client_id: "concourse_client"
    credhub_client_secret: "CLIENT-SECRET"
    

    Where:

    • credhub_ca_cert is the credhub-ca/ca value in the vars-store file credhub-vars-store.yml:

       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      credhub-ca:
      ca: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
      certificate: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
      private_key: |
          -----BEGIN RSA PRIVATE KEY-----
          ...
          -----END RSA PRIVATE KEY-----
      
      • EXTERNAL-IP-OR-URL-OF-CREDHUB: The value of external-ip-address in the vars-file credhub-vars.yml from the Credhub deployment.
      • CLIENT-SECRET: The value of concourse_credhub_client_secret in the vars-store file credhub-vars-store.yml from the Credhub deployment.
  2. Redeploy Concourse.

    The following is a sample BOSH deploy command. You may have to adjust it to match your deployment:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    bosh deploy \
      -e ENVIRONMENT-ALIAS \
      -d DEPLOYMENT-NAME ./cluster/concourse.yml \
      -l versions.yml \
      -l variables.yml \
      -o ./cluster/operations/backup-atc.yml \
      -o ./cluster/operations/basic-auth.yml \
      -o ./cluster/operations/privileged-http.yml \
      -o ./cluster/operations/static-web.yml \
      -o ./cluster/operations/credhub.yml \
      --vars-store cluster-creds.yml
    

    Where:

    • ENVIRONMENT-ALIAS is the BOSH environment alias
    • DEPLOYMENT-NAME is the name of your choice for your Concourse deployment

    BOSH will show you the delta in the manifest you're applying. Your delta should look like this, although your version numbers may vary:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    Using deployment 'concourse-credhub'
    
    Release 'concourse/5.5.8' already exists.
    
    Release 'bpm/1.1.5' already exists.
    
    Release 'postgres/39' already exists.
    
      instance_groups:
      - name: web
        jobs:
        - name: web
          properties:
    +       credhub:
    +         client_id: "<redacted>"
    +         client_secret: "<redacted>"
    +         tls:
    +           ca_cert:
    +             certificate: "<redacted>"
    +         url: "<redacted>"
    

Verify Integration

Now that Credhub and UAA have been deployed, you can verify that everything works by setting a secret using the Credhub CLI, and then fetching the secret from a Concourse pipeline.

  1. Extract Credhub CA Certificate into a certificate file called credhub-ca.crt. You can get the value of the CA certificate from the vars-store file credhub-vars-store.yml.

    Copy the value of credhub-ca/ca into credhub-ca.crt.

    1
    2
    3
    4
    5
    credhub-ca:
      ca: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
    

    The contents of credhub-ca.crt should look like this:

    1
    2
    3
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    
  2. Using the Credhub CLI, log into the Credhub server.

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    $ credhub api \
    --server https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844 \
    --ca-cert=credhub-ca.crt
    
    Setting the target url: https://EXTERNAL-IP-OR-URL-OF-CREDHUB:8844
    
    $ credhub login \
    --client-name concourse_client \
    --client-secret "CLIENT-SECRET"
    
    Login Successful
    

    Where:

    • CLIENT-SECRET is the value of concourse_credhub_client_secret in the Credhub deployment's vars-store file credhub-vars-store.yml
  3. Create a secret in Credhub that you'll use a pipeline in Concourse to retrieve later. The next step assumes you login to Concourse with the main team:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    $ credhub set \
    -n /concourse/main/test \
    --type value \
    --value "a secret"
    
    id: 56caaffc-64f2-4333-a4a5-78f8e78efa3b
    name: /concourse/main/test
    type: value
    value: <redacted>
    version_created_at: "2019-11-20T21:28:14Z"
    
  4. Create a file called test-pipeline.yml. For example, with vim:

    1
    vim test-pipeline.yml
    
  5. Copy and paste the contents of the following code block into test-pipeline.yml:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    ---
    jobs:
      - name: job
        public: true
        plan:
          - task: simple-task
            config:
              platform: linux
              image_resource:
                type: registry-image
                source: { repository: alpine }
              run:
                path: echo
                args: ["This is ((test))"]
    
  6. Set and run the pipeline:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    $ fly -t <target> login -c http://<web_ip>:8080 -u <username> -p <password> -n main
    logging in to team 'main'
    target saved
    
    $ fly -t <target> sp -p test -c test-pipeline.yml
    jobs:
      job job has been added:
    + name: job
    + plan:
    + - config:
    +     container_limits: {}
    +     image_resource:
    +       source:
    +         repository: alpine
    +       type: registry-image
    +     platform: linux
    +     run:
    +       args:
    +       - This is ((test))
    +       path: echo
    +   task: simple-task
    + public: true
    
    $ fly -t <target> up -p test
    unpaused 'test'
    
    $ fly -t <target> tj -j test/job
    started test/job #1
    
    initializing
    running echo This is a secret
    This is a secret
    succeeded
    

    Success

    The output of the job should contain this line somewhere:

    1
    This is a secret
    

Tip

To learn more about how Concourse integrates with Credhub, check out the OSS Credhub documentation: https://concourse-ci.org/credhub-credential-manager.html

Setting up Credhub for multiple teams (ACL)

To enable Concourse users to set their secrets in Credhub you need to create accounts for them in UAA, and then set their path permissions in Credhub, referencing their newly created UAA accounts.

  1. To create a UAA account that a Credhub user can use, run these commands:

    1
    2
    3
    uaac create-user USER-NAME --email EMAIL-ADDRESS --password PASSWORD --familyName FAMILY-NAME --givenName GIVEN-NAME
    uaac add-member credhub.read USER-NAME
    uaac add-member credhub.write USER-NAME
    
  2. Next, get the UAA ID of the user you just created.

    1
    uaac get-user USER-NAME
    
  3. Then grant access to the path in Credhub for the user you just created:

    1
    credhub set-permission --actor uaa-user:USER-UAA-ID --operations read,write,delete --path "/concourse/TEAM-NAME/*"