Skip to content

Troubleshooting

Worker public key is no longer an array

In situations where a Concourse instance has multiple workers in different pools, a Concourse manifest might have more than one worker public key. Some users have operations files to append public keys to their manifest at /instance_groups/name=web/jobs/name=web/properties/worker_gateway/authorized_keys. This ops file will fail to interpolate with v5.x.x, since the field is now a string instead of a list.

Instead of appending keys, you can concatenate the two public keys in an ops file using a multi-line yaml string. Pivotal suggests using an ops file that looks like this:

1
2
3
4
5
6
---
- type: replace
  path: /instance_groups/name=web/jobs/name=web/properties/worker_gateway/authorized_keys?
  value: |
    ((first.public_key))
    ((second.public_key))

Missing variables interpolated by Credhub now error

Many Concourse operators use tools like Credhub for centralized credential management for their Concourse instances. In the concourse-bosh-deployment repository referenced in these upgrade guides, there are various examples where variables are used as placeholders which are meant to be replaced at deployment time. In the past, you could use these values as-is without specifying variables, and Credhub would seamlessly take over to interpolate anything that's missing at the time of deployment.

For example, if foo is a key in Credhub, an operator would need to pass ((foo)) to BOSH without interpolating some value for foo first. This results in something like this in your pipeline.yml:

1
2
3
4
5
put: some-bosh-deployment
params:
  ...
  vars:
    secret: "((/bosh-name/group/foo))"

In v5.x.x, this scenario will fail with an error message stating that BOSH cannot find the variable /bosh-name/cf/cf_admin_password. To fix this, move the variable into a variables file:

1
2
---
secret: "((/bosh-name/group/foo))"

This strategy allows us to pass the variable ((/bosh-name/group/foo)) literally to the BOSH deployment. This way, it can be Credhub-managed within the foundation you're deploying.


Enable Certificate Rotation

As of this writing, there’s an operations file on concourse-bosh-deployment master that turns on the Let’s Encrypt ACME service that is not in the v5.x.x release. Using this operations files can help reduce the number of certificates you need to rotate. If you'd like to try this with your v6.x.x deployment, duplicate that ops file into your own repo like so:

1
2
3
4
---
- type: replace
  path: /instance_groups/name=web/jobs/name=web/properties/lets_encrypt?/enabled
  value: true

You may also have to remove the ops file that specifies web TLS certificates, because it doesn’t make sense to say that these certificates automatically rotate and try to specify them at the same time. You can do this by removing the following operations file from the BOSH command that deploys your Concourse:

1
2
# remove this ops file as part of enabling Lets Encrypt ACME
concourse-bosh-deployment/cluster/operations/tls.yml