Troubleshooting
Upgrading Postgres
Older versions of Concourse using Postgres release 28 or previous might have difficulty upgrading smoothly. In these cases, during a BOSH deploy
step you might encounter an error like this when BOSH attempts to update the DB instance:
1 2 |
|
In order to avoid this error, migrate in two steps, using Postgres 30 as an intermediary. For example, if you were upgrading from release 28 to 36:
- Upgrade from release 28 to release 30
- Upgrade from release 30 to release 36
However, if you've encountered the above error and try to redeploy with different versions, you might then encounter this error:
1 2 |
|
The database instance is locked, and must be 'unlocked' before we can continue.
Unlock the database instance
-
SSH into the database VM:
1
bosh -e BOSH-ENVIRONMENT-ALIAS -d DEPLOYMENT-NAME ssh db
Where:
BOSH-ENVIRONMENT-ALIAS
is your BOSH environment aliasDEPLOYMENT-NAME
is your deployment name
-
Switch to the root user:
1
sudo su
-
Find and remove a lock file called
POSTGRES_UPGRADE_LOCK
1
find / -name "POSTGRES_UPGRADE_LOCK"
For example, in the case that the lock file is located in
~/var/vcap/store/postgres/
:1
rm ~/var/vcap/store/postgres/POSTGRES_UPGRADE_LOCK
-
Exit the SSH session:
1 2
exit exit
At this point you will be able to run your bosh deploy
command again, and the database instance should update successfully.
Worker public key is no longer an array
In situations where a Concourse instance has multiple workers in different pools, a Concourse manifest might have more than one worker public key. Some users have operations files to append public keys to their manifest at /instance_groups/name=web/jobs/name=web/properties/worker_gateway/authorized_keys
. This ops file will fail to interpolate with v5.5.x, since the field is now a string instead of a list.
Instead of appending keys, you can concatinate the two public keys in an ops file using a multi-line yaml string. Pivotal suggests using an ops file that looks like this:
1 2 3 4 5 6 |
|
Missing variables interpolated by Credhub now error
Many Concourse operators use tools like Credhub for centralized credential management for their Concourse instances. In the concourse-bosh-deployment
repository referenced in these upgrade guides, there are various examples where variables are used as placeholders which are meant to be replaced at deployment time. In the past, you could use these values as-is without specifying variables, and Credhub would seamlessly take over to interpolate anything that's missing at the time of deployment.
For example, if foo
is a key in Credhub, an operator would need to pass ((foo))
to BOSH without interpolating some value for foo
first. This results in something like this in your pipeline.yml
:
1 2 3 4 5 |
|
In v5.5.x, this scenario will fail with an error message stating that BOSH cannot find the variable /bosh-name/cf/cf_admin_password
. To fix this, move the variable into a variables file:
1 2 |
|
This strategy allows us to pass the variable ((/bosh-name/group/foo))
literally to the BOSH deployment. This way, it can be Credhub-managed within the foundation you're deploying.
Enable Certificate Rotation
As of this writing, there’s an operations file on concourse-bosh-deployment
master that turns on the Let’s Encrypt ACME service that is not in the v5.5.x releases. Using this operations files can help reduce the number of certificates you need to rotate. If you'd like to try this with your v5.5.x deployment, duplicate that ops file into your own repo like so:
1 2 3 4 |
|
You may also have to remove the ops file that specifies web TLS certificates, because it doesn’t make sense to say that these certificates automatically rotate and try to specify them at the same time. You can do this by removing the following operations file from the BOSH command that deploys your Concourse:
1 2 |
|