Skip to content

Monitoring and Troubleshooting

This topic broadly outlines techniques for troubleshooting your Pivotal Concourse installation.

Troubleshooting with Fly Commands

To get help on all fly commands, run: fly --help.

The following selected fly commands give you useful information to help you troubleshoot Pivotal Concourse environments and pipelines.

Troubleshooting Pivotal Concourse Environments

You can use the following fly commands to troubleshoot possible environment problems.

Fly Command Description
containers Lists active containers.
This confirms which container or task is placed on which worker.
workers Lists registered workers.
This helps you verify that the number of containers does not exceed the maximum number allowable.
prune-worker Removes a non-running worker.
Stops Pivotal Concourse from tracking an out-of-commission worker.
volumes Lists active volumes.
Checks disk usage across workers.

Troubleshooting Pipelines

You can use the following fly commands to troubleshoot possible pipeline problems.

Fly Command Description
pipelines Lists configured pipelines.
builds Shows build history.
This is useful for listing build IDs of one-off tasks ran previously using execute.
validate-pipeline Validates a pipeline's configuration without calling set-pipeline.
check-resource Checks for new versions.
This is useful when developing a new resource.
watch Shows logs of in-progress builds.
intercept Displays build steps for a running or recent build.
execute Submits local tasks.
This is useful for spinning up a task quickly to test it before putting it in a job.

Common Pivotal Concourse Issues

The following shows some common problems and solutions.

Problem Error Description Solution
The worker is out of disk space An error states there is an inability to create a storage volume and may state that permissions are denied. Increase persistent disk size for the worker or increase the number of worker VMs.
Container limit is reached Cannot create container: limit of 250 containers reached. This error state is unlikely to appear. Check fly containers.

Increase number of worker VMs.

Decrease gc_interval if set to custom value. A large interval could mean that there are too many expired containers.
Job does not start This error may present as the build getting stuck in Pending state. Restart the ATC:
Log in as a root user on Pivotal Concourse web VMs where the ATC job is located. Run the monit restart atc command.
Build fails when updating Pivotal Concourse from a job on the same instance When a build fails after BOSH deploys a Pivotal Concourse update from a job running on that Pivotal Concourse instance, typically the job fails with a "worker for container not found" error. This is expected behavior; the BOSH Director will recreate the worker VM. Run the job again.
BOSH cannot finish worker upgrade while tasks are running BOSH is not able to restart the worker job to finalize the upgrade until all work is completed. If you have a long-running task, wait for the task to be completed. If you need to upgrade quickly, cancel running tasks and jobs.

Access Concourse Logs

You might need to contact Pivotal Support for help identifying a problem. In that case, support might ask you to send job log files.

For general information about accessing log files, see Location and use of logs and Advanced Troubleshooting with the BOSH CLI.

Other Troubleshooting Resources

The following links provide other troubleshooting resources.

Enabling syslog forwarding and getting logs for other Concourse componentsVM Logs
Common BOSH issuesBOSH tips