Monitoring and Troubleshooting
This topic broadly outlines techniques for troubleshooting your Pivotal Concourse installation.
Troubleshooting with Fly Commands
To get help on all
fly commands, run:
The following selected
fly commands give you useful information to help you troubleshoot Pivotal Concourse environments and pipelines.
Troubleshooting Pivotal Concourse Environments
You can use the following
fly commands to troubleshoot possible environment problems.
||Lists active containers.
This confirms which container or task is placed on which worker.
||Lists registered workers.
This helps you verify that the number of containers does not exceed the maximum number allowable.
||Removes a non-running worker.
Stops Pivotal Concourse from tracking an out-of-commission worker.
||Lists active volumes.
Checks disk usage across workers.
You can use the following
fly commands to troubleshoot possible pipeline problems.
||Lists configured pipelines.|
||Shows build history.
This is useful for listing build IDs of one-off tasks ran previously using execute.
||Validates a pipeline's configuration without calling set-pipeline.|
||Checks for new versions.
This is useful when developing a new resource.
||Shows logs of in-progress builds.|
||Displays build steps for a running or recent build.|
||Submits local tasks.
This is useful for spinning up a task quickly to test it before putting it in a job.
Common Pivotal Concourse Issues
The following shows some common problems and solutions.
|The worker is out of disk space||An error states there is an inability to create a storage volume and may state that permissions are denied.||Increase persistent disk size for the worker or increase the number of worker VMs.|
|Container limit is reached||Cannot create container: limit of 250 containers reached. This error state is unlikely to appear.||Check fly containers.
Increase number of worker VMs.
Decrease gc_interval if set to custom value. A large interval could mean that there are too many expired containers.
|Job does not start||This error may present as the build getting stuck in Pending state.||Restart the ATC:
Log in as a root user on Pivotal Concourse web VMs where the ATC job is located. Run the monit restart atc command.
|Build fails when updating Pivotal Concourse from a job on the same instance||When a build fails after BOSH deploys a Pivotal Concourse update from a job running on that Pivotal Concourse instance, typically the job fails with a "worker for container not found" error. This is expected behavior; the BOSH Director will recreate the worker VM.||Run the job again.|
|BOSH cannot finish worker upgrade while tasks are running||BOSH is not able to restart the worker job to finalize the upgrade until all work is completed.||If you have a long-running task, wait for the task to be completed. If you need to upgrade quickly, cancel running tasks and jobs.|
Access Concourse Logs
You might need to contact Pivotal Support for help identifying a problem. In that case, support might ask you to send job log files.
Other Troubleshooting Resources
The following links provide other troubleshooting resources.
|Enabling syslog forwarding and getting logs for other Concourse components||VM Logs|
|Common BOSH issues||BOSH tips|