Skip to content

Concourse Architecture

This topic describes the VMs and software components that make up a Concourse installation.

Component Diagram

Concourse software components can be logically split into two main subsystems: a web (or server) subsystem, and one or more workers. In a less complicated setup, both subsystems can run on a single VM. In more complex systems, Concourse web and worker subsystems run on multiple VMs.

This diagram shows a Concourse deployment with web and server subsystems on separate VMs. You can deploy each subsystem on a single VM, as shown, or on multiple VMs in parallel. A dedicated database VM runs Concourse’s PostgreSQL server. If needed, an optional load balancer points external traffic to web VMs.

Arch Doc

The Component Descriptions below explain what each component does. The Internal vs. External Workers section below describes the two setups that enable the Air Traffic Controller (ATC) to communicate with workers approved by the Transit Security Administration (TSA), depending on whether or not it can access the worker VMs directly.

Component Descriptions

The following table describes the function of each Concourse component:

Subsystem Component Function
Web ATC Runs an HTTP server that:
  • Runs the Concourse web UI.
  • Serves the Concourse API on port 8080, to support the Fly CLI and other tools.
  • Schedules pipelines to run on worker VMs.
TSA Runs an SSH server that:
  • Listens on port 2222.
  • Securely registers worker VMs for use by the ATC.
  • For external worker VMs, handshakes with Beacon to open a reverse SSH tunnel for ongoing communication between the ATC and the worker.
  • Pings workers for heartbeat every 30 seconds, updates ATC with worker metric changes, and, if there is no response, notifies ATC that the worker is in a stalled state.
Database PostgreSQL Stores pipeline data, such as build configurations, build logs, and resource versions, for use by worker VMs.
Worker Beacon
  • Sends Garden and Baggageclaim addresses to the ATC, to register that the worker is ready to run Concourse build pipelines
  • For external worker VMs, handshakes with TSA to open a reverse SSH tunnel for ongoing communication between the ATC and the worker.
Garden Runs an HTTP server that:
  • Listens on port 7777.
  • Provides an API for the ATC to manage containers to do things like:
Baggageclaim Runs an HTTP server that:
  • Listens on port 7788.
  • Provides an API for the ATC to manage disk resources, or volumes, mounted to containers.
  • Uses volumes to store persistent caches of resource versions and to propagate artifacts such as caches and task outputs through the containers used for a build.

For more information, see Concepts in the Concourse documentation.

Internal vs. External Workers

Concourse workers can be either internal, also called local; or external, also called remote. These two types of worker VMs differ as follows:

  • A worker VM is internal when the web VM, which hosts the ATC, can make direct HTTP calls to the worker VM. This is typical when web and worker VMs are all on the same private network.

  • A worker VM is external when the web VM cannot directly connect to the worker VM through HTTP or SSH. This is typical when the worker VMs are behind a firewall in a separate network from the web VM or when security groups prevent communication from web VM to worker.

You might use both internal and external workers in the same deployment. For example, if your main Concourse installation runs on Linux and you need Windows workers to run tests in a remote, Windows-based infrastructure.

Both internal and external workers can use SSH to securely log in to the web VM.

Concourse sets up secure communications from the ATC to worker VMs in two ways, depending on whether the worker VMs are internal or external, as described below.

Internal Worker Communications

Concourse sets up communications between the ATC and internal worker VMs as follows:

  1. The worker's Beacon connects to the TSA through SSH and runs the register-worker command to the TSA, passing in the worker payload. For more information, see payload in GitHub.

  2. The TSA starts heartbeating every 30 seconds, pinging the worker for its current number of Garden containers and Baggageclaim volumes, and sending updated information to the ATC. If the worker fails to respond, the TSA notifies ATC that worker is stalled.

  3. ATC makes direct HTTP calls to Garden and Baggageclaim APIs on the worker, to create containers and volumes for running containerized processes.

External Worker Communications

In general, an external worker will reach out of it's network, and attempt to connect to the tsa_host in it's configuration. If the TSA Host Public key and the authorized_keys match, the worker will register with the TSA & start processing JOBs.

Concourse sets up communications between the ATC and external worker VMs as follows:

  1. The worker's Beacon connects to the TSA through SSH and runs the forward-worker command to the TSA, passing in the worker payload. For more information, see payload in GitHub.

  2. The forward-worker command establishes two unique hidden local ports on the web VM, and creates a reverse-SSH tunnel from these local ports to the external worker's Garden and Baggageclaim ports.

  3. The TSA replaces the ATC's Garden and Baggageclaim addresses for the worker to the hidden local port addresses.

  4. The TSA starts heartbeating every 30 seconds, pinging the worker for its current number of Garden containers and Baggageclaim volumes, and sending updated information to the ATC. If the worker fails to respond, the TSA notifies ATC that worker is stalled.

  5. The ATC calls Garden and Baggageclaim APIs on the worker by sending the HTTP requests to its own hidden local ports. The SSH daemon on the web VM then forwards the requests to the worker VM through the reverse-SSH tunnel.

Setting up External Workers

For information on how to set up an external worker, see the Setting up an External Worker operations guide.

Operating Concourse at Scale

The Concourse for VMware Tanzu team is frequently asked questions about the best practices for operating Concourse deployments at scale. We're exploring a more comprehensive approach to prescribing scaling advice, but in the meantime, situations and contexts vary. To date, the team has learned a lot about scaling from operating and testing Concourse at the following scales:

BOSH Deployment

  • A single BOSH-deployed Concourse containing:
  • 4 Web Nodes
  • 34 Workers
  • 1 Postgres Database
  • Workloads:
  • 134 Teams
  • 534 Pipelines
  • Average of 15 builds/minute

Helm Deployment

  • A single Helm-deployed Concourse containing:
  • 2 Web Nodes
  • 7 Workers
  • 1 Postgres Database
  • Workloads:
  • 30 Teams
  • 91 Pipelines
  • Average of 3 builds/minute

Concourse is a powerful, flexible tool that can be successfully scaled well beyond these numbers. However, this is the extent of scale that we actively maintain and test against to ensure a stable operating experience, as it reflects the average size of customer production deployments. Contact your designated support engineer for help with scaling, configuration, and deployment strategies that will meet your needs.