Control Plane Reference Architectures

Page last updated:

This topic describes topologies and best practices for deploying Concourse on BOSH and using Concourse to manage Ops Manager foundations. For production environments, VMware recommends deploying Concourse with BOSH.

Overview

Concourse is the main continuous integration and continuous delivery (CI/CD) tool that VMware and the open-source Cloud Foundry communities use to develop, test, deploy and manage Ops Manager foundations.

The three topologies described in this topic govern the network placement and relationship between two systems:

  • The control plane runs Concourse to gather sources for, integrate, update, and otherwise manage foundations. This layer may also host internal Docker registries, S3 buckets, Git repositories, and other tools.

  • Each foundation runs Ops Manager on an instance of BOSH.

Each topology described below has been developed and validated in multiple customer and Pivotal Labs by VMware environments.

Deployment Topologies

Security policies are usually the main factor that determines which CI/CD deployment topology best suits a site’s needs. Specifically, the decision depends on what network connections are allowed between the control plane and the Ops Manager foundations that it manages.

These three topologies answer a range of security needs, ordered by increasing level of security around the Ops Manager foundations:

  • Topology 1: Concourse server and worker VMs all colocated on the control plane

  • Topology 2: Concourse server on the control plane, and remote workers colocated with Ops Manager foundations

  • Topology 3: Multiple Concourse servers and workers colocated with Ops Manager foundations

Across these three topologies, the increasing level of Ops Manager foundation security correlates with:

  • Increasing network complexity
  • Increasing effort required for initial deployment and ongoing maintenance

Security Decision Factors

The graph below illustrates the decision factors dictated by network security policy and the Concourse deployment topologies that adhere to those policies.

Concourse decision

Additional Decision Factors

In addition to security policy, deciding on a CI/CD deployment topology may also depend on factors such as:

  • Network latency across network zones
  • Air-gapped vs. Internet-connected environments
  • Resource limitations

Credentials Management and Storage Services include notes and recommendations regarding credentials management, Docker registries, S3 buckets, and Git repositories, but does not cover these tools extensively.

Topology Objects

Complete deployment topologies dictate the internal placement or remote use of:

  • Concourse server and worker VMs
  • Credentials managers such as CredHub or Vault
  • Docker registries, public or private
  • S3 or other storage buckets, public or private
  • Git or other code repositories, public or private

Topology 1: Concourse Server and Worker VMs All Colocated on the Control Plane

This simple topology follows network security policies that allow a single control plane to connect to all Ops Manager foundations deployed across multiple network zones or data centers, as shown in the diagram below:

Concourse single plane

In this topology, the Concourse server and worker VMs, along with other tools (such as Docker registry, S3 buckets, Vault server), are all deployed to the same subnet.

Connectivity Requirements

Concourse worker VMs must be allowed to connect to:

  • The Ops Manager VM or a jumpbox VM in all of the Ops Manager foundations networks.

  • The vCenter API for each Ops Manager foundation on vSphere.

Performance Notes

Network data transfers between the control plane and each Ops Manager foundation network zone may carry large files such as Ops Manager tiles, release files, and Ops Manager foundation backup pipelines output.

VMware recommends that you test network throughput and latency between those network zones to make sure that data transfer rates do not make pipeline execution times unacceptably long.

Firewall Requirements

All Concourse worker VMs are required to connect to certain VMs on Ops Manager foundation subnets across networks zones or data centers.

For required VMs, ports, and external websites required for Ops Manager CI/CD pipelines, see Ops Manager CI/CD Pipelines.

Pros:

  • Simplified deployment and maintenance of centralized control plane, which requires only one BOSH deployment and runs one BOSH Director.

  • Simplified setup and maintenance of Ops Manager CI/CD pipelines. All pipelines and Concourse teams use a single, centralized server.

Cons:

  • You may have to configure firewall rules in each Ops Manager network to allow connectivity from workers in CI/CD zone, as mentioned above.

Topology 2: Concourse Server on the Control Plane, and Remote Workers Colocated with Ops Manager Foundations

This topology supports environments where Ops Manager foundation VMs cannot receive incoming traffic from IP addresses outside their network zone or data center, but they can initiate outbound connections to outside zones.

As in Topology 1, the Concourse server, and potentially other VMs for tools such as Docker registries or S3 buckets, are deployed to a dedicated subnet. The difference is that Concourse worker VM pools are deployed inside each Ops Manager foundation subnet, network zone, or data center, as shown in the diagram below:

Concourse multi zone

Connectivity Requirements

Concourse worker VMs in each Ops Manager foundation network zone or data center must:

  • Connect to the Ops Manager VM or a jumpbox VM in each Ops Manager foundations network.

  • Connect to the vCenter API for each Ops Manager foundation on vSphere.

  • Have outbound connection access to the Concourse web/ATC server on port 2222. This allows them to handshake with the Concourse server to open a reverse SSH tunnel for ongoing communication between them and the Concourse ATC.

For more information, see Concourse Architecture in the Concourse documentation.

Performance Notes

Remote workers have to download large installation files from either the Internet or from a configured S3 artifacts repository. For Ops Manager backup pipelines, workers may also have to upload large backup files to the S3 repository.

VMware recommends that you test network throughput and latency between those network zones to make sure that data transfer rates do not make pipeline execution times unacceptably long.

Firewall Requirements

Remote worker VMs require outbound access to the Concourse web/ATC server on port 2222.

Remote worker VMs inside each Ops Manager foundation network zone or data center are required to connect to VMs in their colocated foundation. They may also require access to external web sites or S3 repositories for downloading installation files.

For required VMs and ports required for Ops Manager CI/CD pipelines, see Ops Manager CI/CD Pipelines.

Pros:

  • Relatively simple maintenance of centralized control plane, which contains single Concourse server and other tools.

  • Simplified setup and maintenance of Ops Manager CI/CD pipelines. All pipelines and Concourse teams use a single, centralized server.

Cons:

  • You must reconfigure firewalls to grant outbound access to remote worker VMs.

  • In addition to deploying the control plane, you need an additional BOSH deployment and running BOSH Director for each Ops Manager Foundation network zone or data center.

  • You must manage multiple Concourse worker pools in multiple locations.

Topology 3: Multiple Concourse Servers and Workers Colocated with Ops Manager Foundations

This topology supports environments where Ops Manager foundation VMs can only be accessed from within the same network zone or data center. This scenario requires deploying complete and dedicated control planes within each deployment zone, as shown in the diagram below:

Concourse multi foundation

Performance Notes

Since workers run in the same network zone or data center as the Ops Manager foundation, data transfer throughput should not limit pipeline performance.

Firewall Requirements

Air-gapped environments require some way to bootstrap S3 repository with Docker images and Ops Manager releases files for pipelines from external sites.

Non-air-gapped environments, in which workers can download required files from external websites, need those websites to be on the allow list for the proxy or firewall setup.

Pros:

  • Requires little or no firewall rules configuration for control plane VMs. In non-air-gapped environments, worker VMs download Ops Manager releases and Docker images for pipelines from external websites.

Cons:

  • Requires deploying and maintaining multiple Concourse and other tools.

  • Requires deploying and maintaining multiple Ops Manager pipelines for each Concourse server.

  • For air-gapped environments, requires setting up an S3 repository for each control plane.

Deploying CI/CD to the Control Plane

There are a few alternatives to deploy BOSH Directors, Concourse servers, and other tool releases to the control plane. The links below provide information about those alternatives:

Manual Deployments

Automated Deployments

Control Plane Deployment Best Practices

These sections describe some best practices for control plane components that apply across all deployment topologies.

Dedicated BOSH Director

VMware recommends deploying Concourse and other control plane tools on their own dedicated BOSH layer, with their own BOSH Director that runs separately from the Ops Manager foundations that the control plane manages.

VMware does not recommend using an existing Ops Manager BOSH Director instance to deploy Concourse and other software such as Minio S3, a private Docker registry, or CredHub. Sharing the same BOSH Director with Ops Manager deployments increases the risk of accidental or undesired updates or deletion of those deployments.

Dedicating a BOSH Director to Concourse other control plane tools also provides higher flexibility for Ops Manager foundation upgrades and updates, such as stemcell patches.

The diagram below illustrates deploying Concourse and other control plane tools on their own dedicated BOSH layer:

Concourse dedicated bosh

Credentials Management

All credentials and other sensitive information that feeds into a Concourse pipeline should be encrypted and stored using credentials management software such as CredHub or Vault. Never store credentials as plain text in parameter files in file repositories.

Credentials Management with CredHub

Concourse integrates with CredHub to manage credentials in its pipelines. The pipelines reference encrypted secrets stored in a CredHub server and retrieve them automatically during execution of tasks.

To integrate Concourse with a CredHub server, you configure its ATC job’s deployment properties with information about the CredHub server and corresponding UAA authentication credentials.

You can deploy CredHub in multiple ways:

  • As a dedicated VM
  • Integrated with other VMs, such as colocating the CredHub server with the BOSH Director VM or Concourse’s ATC/web VM

Colocating a CredHub server with Concourse’s ATC VM dedicates it to the Concourse pipelines and lets Concourse administrators manage the credentials server. This configuration also means that during Concourse upgrades, the CredHub server only goes down when the Concourse ATC job is also down, which minimizes potential credential server outages for running pipelines.

The diagram below illustrates the jobs of Concourse VMs, along with the ones for the BOSH Director VM, when a dedicated CredHub server is deployed with Concourse:

Concourse bosh jobs

For more information about how to deploy a CredHub server integrated with Concourse, see Concourse Pipelines Integration with CredHub in the PCF Pipelines repository on GitHub.

Credentials Management with Vault

For information about how to configure Vault to manage credentials for Concourse pipelines, see Secure credential automation with Vault and Concourse in the PCF Pipelines repository on GitHub.

Storage Services

These sections describe the storage services your Concourse deployment uses.

Git Server

BOSH and Concourse implement the concepts of infrastructure-as-code and pipelines-as-code. As such, it is important to store all source code for deployments and pipelines in a version-controlled repository.

Ops Manager CI/CD pipelines assume that their source code is kept in Git-compatible repositories. GitHub is the most popular Git-compatible repository for Internet-connected environments. GitLab, BitBucket, and GOGS are examples of Git servers that can be used for both connected and air-gapped environments.

A Git server that contains configuration and pipeline code for a Ops Manager foundation must be accessible to the corresponding worker VMs that run CI/CD pipelines for that foundation.

S3 Repository

In all environments, Concourse requires an S3 repository to store Ops Manager backups and possibly other files. If your deployment requires a private, internal S3 repository but your IaaS lacks built-in options, you can use BOSH to deploy your own S3 releases, such as Minio S3 and EMC Cloud Storage, to your control plane. For more information, see the MinIO BOSH Release and EMC Cloud Storage repositories on GitHub

For air-gapped environments, an S3 repository is also the preferred method to store release files for Ops Manager tiles, stemcells, and buildpacks. Docker images can also be stored in an S3 repository as an alternative to a private Docker registry. For more information, see Offline Pipelines for Airgapped Environments in the PCF Pipelines repository on GitHub.

Private Docker Registry

For air-gapped environments, Docker images for Concourse pipelines must be stored either on a private Docker registry or in an S3 repository. For BOSH-deployed private registry alternatives, see Docker Registry in the Concourse Samples and Recipes on GitHub or the Harbor repository on GitHub.

High Availability

For information about setting up a load balancer to handle traffic across multiple instances of the ATC/web VM and deploying multiple worker instances, see Concourse Architecture in the Ops Manager Concourse documentation.

Ops Manager CI/CD Pipelines

PCF Pipelines

Platform Automation with Concourse (PCF Pipelines) is a collection of Concourse pipelines for installing and upgrading Ops Manager. For more information, see the Platform Automation documentation. To download Ops Manager Platform Automation with Concourse, go to the Platform Automation page on VMware Tanzu Network.

Warning: At time of publication, the PCF Pipelines repository is no longer supported.

To run Pipelines, you need:

  • Ops Manager web UI, API, and VM installed
  • vCenter API installed (for vSphere environments)
  • For proxy- or Internet-connected environments, put these sites on your allow list:

BOSH Backup and Restore (BBR) Pipelines

BBR Pipelines automate foundation backups. For more information, see the BBR PCF Pipeline Tasks repository on GitHub.

To run BBR Pipelines, you need an S3 repository for storing backup artifacts.

Note: BBR Pipelines is a community project not officially supported by VMware.

Pipelines Orchestration Frameworks

PCF Pipelines Maestro uses a Maestro framework to:

  • Automate pipeline creation and management for multiple Ops Manager foundations

  • Promote and audit configuration changes and version upgrades across all foundations

For more information, see the PCF Pipelines Maestro repository on GitHub.

Note: PCF Pipelines Maestro is a community project not officially supported by VMware.

Concourse Team Management Best Practices

When a single Concourse server hosts CI/CD pipelines for more than one Ops Manager foundation, VMware recommends creating one Concourse team (not main) specific to each foundation and associating that team with all pipelines for that foundation, as shown in the diagram below:

Concourse pipelines

Dedicating a Concourse team to each Ops Manager foundation has these benefits:

  • It avoids the clutter of pipelines in a single team. The list of pipelines for each foundation may be long, depending how many tiles are deployed to it.

  • It avoids the risk of operators running a pipeline for the wrong foundation. When a single team hosts maintenance pipelines for multiple foundations, the clutter of dozens of pipelines may lead operators to accidentally run a pipeline targeted at the wrong Ops Manager foundation.

  • It allows for more granular access control settings per team. Pipelines for higher environments may require a more restricted access control than ones from lower environments. Authentication settings for Concourse teams enable that level of control.

  • It allows workers to be assigned to pipelines of a specific foundation. Concourse deployment configuration allows for the assignment of workers to a single team. If that team contains pipelines of only one foundation, then the corresponding group of workers run pipelines only for that foundation. This is useful when security policy requires tooling and automation for a foundation to run on specific VMs.

Concourse Teams for App Development Orgs and Spaces

When a Concourse server hosts pipelines for an app development team, VMware recommends creating a Concourse team associated with that development team, and associating Concourse team membership with org and space membership defined in the VMware Tanzu Application Service for VMs (TAS for VMs) user authentication and authorization (UAA) server.

Associating Concourse teams with TAS for VMs org and space member lists synchronizes access between TAS for VMs and Concourse, allowing developers to see and operate the build pipelines for the apps they develop.