Reference Architecture for Pivotal Cloud Foundry on GCP
Page last updated:
This guide presents a reference architecture for Pivotal Cloud Foundry (PCF) on Google Cloud Platform (GCP). This document also outlines multiple networking solutions. All these architectures are validated for production-grade PCF deployments using multiple (3+) AZs.
The following diagram provides an overview of a reference architecture deployment of PCF on GCP.
The following table lists the components that are part of a reference architecture deployment with three availability zones.
|Component||Reference Architecture Notes|
|Domains & DNS||CF Domain Zones and routes in use by the reference architecture include: domains for *.apps and *.system (required), a route for Ops Manager (required), a route for doppler (required), a route for Loggregator (required), a route for ssh access to app containers (optional) and a route for TCP routing to apps (optional). Reference architecture uses GCP Cloud DNS as the DNS provider.|
|Ops Manager||Deployed on the infrastructure subnet and accessible by FQDN or through an optional Jumpbox.|
|BOSH Director||Deployed on the infrastructure subnet.|
|Gorouter(s)||Accessed through the HTTP and TCP WebSockets load balancers. Deployed on the ERT subnet, one job per availability zone.|
|Diego Brain(s)||This component is required, however the SSH container access functionality is optional and enabled through the SSH Proxy load balancer. Deployed on the ERT subnet, one job per availability zone.|
|TCP Router(s)||Optional feature for TCP routing. Deployed on the ERT subnet, one job per availability zone.|
|CF Database||Reference architecture uses GCP Cloud SQL rather than internal databases. Configure your database with a strong password and limit access only to components that require database access.|
|CF Blob Storage and Buckets||For buildpacks, droplets, packages and resources. Reference architecture uses Google Cloud Storage rather than internal file storage.|
|Services||Deployed on the PCF managed services subnet. Each service is deployed to each availability zone.|
|Dynamic Services||Reserved for future use, dynamic services are deployed on their own subnet. Dynamic services are services autoprovisioned by BOSH based on a trigger, such as a request for that service. Pivotal recommends provisioning the multi-tenant dynamic services subnet as a /22 block.|
This section describes the possible network layouts for PCF deployments as covered by the reference architecture of PCF on GCP.
At a high level, there are currently two possible ways of granting public Internet access to PCF as described by the reference architecture:
- NATs provide public access to PCF internals. This scenario is currently outlined in the existing installation documentation for GCP deployments of PCF.
- Every PCF VM receives its own public IP address (no NAT).
Providing each PCF VM with a public IP address is the most recommended architecture, because of increased latency due to NATs as well as extra maintenance required for NAT instances that cannot be deployed with BOSH.
However, if you require NATs, you may refer to the following section.
This diagram illustrates the case where you want to expose only a minimal number of public IP addresses.
If you prefer not to use a NAT solution, you can configure PCF on GCP to assign public IP addresses for all components. This type of deployment may be more performant since most of the network traffic between Cloud Foundry components are routed through the front end load balancer and the Gorouter.
The following table lists the network objects expected for each type of reference architecture deployment with three availability zones (assumes you are using NATs).
|Network Object||Notes||Minimum Number: NAT-based||Minimum Number: Public IP Addresses|
|External IPs||For a NAT solution, use 1) global IP address for apps and system access 2) Ops Manager (or optional Jumpbox)||2||30+|
|NATs||One NAT per availability zone.||3||0|
|Network||One per deployment. GCP Network objects allow multiple subnets with multiple CIDRs, so a typical deployment of PCF will likely only ever require one GCP Network object||1||1|
|Subnets||Separate subnets for 1) infrastructure (Ops Manager, Ops Manager Director, Jumpbox) 2) ERT 3) services. Using separate subnets allows you to configure different firewall rules due to your needs.||3||3|
|Routes||Routes are typically created by GCP dynamically when subnets are created, but you may need to create additional routes to force outbound communication to dedicated SNAT nodes. These objects are required to deploy PCF without public IP addresses.||3+||3|
|Firewall Rules||GCP firewall rules are bound to a Network object and can be created to use IP ranges, subnets, or instance tags to match for source & destination fields in a rule. The preferred method use in the reference architecture deployment is instance tags.||6+||6+|
|Load balancers||Used to handle requests to Gorouters and infrastructure components. GCP uses 2 or more load balancers. The HTTP load balancer and TCP WebSockets load balancer are both required. The TCP Router load balancer used for TCP routing feature and the SSH load balancer that allows SSH access to Diego apps are both optional. The HTTP load balancer provides SSL termination.||2+||2+|
|Jumpbox||Optional. Provides a way of accessing different network components. For example, you can configure it with your own permissions and then set it up to access to Pivotal Network to download tiles. Using a Jumpbox is particularly useful in IaaSes where Ops Manager does not have a public IP address. In these cases, you can SSH into Ops Manager or any other component through the Jumpbox.||(1)||(1)|
This section provides more background on the reasons behind certain network configuration decisions, specifically for the Gorouter.
Load Balancer to Gorouter Communications and TLS Termination
In a PCF on GCP deployment, the Gorouter receives two types of traffic:
Unencrypted HTTP traffic on port 80 that is decrypted by the HTTP(S) load balancer.
Encrypted secure web socket traffic on port 443 that is passed through the TCP WebSockets load balancer.
TLS is terminated for HTTPS traffic on the HTTP load balancer and is terminated for WebSockets (WSS) traffic on the Gorouter.
PCF deployments on GCP use two load balancers to handle Gorouter traffic because HTTP load balancers currently do not support WebSockets.
GCP routers do not respond ICMP; therefore, Pivotal recommends disabling ICMP checks in Ops Manager Director network configuration.