Reference Architecture for Pivotal Cloud Foundry on AWS
Page last updated:
This guide presents a reference architecture for Pivotal Cloud Foundry (PCF) on Amazon Web Services (AWS). This architecture is valid for most production-grade PCF deployments using three availability zones (AZs).
See AWS on PCF Requirements for general requirements for running PCF and specific requirements for running PCF on AWS.
A PCF reference architecture describes a proven approach for deploying Pivotal Cloud Foundry on a specific IaaS, such as AWS, that meets the following requirements:
- Includes common PCF-managed services such as MySQL, RabbitMQ, and Spring Cloud Services
- Can host at least 100 app instances, or far more
Pivotal provides reference architectures to help you determine the best configuration for your PCF deployment.
The following diagram provides an overview of a reference architecture deployment of PCF on AWS using three AZs.
Note: Each AWS subnet must reside entirely within one AZ. As a result, a multi-AZ deployment topology requires a subnet for each AZ.
The following table lists the components that are part of a base reference architecture deployment on AWS with three AZs.
|Component||Reference Architecture Notes|
|Domains & DNS||CF Domain Zones and routes in use by the reference architecture include:
|Ops Manager||Deployed on one of the three public subnets and accessible by FQDN or through an optional jumpbox.|
|BOSH Director||Deployed on the infrastructure subnet.|
|Elastic Load Balancers - HTTP, HTTPS, and SSL||Required. Load balancer that handles incoming HTTP, HTTPS, and SSL traffic and forwards them to the Gorouters. Deployed on all three public subnets.|
|Elastic Load Balancers - SSH||Optional. Load balancer that provides SSH access to app containers. Deployed on all three public subnets, one per AZ.|
|Gorouters||Accessed through the HTTP, HTTPS, and SSL Elastic Load Balancers. Deployed on all three Pivotal Application Service (PAS) subnets, one per AZ.|
|Diego Brains||Required. However, the SSH container access functionality is optional and enabled through the SSH Elastic Load Balancers. Deployed on all three PAS subnets, one per AZ.|
|TCP Routers||Optional feature for TCP routing. Deployed on all three PAS subnets, one per AZ.|
|CF Database||Reference architecture uses AWS RDS. Deployed on all three RDS subnets, one per AZ.|
|Storage Buckets||Reference architecture uses 4 S3 buckets: buildpacks, droplets, packages, and resources.|
|Service Tiles||Deployed on all three service subnets, one per AZ.|
|Dynamic Services||Reserved for future use, dynamic services are deployed on their own subnet. Dynamic services are services autoprovisioned by BOSH based on a trigger, such as a request for that service. Pivotal recommends provisioning the multi-tenant dynamic services subnet as a /22 block.|
|Service User & Roles||One IAM role and one IAM user are recommended: the IAM role for Terraform, and the IAM user for Ops Manager and BOSH. Consult the following list:
|EC2 Instance Quota||The default EC2 instance quota on a new AWS subscription only has around 20 EC2 instances, which is not enough to host a multi-AZ deployment. The recommended quota for EC2 instances is 100. AWS requires the instances quota tickets to include Primary Instance Types, which should be t2.micro.|
The following table lists the network objects in this reference architecture.
|Network Object||Notes||Estimated Number|
|External Public IPs||One per deployment, assigned to Ops Manager.||1|
|Virtual Private Network (VPC)||One per deployment. A PCF deployment exists within a single VPC and a single AWS region, but should distribute PCF jobs and instances across 3 AWS AZs to ensure a high degree of availability.||1|
|Subnets||The reference architecture requires the following subnets:
|Route Tables||This reference architecture requires 4 route tables: one for the public subnet, and one each for all 3 private subnets across 3 AZs. Consult the following list:
Note: If an EC2 instance sits on a subnet with an Internet gateway attached as well as a public IP address, it is accessible from the Internet through the public IP address; for example, Ops Manager. PAS needs Internet access due to the access needs of using an S3 bucket as a blobstore.
|Security Groups||The reference architecture requires 5 Security Groups. For more information, see the Terraform Security Group rules script. The following table describes the Security Group ingress rules:
|Load Balancers||PCF on AWS requires the Elastic Load Balancer, which can be configured with multiple listeners to forward HTTP/HTTPS/TCP traffic. Two Elastic Load Balancers are recommended: one to forward the traffic to the Gorouters,
The following table describes the required listeners for each load balancer:
|Jumpbox||Optional. Provides a way of accessing different network components. For example, you can configure it with your own permissions and then set it up to access to Pivotal Network to download tiles. Using a jumpbox is particularly useful in IaaSes where Ops Manager does not have a public IP address. In these cases, you can SSH into Ops Manager or any other component through the jumpbox.||1|
At times, applications on PCF need to access on-premise data. The connection between an AWS VPC and an on-premise datacenter is made through VPN peering. When employing non-VPN peering, there are several points to consider:
Assign routable IP addresses with the following in mind:
- It may not be realistic to request multiple routable /22 address spaces, due to IP exhaustion.
- Using different VPC address spaces can cause snowflakes deployments and present difficulties in automation.
- Only make the load balancer, NAT devices, and Ops Manager routable.
- PCF components can route egress through a NAT instance. As a result, operators do not need to assign routable IP addresses to PCF components.
Inbound traffic from the datacenter should come through an internal load balancer.
Outbound traffic to the datacenter should go through AWS NAT instances.