AWS Reference Architecture

Page last updated:

This topic describes a reference architecture for Pivotal Platform, including Pivotal Application Service (PAS) and Enterprise Pivotal Container Service (Enterprise PKS), on Amazon Web Services (AWS). It builds on the common base architectures described in Platform Architecture and Planning Overview.

For general requirements for running Pivotal Platform and specific requirements for running Pivotal Platform on AWS, see Pivotal Platform on AWS Requirements.

PAS

This section describes a reference architecture for a PAS installation on AWS.

Diagram

The diagram below illustrates the reference architecture for PAS on AWS.

This diagram illustrates PAS on AWS. For specific details about the components and communication represented by the diagram, see the base and AWS reference architectures.

View a larger version of this diagram.

Networking

These sections provide guidance about networking resources.

DNS

You can use AWS Route 53 for DNS resolution to host your PAS domains.

Load Balancing

AWS offer Application Load Balancers (ALBs) and Network Load Balancers (NLBs). VMware recommends using NLBs because of the simplicity and security of leaving all encrypted traffic in place until it reaches the Gorouters. If you require Amazon Certificate Management or more complex layer 7 routing rules, you can front or replace the NLB with an ALB.

Networks, Subnets, and IP Spacing Planning

When planning networks, subnets, and IP spacing, some things to consider are:

  • In AWS, each availability zone (AZ) requires its own subnet of which the first five IPs are reserved.

  • Your organization might not have the IP space necessary to deploy PAS in a consistent manner. To alleviate this, VMware recommends using the 100.64 address space to deploy a NAT instance and router so that the Pivotal Platform deployment can use few IP addresses.

  • If you want the front end of PAS to be accessible from your corporate network or the services running on PAS to be able to access corporate resources, you must either provide routable IPs to your VPC or use NAT.

  • If you want a VPC that is only public-facing, no special consideration is necessary for the use of IPs.

  • To understand the action required depending on the type of traffic you want to allow for your Pivotal Platform deployment, see the table below:

    If you want… then…
    Internet ingress Create the load balancers with Elastic IPs.
    Internet egress Create the NAT Gateway toward the Internet.
    Corporate Ingress Create load balancers on your corporate subnet.
    Corporate Egress (For accessing corporate resources) Create the NAT instance on your corporate subnet or if business requirements dictate, make the entire VPC corporate-routable.

  • If you plan to install Pivotal Platform Service Broker for AWS, you might want to create another set of networks in addition to those outlined in the base reference architecture. This set of networks would be used for RDS and other AWS managed instances.

Alternative Network Layouts

These options are available for choosing your network topology:

Type of Traffic Options
Internet Ingress
  • Allow Internet ingress by associating load balancers with elastic IPs.
  • Do not allow Internet ingress.
Internet Egress
  • Allow PAS to access the Internet through a NAT gateway.
  • Do not allow PAS to access the Internet. Use only a corporate network.
Peering
  • Isolated. This is normally internet-only facing.
  • Peered with corporate or other VPCs through Transit Gateway.
  • Peered directly with other VPCs and corporate connections.
Corporate Peering
  • SNAT for corporate resources. This requires three corporate IPs.
  • Routable IP Addresses for VPC using /20 or a larger CIDR block necessary.

RDS

Use a single RDS instance for BOSH and PAS. This instance requires several databases. For more information, see Step 3: Director Config Page in Configuring BOSH Director on AWS and External System Database Configuration in Configuring PAS.

Blobstore Storage Accounts

Pivotal Operations Manager requires a bucket for the BOSH blobstore.

PAS requires these buckets:

  • Buildpacks
  • Droplets
  • Packages
  • Resources

These buckets require an associated role for read/write access.

Identity Management

For identity management, use Instance Profiles whenever possible. For example, the AWS Config Page of the BOSH Director tile provides a Use AWS Instance Profile option. For more information, see Step 2: AWS Config Page in Configuring BOSH Director on AWS.

Enterprise PKS

These sections describe a reference architecture for a Enterprise PKS installation on AWS.

Diagram

The diagram below illustrates the reference architecture for Enterprise PKS on AWS.

This diagram illustrates Enterprise PKS on AWS. For specific details about the components and communication represented by the diagram, see the base and AWS reference architectures.

View a larger version of this diagram.

Networking

These sections provide guidance about networking resources.

DNS

You can use AWS Route 53 for DNS resolution to host your Enterprise PKS domains.

Load Balancing

You can configure a front end load balancer to enable public access to the Enterprise PKS API VM.

For each Enterprise PKS cluster, VMware recommends creating a load balancer for the corresponding master node or nodes. For more information, see Creating and Configuring an AWS Load Balancer for Enterprise Enterprise PKS Clusters.

For deployed workloads, Kubernetes can automatically create load balancers when applying the service definition to the app instances. For more information, see Deploying and Exposing Basic Workloads.

Networks, Subnets, and IP Spacing Planning

When planning networks, subnets, and IP spacing, some things to consider are:

  • In AWS, each AZ requires its own subnet of which the first five IPs are reserved.

  • If you want the front end of Enterprise PKS to be accessible from your corporate network or the services running on Enterprise PKS to be able to access corporate resources, you must either provide routable IPs to your VPC or use NAT.

  • Create the Enterprise PKS subnets that host the Kubernetes cluster VMs with enough IP address capacity to accommodate the number of VMs and clusters you expect to deploy. The reference architecture diagram above shows a /24 CIDR for the Enterprise PKS subnets, which allows for about 250 VMs per AZ. If you expect a larger number of clusters for your organization, consider defining larger subnets, such as /20, to make room for future growth.

Alternative Network Layouts

These options are available for choosing your network topology:

Type of Traffic Options
Internet Ingress
  • Allow Internet ingress by associating load balancers with elastic IPs.
  • Do not allow Internet ingress.
Internet Egress
  • Allow nodes and pods to access the Internet through a NAT gateway.
  • Do not allow nodes or pods to access the Internet. Use only a corporate network.
Peering
  • Isolated. This is normally internet-only facing.
  • Peered with corporate or other VPCs through Transit Gateway.
  • Peered directly with other VPCs and corporate connections.
Corporate Peering
  • SNAT for corporate resources. This requires three corporate IPs.
  • Routable IP Addresses for VPC using /20 or a larger CIDR block necessary.

RDS

Use a single RDS instance for BOSH. This instance requires a database. For more information, see Step 3: Director Config Page in Configuring BOSH Director on AWS.

Identity Management

To create and manage Kubernetes master and worker nodes, use Instance Profiles. For more information, see Kubernetes Cloud Provider in Installing Enterprise Enterprise PKS on AWS.

For information about installing the required AWS constructs for Enterprise PKS, see Installing Pivotal Platform on AWS.