vSphere Reference Architecture

This topic provides reference architectures for Pivotal Cloud Foundry (PCF) on vSphere. It builds on the common base architectures described in Platform Architecture and Planning.

See Installing PCF on vSphere for additional requirements and installation instructions for PCF on vSphere.

Overview

The vSphere reference architecture for the Pivotal Application Service (PAS) runtime is based on software-defined networking (SDN) infrastructure. vSphere offers NSX-T and NSX-V to support SDN infrastructure.

For PAS deployments, the VMware NSX-T Container Plugin for PCF is required to use the SDN features of NSX-T.

PCF supports the following configurations for PCF on vSphere deployments:

PAS on vSphere with NSX-T

The following sections describe the reference architecture for PAS on vSphere with NSX-T deployments. They also provide requirements and recommendations for deploying PAS on vSphere with NSX-T, such as network, load balancing, and storage capacity requirements and recommendations.

PAS on vSphere with NSX-T supports the following SDN features:

  • Virtualized, encapsulated networks and encapsulated broadcast domains
  • VLAN exhaustion avoidance with the use of virtualized Logical Networks
  • DNAT/SNAT services to create separate, non-routable network spaces for the PAS installation
  • Load balancing services to pass traffic thru (at layer 4) to pools of platform routers (at layer 7)
  • SSL termination at the load balancer at layer 7 with the option to forward on at layer 4 or 7 with unique certificates
  • Virtual, distributed routing and firewalling services native to the hypervisor

Architecture

The following reference architecture diagram describes the architecture for PAS on vSphere with NSX-T deployments.

The diagram shows the archiecture for a PAS on vSphere with NSX-T deployment. For more information about the components and networking demonstrated by the diagram, read the description below this diagram.

View a larger version of this diagram.

As shown in the diagram above, PAS deployments with NSX-T are deployed with three clusters and three availability zones (AZs).

A NSX-T Tier-0 router is on the front end of the PAS deployment. This router is a central logical router into the PAS platform. You can configure static or dynamic routing using BGP from the routed IP backbone through the Tier-0 router with the gateway Edge.

Several Tier-1 routers, such as the router for the PAS and infrastructure subnets, connect to the Tier-0 router.

NSX-T Container Plugin Requirement

For PAS deployments, the VMware NSX-T Container Plugin for PCF is required to enable the SDN features available through NSX-T.

The NSX-T Container Plugin enables a container networking stack and integrates with NSX-T.

Note: To use NSX-T with PAS, the NSX-T Container Plugin must be installed, configured, and deployed at the same time as the PAS tile. To download the NSX-T Container Plugin, see VMware NSX-T Container Plug-in for PCF on Pivotal Network.

Networking

The following sections describe networking requirements and recommendations for PAS on vSphere with NSX-T deployments.

Routable IPs

The Tier-0 router must have routable external IP address space to advertise on the BGP network with its peers. Select a network range for the Tier-0 router with enough space so that the network can be separated into the following two jobs:

  • Routing incoming & outgoing traffic.
  • DNATs and SNATs, load balancer VIPs, and other PCF components.

Note: Compared to NSX-V, NSX-T consumes much more address space for SNATs.

DNS

PAS requires a system domain, app domain, and several wildcard domains.

For more information about DNS requirements for PAS, see the Domain Names section in Platform Planning and Architecture.

Load Balancing

The following are load balancing requirements and recommendations for PAS on vSphere with NSX-T deployments:

  • You must configure NSX-T load balancers for the Gorouters.
    • The domains for the PAS system and apps must resolve to the load balancer VIP.
    • You must assign either a private or a public IP address assigned to the domains for the PAS system and apps.
  • Pivotal recommends that you configure Layer 4 NSX-V load balancers for the Gorouters. With Layer 4 load balancers, traffic passes through the load balancers and SSL is terminated at the Gorouters. This approach reduces overhead processing.

    Note: It is possible to use Layer 7 load balancers and terminate SSL at the load balancers. However, this approach adds additional overhead processing and is not recommended.

  • Any TCP Gorouters and SSH Proxies within the platofrm also require NSX-T load balancers.
  • Layer 4 and Layer 7 NSX-T load balancers are created automatically during app deployment.

Networking, Subnets, and IP Spacing

The following are requirements and recommendations related to networks, subnets, and IP spacing for PAS on vSphere with NSX-T deployments:

  • PAS requires statically-defined networks to host PAS component VMs.
  • The client side of a NSX-T deployment uses a series of non-routable address blocks when using DNAT/SNAT at the Tier-0 interface.
  • The reference architecture for PAS on vSphere with NSX-T deployments uses a pattern in which all networks are calculated on the /24 8-bit network boundary. The network octet is numerically sequential.
  • NSX-T dynamically assigns PAS org networks and adds a Tier-1 router. These org networks are automatically instantiated based on a non-overlapping block of address space. You can configure the block of address space in the NCP Configuration section of the NSX-T tile in Ops Manager. The default is /24. This means that every org in PAS is assigned a new /24 network.

For more information about PAS subnets, see the Required Subnets section in Platform Architecture and Planning Overview.

High Availability

For information about high availability requirements and recommendations for PAS on vSphere, see the High Availability section of Platform Architecture and Planning Overview.

Shared Storage

Shared storage is a requirement for PCF. You can allocate networked storage to the host clusters following one of two common approaches: horizontal or vertical. The approach you follow reflects how your data center arranges its storage and host blocks in its physical layout.

Horizontal Shared Storage

With the horizontal shared storage approach, you grant all hosts access to all datastores and assign a subset to each PCF installation.

For example, with six datastores ds01 through ds06, you grant all nine hosts access to all six datastores. You then provision your first PCF installation to use stores ds01 through ds03 and your second PCF installation to use ds04 through ds06.

Vertical Shared Storage

With the vertical shared storage approach, you grant each cluster its own datastores, creating a cluster-aligned storage strategy. vSphere VSAN is an example of this architecture.

For example, with six datastores ds01 through ds06, you assign datastores ds01 and ds02 to a cluster, ds03 and ds04 to a second cluster, and ds05 and ds06 to a third cluster. You then provision your first PCF installation to use ds01, ds03, and ds05, and your second PCF installation to use ds02, ds04, and ds06.

With this arrangement, all VMs in the same installation and cluster share a dedicated datastore.

Storage Capacity

Pivotal recommends the following storage capacity allocation for production and non-production PAS environments:

  • Production environments: Configure at least 8 TB of data storage. You can configure this as either one 8 TB store or a number of smaller volumes that sum to 8 TB. Frequently-used developments may require significantly more storage to accommodate new code and buildpacks.
  • Non-production environments: Configure 4 to 6 TB of data storage.

Note: PCF does not support using vSphere Storage Clusters with the latest versions of PCF validated for the reference architecture. Datastores should be listed in the vSphere tile by their native name, not the cluster name created by vCenter for the storage cluster.

Note: If a datastore is part of a vSphere Storage Cluster using DRS storage (sDRS), you must disable the s-vMotion feature on any datastores used by PCF. Otherwise, s-vMotion activity can rename independent disks and cause BOSH to malfunction. For more information, see How to Migrate PCF to a New Datastore in vSphere.

For more information about general storage requirements and recommendations for PAS, see the Storage section of Platform Architecture and Planning Overview.

SQL Server

An internal MySQL database is sufficient for use in production environments.

However, an external database provides more control over database management for large environments that require multiple data centers.

For information about configuring system databases on PAS, see the Configure System Databases section of the deploying PAS topic for your IaaS. To find the deploying PAS topic for your IaaS, see Installation Overview.

Security

For information about security requirements and recommendations for PAS deployments, see the Security section of Platform Architecture and Planning Overview.

Blobstore Storage

Pivotal recommends that you use the following blobstore storage for production and non-production PAS environments:

  • Production environments: Use a S3 storage appliance as the blobstore.
  • Non-production environments: Use a NFS/WebDAV blobstore.

Note: For non-production environments, the NFS/WebDAV blobstore can be the primary consumer of storage, as the NFS/WebDAV blobstore must be actively maintained. There will be down time for deployment during events such as storage upgrades or migrations to new disks.

For information about blobstore storage requirements and recommendations, see the Configure File Storage section of the deploying PAS topic for your IaaS. To find the deploying PAS topic for your IaaS, see Installation Overview.

PAS on vSphere with NSX-V

The following sections describe the reference architecture for PAS on vSphere with NSX-V deployments. They also provide requirements and recommendations for deploying PAS on vSphere with NSX-V, such as network, load balancing, and storage capacity requirements and recommendations.

PAS on vSphere with NSX-V enables services provided by NSX on the PAS platform, such as Edge services gateway (ESG), load balancers, firewall services, and NAT/SNAT services.

Architecture

The following reference architecture diagram describes the architecture for PAS on vSphere with NSX-V deployments.

The diagram shows the archiecture for a PAS on vSphere with NSX-V deployment. For more information about the components and networking demonstrated by the diagram, read the description below this diagram.

View a larger version of this diagram.

As shown in the diagram above, PAS deployments with NSX-V are deployed with three clusters and three availability zones (AZs).

PAS deployments with NSX-V also include a NSX-V Edge router on the front end. You can install the NSX-V Edge router as an edge services gateway (ESG) or as a distributed logical router (DLR).

The Edge router is a central logical router into the PAS platform. You can configure VLAN routing from the routed backbone into NSX-V through the Edge router.

Compared to NSX-T architecture, NSX-V architecture does not use Tier-1 routers to connect the central router to the various subnets for the PAS deployment.

For more information about using ESG on vSphere, see Using Edge Services Gateway on VMware NSX.

Networking

The following sections describe networking requirements and recommendations for PAS on vSphere with NSX-V deployments.

Routable IPs

You must assign routable external IPs on the server side, such as routable IPs for NATs and load balancers, to the Edge router.

DNS

PAS requires a system domain, app domain, and several wildcard domains.

For more information about DNS requirements for PAS, see the Domain Names section in Platform Planning and Architecture.

Load Balancing

The following are load balancing requirements and recommendations for PAS on vSphere with NSX-V deployments:

  • NSX-V includes an Edge router. The Edge router supports ESG. ESG provides load balancing and is configured to route to the PAS platform.
  • Pivotal recommends that you configure Layer 4 NSX-V load balancers for the Gorouters. With Layer 4 load balancers, traffic passes through the load balancers and SSL is terminated at the Gorouters. This approach reduces overhead processing.

    Note: It is possible to use Layer 7 load balancers and terminate SSL at the load balancers. However, this approach adds additional overhead processing and is not recommended.

  • The domains for the PAS system and apps must resolve to the load balancer. You must assign either a private or a public IP address assigned to the domains for the PAS system and apps.
  • Any TCP routers and SSH Proxies also require NSX-V load balancers.
  • Pivotal recommends that you configure external load balancers in front of the Edge router. For example, you can configure an F5 external load balancer.

Networks, Subnets, and IP Spacing

For information about network, subnet, and IP space planning requirements and recommendations, see the Required Subnets section in Platform Architecture and Planning Overview.

High Availability

For information about high availability requirements and recommendations for PAS on vSphere, see the High Availability section of Platform Architecture and Planning Overview.

Shared Storage

Shared storage is a requirement for PCF. You can allocate networked storage to the host clusters following one of two common approaches: horizontal or vertical. The approach you follow reflects how your data center arranges its storage and host blocks in its physical layout.

For information about horizontal and vertical shared storage, see Shared Storage.

Storage Capacity

Pivotal recommends the following storage capacity allocation for production and non-production PAS environments:

  • Production environments: Configure at least 8 TB of data storage. You can configure this as either one 8 TB store or a number of smaller volumes that sum to 8 TB. Frequently-used developments may require significantly more storage to accommodate new code and buildpacks.
  • Non-production environments: Configure 4 to 6 TB of data storage.

Note: PCF does not support using vSphere Storage Clusters with the latest versions of PCF validated for the reference architecture. Datastores should be listed in the vSphere tile by their native name, not the cluster name created by vCenter for the storage cluster.

Note: If a datastore is part of a vSphere Storage Cluster using DRS storage (sDRS), you must disable the s-vMotion feature on any datastores used by PCF. Otherwise, s-vMotion activity can rename independent disks and cause BOSH to malfunction. For more information, see How to Migrate PCF to a New Datastore in vSphere.

For more information about general storage requirements and recommendations for PAS, see the Storage section of Platform Architecture and Planning Overview.

SQL Server

An internal MySQL database is sufficient for use in production environments.

However, an external database provides more control over database management for large environments that require multiple data centers.

For information about configuring system databases on PAS, see the Configure System Databases section of the deploying PAS topic for your IaaS. To find the deploying PAS topic for your IaaS, see Installation Overview.

Security

For information about security requirements and recommendations for PAS on vSphere deployments, see the Security section in Platform Architecture and Planning Overview.

Blobstore Storage

Pivotal recommends that you use the following blobstore storage for production and non-production PAS environments:

  • Production environments: Use a S3 storage appliance as the blobstore.
  • Non-production environments: Use a NFS/WebDAV blobstore.

Note: For non-production environments, the NFS/WebDAV blobstore can be the primary consumer of storage, as the NFS/WebDAV blobstore must be actively maintained. There will be down time for deployment during events such as storage upgrades or migrations to new disks.

For information about blobstore storage requirements and recommendations, see the Configure File Storage section of the deploying PAS topic for your IaaS. To find the deploying PAS topic for your IaaS, see Installation Overview.