Azure Reference Architecture
Page last updated:
This guide presents a reference architecture for Pivotal Platform on Azure. It builds on the common base architectures described in Platform Architecture and Planning.
See Installing Pivotal Platform on Azure for additional requirements and installation instructions for running Pivotal Platform on Azure.
Azure Virtual Data Center (VDC) uses a hub and spoke model for extending on-premises data centers. For more information about VDC, see the Azure Virtual Datacenter e-book.
When installing Pivotal Platform, you can use this model to place resources in resource groups. For example, a Pivotal Platform installation contains the following:
- A central IT resource group included in Azure VDC setup. This is where you define firewall, access control, and security policies.
- A Network resource group for all network resources.
- PAS/PKS resource groups for all PAS/PKS resources.
The following architecture diagram illustrates Pivotal Platform on Azure using the hub and spoke model described above:
The sections below provide guidance about the resources you use for running Pivotal Platform on Azure.
The following sections provide guidance about networking resources.
The following are general requirements and recommendations related to networking when running Pivotal Platform on Azure:
- You must enable Virtual Network peering between hub and spoke resource groups. See Virtual network peering.
- You must have separate subscriptions for each resource group. This includes the following:
- The central network resource group. This is the hub.
- Each workspace where workloads run. These are the spokes.
- Use ExpressRoutes for a dedicated connection from an on-premises datacenter to VDC. See ExpressRoute Documentation. As a back up for when ExpressRoutes goes down, you can use Azure VPN Gateways, whichi pass encrypted data into VDC through the Internet.
- Use a central firewall in the hub resource group and a network appliance to prevent data infiltration and exfiltration.
- You can place network resources, such as DNS and NTP servers, either on-premises or on Azure Cloud.
For your Pivotal Platform domains, use a sub-zone of an Azure DNS zone.
Azure DNS supports DNS delegation, which allows sub-level domains to be hosted within Azure. For example, if your company’s domain is
example.com, your Pivotal Platform zone in Azure DNS would be
Azure DNS does not support recursion. To properly configure Azure DNS, do the following:
- Create a
NSrecord with your registrar that points to the four name servers supplied by your Azure DNS Zone configuration.
- Create the required wildcard
Arecords for the Pivotal Platform app and system domains, as well as any other records desired for your Pivotal Platform deployments.
You do not need to make any configuration changes in Pivotal Platform to support Azure DNS.
Use Standard Azure Load Balancers (ALBs). If you want Internet ingress, you must create the ALBs with elastic IPs.
During installation, you configure ALBs for PAS Gorouters. See Create Load Balancers in Deploying Ops Manager on Azure Manually. When applicable, TCP Routers and SSH Proxies also require load balancers.
The PAS system and app domains must resolve to your ALB and have either a private or a public IP address assigned. You set these domains in the Domains pane of the PAS tile.
Pivotal Platform requires the networks defined in the base reference architectures for PAS and PKS. See the Networks sections in Platform Architecture and Planning Overview.
If shared network resources do not exist in an Azure subscription, you can use a single resource group to deploy Pivotal Platform and define network constructs.
Note: As of Pivotal Platform v2.5.3, Pivotal Platform on Azure supports both availability zones (AZ) and Availability Sets.
Use AZs instead of Availability Sets in all Azure regions where they are available. AZs solve the placement group pinning issue associated with Availability Sets and allow greater control when defining high availability configuration when using multiple Azure services.
See the following table for additional guidance depending on your use case:
|New foundation||If you do not have strict requirements, select AZ enabled regions. Examples of strict requirements include existing ExpressRoute connections from non-AZ enabled regions to customer data centers and app-specific latency requirements.|
|Existing foundation in AZ enabled region||Migrate to a new Pivotal Platform v2.5.3+ foundation so that you can use AZs. These existing foundations, especially those using older families of VMs such as
|Existing foundation not in AZ enabled region||Evaluate your requirements to determine the cost/benefit of migrating to an AZ enabled region.|
For more information about availability modes in Azure, see the Availability Zones and Availability Sets sections below, as well as Manage the availability of Windows virtual machines in Azure in the Microsoft Azure documentation.
AZs are an Azure primitive that supports high availability within a region. They are physically separate data centers in the same region, each with of separate power, cooling, and network.
The following diagram illustrates PAS deployed on Azure using AZs:
AZs have the following advantages:
- VMs and other zonal services deployed to independent zones provide greater physical separation than Fault Domains and avoid the placement group pinning issue associated with availability sets.
- Zonal services are effectively in separate Update Domains as updates are rolled out to one zone at a time.
- AZs are visible and configurable. Two zonal services placed in the same logical zone are also placed in the same physical zone. This allows services of different types to take advantage of the highly available qualities a zone provides.
Availability Sets were the first Azure primitive to support high availability within a region.
Warning: Availability Sets are implemented at the hardware rack level, which can cause issues when deploying Pivotal Platform on Azure. See Placement Group Pinning Issues.
They have the following properties:
- They are a logical concept that ensures multiple VMs in the same set are spread evenly across 3 Fault Domains and either 5 or 20 Update Domains.
- When a hardware issue occurs, VMs on independent Fault Domains are guaranteed not to share power supply or networking and therefore maintain availability of a subset of VMs in the set.
- When updates are rolled out to a Cluster, VMs in independent Update Domains are not updated at the same time, ensuring downtime incurred by an update does not affect the availability of the set.
In Pivotal Platform on Azure in AS HA mode, VMs for a particular job are pinned to a placement group determined when the first VM in the Availability Set is provisioned. This is because BOSH places VMs for each job within the same Availability Set.
Cluster pinning causes the following:
Issues migrating jobs from one family of VMs to another, backed by different hardware, such as
Issues scaling up if there is no capacity remaining in the placement group
For storage, Pivotal recommends the following:
- For the storage account type, use the premium performance tier.
- Configure storage accounts for Zone Redundant Storage (ZRS) in AZ-enabled Regions.
Note: In non-AZ enabled regions, locally Redundant Storage (LRS) is sufficient.
- Configure VMs to use managed disks instead of manually-managed VHDs on storage accounts. Managed disks are an Azure feature that handles the correct distribution of VHDs over storage accounts to maintain high IOPS as well as high availability for the VMs.
- Use 5 storage accounts. Pivotal Platform on Azure requires 5 storage accounts: 1 BOSH, 1 Pivotal Operations Manager, and 3 PAS storage accounts. Each account comes with a set amount of disk. Azure storage accounts have an IOPs limit of about 20k per account, which generally corresponds to a BOSH job/VM limit of 20 VMs each.
The Internal MySQL database provided by PAS is sufficient for production use.
Use Azure Blob Storage as the external file storage option for Pivotal Platform. This provides redundancy for high-availability deployments of Pivotal Platform and unlimited scaling. Azure Blob Storage provides fully-redundant hot, cold, or archival storage in either local, regional, or global offerings.
Ops Manager requires a bucket for the BOSH blobstore.
PAS requires the following buckets:
These buckets require an associated role for read/write access.
Use unique managed identities to deploy Ops Manager, BOSH Director, Microsoft Azure Service Broker, and any other tiles that require independent credentials. Ensure that each managed identity is scoped to the least privilege necessary to operate.
Azure uses managed identities to handle service authorization. See What is managed identities for Azure resources?.