Azure Reference Architecture

This guide presents a reference architecture for Pivotal Cloud Foundry (PCF) on Azure.

Azure does not provide resources in a way that translates directly to PCF availability zones. Instead, Azure provides high availability through fault domains and availability sets.

All reference architectures described in this topic are validated for production-grade PCF deployments using fault domains and availability sets that include multiple job instances.

See Azure on PCF Requirements for general requirements for running PCF and specific requirements for running PCF on Azure.

PCF Reference Architectures

A PCF reference architecture describes a proven approach for deploying PCF on a specific IaaS, such as Azure, that meets the following requirements:

  • Secure
  • Publicly-accessible
  • Includes common PCF-managed services such as MySQL, RabbitMQ, and Spring Cloud Services
  • Can host at least 100 app instances, or far more

Pivotal provides reference architectures to help you determine the best configuration for your PCF deployment.

Base Azure Reference Architecture

The following diagram provides an overview of a reference architecture deployment of PCF on Azure.

Azure overview arch

View a larger version of this diagram.

Base Reference Architecture Components

The following table lists the components that are part of a base reference architecture deployment on Azure using a single resource group.

Component Reference Architecture Notes
Domains and DNS CF Domain Zones and routes in use by the reference architecture include:
  • domains for *.apps and *.system (required),
  • a route for Ops Manager (required),
  • a route for Doppler (required),
  • a route for Loggregator (required),
  • a route for SSH access to app containers (optional),
  • and a route for TCP routing to apps (optional).
Ops Manager Deployed on the management subnet and accessible by FQDN or through an optional jumpbox.
BOSH Deployed on the management subnet.
Azure Load Balancer - API and Apps Required. Load balancer that handles incoming API and apps requests and forwards them to the Gorouters.
Azure Load Balancer - ssh-proxy Optional. Load balancer that provides SSH access to app containers.
Azure Load Balancer - tcp-router Optional. Load balancer that handles TCP routing requests for apps.
Azure Load Balancer - MySQL Required to provide high availability for MySQL backend to Pivotal Application Service (PAS).
Gorouters Accessed through the API and Apps load balancer. Deployed on the PAS subnet, one job per Azure availability set.
Diego Brains Required. However, the SSH container access functionality is optional and enabled through the SSH Proxy load balancer. Deployed on the PAS subnet, one job per Azure availability set.
TCP Routers Optional feature for TCP routing. Deployed on the PAS subnet, one job per availability zone.
MySQL Reference architecture uses internal MySQL provided with PCF. Deployed on the PAS subnet, one job per Azure availability set.
PAS Required. Deployed on the PAS subnet, one job per Azure availability set.
Storage Accounts PCF on Azure requires 5 standard storage accounts: BOSH, Ops Manager, and three PAS storage accounts. Each account comes with a set amount of disk. Reference architecture recommends using 5 storage accounts because Azure Storage Accounts have an IOPs limit of approximately 20k per account, which generally relates to a BOSH JOB/VM limit of approximately 20 VMs each.
Service Tiles Deployed on the Services subnet. Each service tile is deployed to an availability set.

Alternative Network Layouts for Azure

This section describes the possible network layouts for PCF deployments as covered by the reference architecture of PCF on Azure.

At a high level, there are currently two possible ways of deploying PCF as described by the reference architecture:

  1. Single resource group, or
  2. Multiple resource groups.

The first scenario is outlined in Installing PCF on Azure. It models a single PCF deployment in a single Azure Resource Group.

If you require multiple resource groups, refer to the Multiple Resource Group deployment section of this topic.

Network Layout

This diagram illustrates the network topology of the base reference architecture for PCF on Azure. In this deployment, you expose only a minimal number of public IP addresses and deploy only one resource group.

Azure net topology base

View a larger version of this diagram.

Network Objects

The following table lists the network objects in PCF on Azure reference architecture.

Network Object Notes Estimated Number
External Public IP addresses Use
  1. global IP address for apps and system access
  2. Ops Manager or optional jumpbox.
Optionally, you can use a public IP address for the ssh-proxy and tcp-router load balancers.
Virtual Network One per deployment. Azure virtual network objects allow multiple subnets with multiple CIDRs, so a typical deployment of PCF will likely only ever require one Azure Virtual Network object. 1
Subnets Separate subnets for
  1. management (Ops Manager, BOSH Director, Jumpbox),
  2. PAS,
  3. and services.
Using separate subnets allows you to configure different firewall rules due to your needs.
Routes Routes are typically created by Azure dynamically when subnets are created, but you may need to create additional routes to force outbound communication to dedicated SNAT nodes. These objects are required to deploy PCF without public IP addresses. 3+
Firewall Rules Azure firewall rules are collected into a Network Security Group (NSG) and bound to a Virtual Network object and can be created to use IP ranges, subnets, or instance tags to match for source and destination fields in a rule. One NSG can be used for all firewall rules. 12
Load Balancers Used to handle requests to Gorouters and infrastructure components. Azure uses 1 or more load balancers. The API and Apps load balancer is required. The TCP Router load balancer used for TCP routing feature and the SSH load balancer that allows SSH access to Diego apps are both optional. In addition, you can use a MySQL load balancer to provide high availability to MySQL. This is also optional. 1-4
Jumpbox Optional. Provides a way of accessing different network components. For example, you can configure it with your own permissions and then set it up to access to Pivotal Network to download tiles. Using a jumpbox is particularly useful in IaaSes where Ops Manager does not have a public IP address. In these cases, you can SSH into Ops Manager or any other component through the jumpbox. 1

Multiple Resource Group Deployment

This diagram illustrates the case where you want to use additional resource groups in your PCF deployment on Azure.

Shared network resources may already exist in an Azure subscription. In this type of deployment, using multiple resource groups allows you to reuse existing resources instead of provisioning new ones.

To use multiple resource groups, you need to provide the BOSH Service Principal with access to the existing network resources.

Azure multi resgroup

View a larger version of this diagram.

Multiple Resource Groups Deployment Notes

To deploy PCF on Azure with multiple resource groups, you can define custom roles to grant resource group access to your BOSH Service Principal. For example, you might develop the following:

  • Dedicated Network Resource Group, limits BOSH Service Principal so that it does not have admin access to network objects.

  • Custom Role for BOSH Service Principal, applied to Network Resource Group, limits the BOSH Service Principal to minimum read-only access.

    az role definition create --role-definition
    { \
    "Name": "PCF Network Read Only", \
    "IsCustom": true, \
    "Description": "MVP PCF Read Network Resgroup", \
    "Actions": [ \
    "Microsoft.Network/virtualNetworks/read", \
    "Microsoft.Network/virtualNetworks/subnets/read", \
    "Microsoft.Network/virtualNetworks/subnets/join/action", \
    "Microsoft.Network/networkSecurityGroups/read", \
    "Microsoft.Network/networkSecurityGroups/join/action", \
    "Microsoft.Network/loadBalancers/read", \
    "Microsoft.Network/publicIPAddresses/read", \
    "Microsoft.Network/publicIPAddresses/join/action" \
    ], \
    "NotActions": [], \
    "AssignableScopes": ["/subscriptions/[YOUR_SUBSCRIPTION_ID]"] \
    The actions prefixed with Microsoft.Network/publicIPAddresses are only required if using IP addresses.

  • Custom Role for BOSH Service Principal, applied to Subscription, allowing the Operator to deploy PCF components

    az role definition create --role-definition
    { \
    "Name": "PCF Deploy Min Perms", \
    "IsCustom": true, \
    "Description": "MVP PCF Terraform Perms", \
    "Actions": [ \
     "Microsoft.Compute/register/action" \
    ], \
    "NotActions": [], \
    "AssignableScopes": ["/subscriptions/[YOUR_SUBSCRIPTION_ID]"] \

  • Custom roles can be assigned to service principals with az role assignment create --role [ROLE] --assignee [SERVICE_PRINCIPAL_ID].

Multiple Resource Group and Multiple Load Balancer Deployment

This diagram illustrates the case where you want to deploy multiple load balancers with your multiple resource group deployment.

The key design points of this deployment are the following:

  • Two Azure load balancer (ALBs) to the Gorouter exist. The first ALB is for API access, which utilizes a private IP address. The system domain should resolve to this ALB. The second ALB is for application access, which can either use a public or private IP address. The apps domain should resolve to this ALB.

  • Azure does not allow the Gorouters to be members of more than one ALB member pool for the same ports, for example 80 and 443. This restriction requires an additional reverse proxy to front the Gorouters to allow hem to expose traffic on these ports for the system domain.

Azure multi resgroup multi lb

View a larger version of this diagram.

Log Analytics Integration

Azure Log Analytics is a service that helps you collect and analyze data generated by resources in your cloud and on-premises environments. The Microsoft Azure Log Analytics Nozzle for PCF receives logs and metrics from the Loggregator Firehose, filters and resolves the events, and then sends them to Log Analytics, where they appear on unified dashboards and can be correlated with and alerted on other Azure resources.

DNS Delegation

Azure DNS supports DNS delegation, allowing for sub-level domains to be hosted within Azure. This functionality is fully supported within PCF.

Pivotal recommends that use a sub-zone for your PCF deployment. For example, if your company’s domain is, your PCF zone in Azure DNS would be

As Azure DNS does not support recursion, in order to properly configure Azure DNS, create an NS record with your registrar which points to the four name servers supplied by your Azure DNS Zone configuration. Once your NS records have been created, you can then create the required wildcard A records for the PCF application and system domains, as well as any other records desired for your PCF deployments.

You do not need to make any configuration changes in PCF to support Azure DNS.

Azure Blob Storage

Due to limitations in PCF, it is not possible to support a high-availability deployment of the backing NFS store needed for droplets, buildpacks, and other resources. Azure Blob Storage provides fully-redundant holt, cold, or archival storage in either local, regional, or global offerings. It is recommended to use Azure Blob Storage as the external File Storage to provide unlimited scaling and redundancy for high-availability deployments of PCF.

To enable Azure Blob Storage, do the following:

  1. Create a Storage Account with the level of redundancy you require. The storage account does not need to be in the same Gesource Group as PCF.

  2. Create Containers for the buildpacks, droplets, packages, and resources required by PCF.

  3. Open Ops Manager.

  4. Select Pivotal Application Services.

  5. Click the Settings Tab.

  6. Select File Storage.

  7. Click External Azure Account.

  8. Enter the Storage Account details.

  9. In the Ops Manager main page, apply the changes when you are ready to reconfigure.

Note: There is no direct path to migrate previous objects to Azure Blob Storage. Contact Pivotal Support if you need assistance with this migration.

Load Balancer Migrations

The Azure Load Balancer has two SKUs: Standard and Basic. The Standard SKU Load Balancer is a new load balancer for all TCP and UDP applications with an expanded and more granular feature set over the Basic Load Balancer. Azure’s Standard SKU Load Balancer lets you scale your applications and create high availability in environments ranging from small scale deployments to large and complex multi-zone architectures. While the Basic SKU Load Balancer works within the scope of an availability set, a Standard Load Balancer covers an entire virtual network. Pivotal recommends using the Standard SKU Load Balancer instead of the Basic SKU Load Balancer. If you currently use the Basic SKU Load Balancer and wish to migrate to the the Standard SKU, please see the Migrate Basic SKU Load Balancer to Standard SKU Load Balancer documentation on GitHub.

Create a pull request or raise an issue on the source for this page in GitHub