Upgrade Load Example: Pivotal Web Services

Page last updated:

This topic provides sample performance measurements of a Pivotal Web Services (PWS) installation undergoing the workload associated with an upgrade.

To obtain these measurements, the PWS team repaved its production deployment. The repave process simulates system load that would be incurred when performing a rolling upgrade of Diego Cells.

Use the measurements and configuration values published in this document as guidance when ensuring you have adequate file storage hardware prior to a platform upgrade.

For more information about the impact of upgrade on file storage performance, see the Configure File Storage section of the Configuring TAS for VMs for Upgrades topic.

Platform Configuration

The following table details the starting parameters and configuration of PWS.

Configuration Value How to Locate
IaaS Amazon Web Services See your BOSH Director configuration or BOSH deployment manifest.
File Storage AWS EBS (External with some elastic capacity) See your Tanzu Application Service for VMs (TAS for VMs) configuration or BOSH deployment manifest.
Version of Cloud Foundry (CF) v252 See your BOSH Director and TAS for VMs configuration or BOSH deployment manifest.
Number of Diego Cells 218 To view the number of Diego Cell instances currently running in your deployment, see the Resource Config pane of your TAS for VMs tile or consult your Diego deployment manifest.
Maximum number of started containers 250 For more information, see the Set a Maximum Number of Starting Containers section of the Configuring TAS for VMs for Upgrades topic.
max_in_flight Configuration for Diego Cells 6 To retrieve the existing max_in_flight value for the Diego Cell job in BOSH Director, use the Ops Manager API. If you are running open source CF, consult your BOSH deployment manifest.
Number of Availability Zones (AZs) 2 See your TAS for VMs or BOSH deployment AZ configuration.
Number of app instances 16231 datadog.nozzle.bbs.LRPsRunning
Number of App Security Groups (ASGs) 43 As an admin user, run cf security-groups. For more information, see App Security Groups.

System Performance Measurements During Diego Cell Repave

This table presents performance measurements taken during the Diego Cell repave.

Note: These measurements indicate the peak cumulative values of the entire system: 250 Diego Cells, ~15,000 app instances, and 2 AZs.

Use these measurements as a baseline for expected system load during Diego Cell upgrade.

Measurement Value Metric Used
Diego Cell CPU consumption 36% bosh.healthmonitor.system.cpu.user
Diego Cell memory consumption ~50% bosh.healthmonitor.system.mem.percent
Diego Cell I/O consumption (read) during normal operations 43 read I/O operations per second aws.ebs.volume_read_ops
Diego Cell I/O consumption (read) during upgrade 1,943 read I/O operations per second aws.ebs.volume_read_ops
Diego Cell I/O consumption (write) during normal operations 2,166 write I/O operations per second aws.ebs.volume_write_ops
Diego Cell I/O consumption (write) during upgrade 21,000 write I/O operations per second aws.ebs.volume_write_ops
Diego Cell network consumption (network out) during normal operations ~1.25 GB per minute aws.ec2.network_out
Diego Cell network consumption (network out) during Upgrade ~1.25 GB per minute (no significant change) aws.ec2.network_out
Diego Cell network consumption (network in) during normal operations 2.11 GB per minute aws.ec2.network_in
Diego Cell network consumption (network in) during upgrade 16.75 GB per minute aws.ec2.network_in

Sample Performance Graphs

These DataDog graphs represent a timeline visualization of read and write operations during the repave event.

Read I/O Operations

The read I/O operations sample was taken over 115 VMs and represent the number of read operations over 300 seconds for a single Diego Cell.

Pws read ops upgrade

Write I/O Operations

The write I/O operations sample was taken over 115 VMs and represent the number of read I/O operations over 300 seconds for a single Diego Cell.

Pws write ops upgrade

Summary

During the repave process, 250 Diego Cells were updated. The repave process took 6 hours overall, or about 3 hours for each Availability Zone.