Upgrade Checklist for PCF v2.1

This topic serves as a checklist for upgrading Pivotal Cloud Foundry (PCF) from v2.0 to v2.1.

Tile Compatibility

Before you upgrade to PCF v2.1, please check whether PCF v2.1 supports the service tile versions that you currently have deployed.

To check PCF version support for any service tile, from Pivotal or a Pivotal partner:

  • Navigate to the tile’s download page on Pivotal Network.
  • Select the tile version in the Releases dropdown.
  • See the Depends On section under Release Details. For more information, refer to the tile’s release notes.

If the currently-deployed version of a tile is not compatible with PCF v2.1, you must upgrade the tile before you upgrade PCF. You do not need to upgrade tiles that are compatible with both PCF v2.0 and v2.1.

The Product Compatibility Matrix provides an overview of which PCF versions support which versions of the most popular service tiles from Pivotal.

When you have completed your upgrade please take some time to complete this survey.

Environment Details

Pivotal provides the empty table below as a model to print out or adapt for recording and tracking the tile versions that you have deployed in all of your environments.

Sandbox Non-Prod Prod Other…
Pivotal Cloud Foundry Ops Manager
Pivotal Application Service (PAS)
Pivotal Cloud Foundry Services MySQL v2
Redis
RabbitMQ
Single Sign On (SSO)
Spring Cloud Services
Concourse
Pivotal Cloud Foundry Partner Services New Relic

PAS v2.1 Known Issues in the Pivotal Knowledge Base

The following Pivotal Knowledge Base (KB) articles describe known issues for PCF v2.1:

PCF v2.1 Breaking Changes

PCF v2.1 has the following breaking changes. For more information, see PCF v2.1 Breaking Changes in the PCF Release Notes.

Breaking Change Details
Ops Manager removed IP address management

Prior to v2.1, Ops Manager allocated and assigned IP addresses manually in the deployment manifest. This caused deployment failures when Ops Manager and the BOSH Director had mismatches on their IP address assignments.

Ops Manager now delegates IP address management to the BOSH Director, which gives operators more flexibility and prevents deployment failures. For 2.1 compatibility, developers must change any service or app code that queries Ops Manager for IP addresses so that it instead retrieves the values from BOSH.

Removal of Automated Backup Configuration for Internal MySQL

The PAS tile Internal MySQL pane no longer includes the Automated Backups Configuration field. Pivotal recommends that operators use BOSH Backup and Restore to back up Internal MySQL components.

See Restore from PAS Automated Database Backup is Not Supported in 1.11 and later in the KB.

Spring Cloud Services no longer stores binding credentials in environment variables.

Spring Cloud Services (SCS) 1.5.0 and later uses CredHub to store binding credentials, rather than saving them in environment variables.

App developers must change any code that reads binding credentials from the environment so that it retrieves the values from CredHub instead. See the SCS 1.5.0 Release Notes for details.

When the Gorouter uses TLS to communicate with apps, it keeps app connections alive to prevent the performance impact from running TLS. By default, each router instance maintains a max of 49 thousand open connections. A checkbox is available to disable this feature in the Ops Manager Director tile Networking pane, under Enable Keepalive Connections for Router.
With the VMware NSX-T tile installed, updating a PCF deployment by clicking Apply Changes causes the BOSH agent to become unresponsive on all Diego cells. To fix this problem, run the BOSH CLI command bosh cck (“cloud check”) after the redeployment completes. This command recreates Diego cell VMs, rebooting their BOSH agents and restoring their responsiveness.

Before Upgrade

Step Note/Reason Product or Component
Upgrade all service tiles to versions compatible with PCF v2.1. Review the Compatibility Matrix and tile documentation to check version compatibility. Service Tiles

Back up your PCF deployment. See Backing Up and Restoring Pivotal Cloud Foundry and Disaster Recovery in Pivotal Cloud Foundry

Pivotal recommends backing up your installation settings before upgrading or otherwise changing your PCF deployment.

PCF
Review Upgrading Pivotal Cloud Foundry in the PCF v2.1 documentation and complete all relevant actions

Review all of the upgrade notes.

  • Ensure that lifecycle errands are enabled for your installed tiles.
  • Ensure that your persistent disk usage is < 50%.
  • Ensure that you know your decryption passphrase and have it stored safely.
PCF
Review the Pivotal Application Service v2.1 Release Notes. See the PAS release notes for details. PAS
Review the PCF Ops Manager Release v2.1 Release Notes

See the Ops Manager release notes for any feature changes that may affect you during or after upgrade.

BOSH CLI output formatting has changed. If your deployment uses scripts that rely on BOSH output, you must update them to interpret BOSH CLI v2 command output

Ops Manager
Review the Known Issues section of the PAS v2.1 release notes This section covers recommended actions, new issues, and existing issues for PAS v2.1. PAS
Review the Known Issues section of the Ops Manager 2.1 release notes. This section covers recommended actions, new issues, and existing issues for Ops Man v2.1 Ops Manager
Review PCF v2.1 Breaking Changes. This topic describes the breaking changes you need to be aware of when upgrading PCF v2.1. PCF
Review Diego Network Communications. This topic describes the internal network communication paths between components of Diego workload management system in PAS. PAS

Run and collect foundation health status in at least one of the following ways:

  • If your PCF deployment has external metrics monitoring set up, verify that VM CPU, RAM, and disk use levels are within reasonable levels.
  • Run BOSH CLI commands to check system status:
    • bosh -e ALIAS -d DEPLOYMENT_NAME instances --ps
    • bosh -e ALIAS vms --vitals
    • bosh -e ALIAS -d DEPLOYMENT_NAME cck --report
  • Check Ops Manager GUI each PAS/Tiles the status page for CPU/RAM/DISK utilization
  • Validate Ops Manager persistent disk usage is below 50%. If not, follow Prep 6 in the Ops Manager upgrade docs.

bosh vms --vitals reveals VMs with high CPU, high memory, high disk utilization, and with state != running.

bosh instances with the flags --ps, --vitals, or --failinghighlights individual job failure.

PCF

Review KPI Changes from PCF v2.0 to v2.1.

For complete descriptions of each KPI, see Key Performance Indicators.

This document highlights new and changed Key Performance Indicators (KPIs) that operators may want to monitor with their PCF deployment to help ensure it is in a good operational state. PAS

Check that Diego cells have sufficient available RAM and disk capacity to support app containers.

The KPIs that monitor these these resources are are:

  • rep.CapacityRemainingMemory
  • rep.CapacityRemainingDisk

Review the Diego Cell Metrics section of the KPI topic for more information about these KPIs.

PAS
Check that a test app can be pushed and scaled horizontally, manually or via automated testing. This check ensures that the platform supports apps as expected before the upgrade. PCF
If needed, adjust the maximum number of Diego cells that the platform can upgrade simultaneously, to avoid overloading the other cells. See Managing Diego Cell Limits During Upgrade. For PCF 1.10 and later, the maximum number of cells that can update at once, max_in_flight is 4%. This setting is configured in the BOSH manifest’s Diego cell definition. See the Preventing Overload section for details. PAS

To save upgrade time, you can disable unused PAS post-deploy errands.

See the Post-Deploy Errands section of the Errands topic for details.

Only disable these errands if your environment does not need them.

PAS

If you are running Elastic Runtime MySQL as a cluster, run the mysql-diag tool to validate health of the cluster.

See the BOSH CLI v2 instructions in the Running mysql-diag topic.

PAS
Run bosh -e ALIAS clean-up --all to clean up old stemcells, releases, orphaned disks, and other resources before upgrade. This cleanup helps prevent the product and stemcell upload process from exceeding the BOSH Director’s available persistent disk space. BOSH

During Upgrade

Step Note/Reason Product/component
(Optional) Periodically take snapshots of storage metrics Pivotal recommends this if you have a large foundation and have experienced storage issues in the past. PCF

Monitor upgrade progress with methods such as:

  • Run BOSH CLI status checks:
    • bosh -e ALIAS task TASK_NUMBER
    • bosh -e ALIAS vms --vitals, bosh -e ALIAS instances --ps
  • Check app availability
  • Run cf CLI Commands
  • Check the availability of the Ops Manager GUI
  • Check NAS performance (if using NAS)
  • Check vSphere performance (if on vSphere)
Monitor the progress of the upgrade, checking the status of the foundation at various locations. PCF

Use the CF Diego Operator Toolkit (cfdot) to check Diego component instance count by current state.

See the cfdot documentation on github for details. PAS

(Optional) If you encounter problems during upgrade, collect the following information:

  • All job logs
  • Task debug logs for VM upgrade tasks
  • Installation log from Ops Manager
This information helps determine the cause of upgrade issues. PCF

After Upgrade

Step Note/Reason Product/component
Complete the steps in the After You Upgrade section of the Upgrading Pivotal Cloud Foundry topic. This ensures that you have the latest version of the Cloud Foundry Command Line Interface (cf CLI). ALL
Push and horizontally scale a test application. This is a performance test for PAS. PCF

Re-create your alias using BOSH:

bosh alias-env ALIAS -e DIRECTOR_IP

To log into BOSH after upgrading PCF, you need to recreate your alias. BOSH
Run BOSH CLI commands to check system status:
  • bosh -e ALIAS -d DEPLOYMENT_NAME instances --ps
  • bosh -e ALIAS vms --vitals
  • bosh -e ALIAS -d DEPLOYMENT_NAME cck --report
To ensure that all jobs and process are running as expected. PCF and all tiles
If you added custom VM Type or Persistent Disk Type options, ensure that these values are correctly set and were not overwritten. Verify that the Ops Manager Resource Config pane still lists your custom options. PCF

If you are running PAS MySQL as a cluster, run the mysql-diag tool to validate health of the cluster.

See the BOSH CLI v2 instructions in the Running mysql-diag topic.

PAS
Run bosh -e ALIAS clean-up --all to clean up old stemcells, releases, orphaned disks, and other unused resources. Tiles

Survey

Please take some time to help us improve this document by completing the Upgrade Checklist Survey.

Create a pull request or raise an issue on the source for this page in GitHub