Upgrade Preparation Checklist for Pivotal Platform v2.7

Page last updated:

This topic serves as a checklist for preparing to upgrade Ops Manager and Pivotal Application Service (PAS) from v2.6 to v2.7.

Overview

This topic contains important preparation steps that you must follow before beginning your upgrade. Failure to follow these instructions may jeopardize your existing deployment data and cause the upgrade to fail.

After completing the steps in this topic, you can continue to Upgrading Pivotal Platform.

VMware recommends upgrading directly to Ops Manager v2.10 from Ops Manager v2.6. To upgrade to Ops Manager v2.10, see Jump Upgrade to Ops Manager v2.10 in Upgrading Pivotal Platform.

Warning: Although you can skip minor versions when upgrading Ops Manager, you should not skip minor versions when upgrading PAS. Skipping minor versions when upgrading PAS may result in additional breaking changes. To avoid this, upgrade PAS to v2.7. For more information, see Upgrade PAS in Upgrading Pivotal Platform.

Back Up Your PCF Deployment

VMware recommends backing up your Pivotal Cloud Foundry (PCF) deployment before upgrading, to restore in the case of failure. To do this, follow the procedure in Backing Up Deployments with BBR.

Find Your Decryption Passphrase for Ops Manager

To complete the Ops Manager upgrade, you must have your Ops Manager decryption passphrase. You defined this decryption passphrase during the initial installation of Ops Manager.

Review Changes in Pivotal Platform v2.7

Review each of the following links to understand the changes in the new release, such as new features, known issues, and breaking changes.

Restage Windows Apps

The windows2016 stack is deprecated in Pivotal Application Service for Windows (PASW) v2.7.

Before upgrading to PASW v2.7 from PASW v2.6, all Windows apps that were pushed with the windows2016 stack must be restaged to use the windows stack.

For more information, see Deprecation of the windows2016 Stack in Pivotal Application Service for Windows v2.6 Release Notes.

If you are running PASW v2.4.4 or later on PCF v2.6, you can also restage your apps before upgrading to PASW v2.6. For more information about upgrading to PASW v2.6, see Upgrade to PASW v2.6 below.

Update Tiles and Add-Ons

These sections describe changes you must make to your product tiles and add-ons before upgrading to Pivotal Platform v2.7.

Upgrade to PASW v2.6

If you are running PASW v2.4.4 or later on PCF v2.6 and want to upgrade to Pivotal Platform v2.7, you must upgrade to PASW v2.6 before upgrading to Pivotal Platform v2.7. To download PASW, go to the Pivotal Application Service for Windows page on Pivotal Network.

Upgrade Stemcell to v2019.4 or Later for PASW v2.6

If you are running PASW v2.6 using Windows stemcell v2019.3, you must upgrade your stemcell to v2019.4 or later before you upgrade to Ops Manager v2.7.

To upgrade your stemcell, go to the Pivotal Stemcells (Windows) page on Pivotal Network.

Upgrade to SCS v2.1 or Remove SCS v2.0

Pivotal Spring Cloud Services (SCS) v2.0.x is not compatible with Pivotal Platform v2.7. You cannot upgrade to Ops Manager v2.7 or PAS v2.7 without either removing SCS v2.0.x or upgrading from SCS v2.0.x to SCS v2.1.x first.

If you need to use the p-circuit-breaker-dashboard service in SCS v2.0.x, you must upgrade SCS v2.0.x to SCS v2.1.x before you upgrade to Pivotal Platform v2.7.

If you are running SCS v3.0.x or SCS v3.1.x alongside SCS v2.0.x and do not require p-circuit-breaker-dashboard functionality, you can remove SCS v2.0.x entirely after migrating completely to SCS v3.1.x. To do this, see How to Upgrade to SCS v3.1.x in Upgrading Pivotal Spring Cloud Services in the SCS documentation.

After you successfully remove SCS v2.0.x or upgrade to SCS v2.1.x, you can proceed with the Pivotal Platform v2.7 upgrade.

Review Service Tile Compatibility

Before you upgrade, check whether the service tiles that you currently have are compatible with the new version of Pivotal Platform.

To check Pivotal Platform versions supported by a service tile, either from Pivotal or a Pivotal partner:

  1. Navigate to the tile’s download page on Pivotal Network.

  2. Select the tile version in the Releases dropdown.

  3. See the Depends On section under Release Details. For more information, see the tile’s release notes.

If the currently-deployed version of a tile is not compatible with Pivotal Platform v2.7, you must upgrade the tile to a compatible version before you upgrade to Pivotal Platform v2.7. You do not need to upgrade tiles that are compatible with both PCF v2.6 and Pivotal Platform v2.7.

Some partner service tiles may be incompatible with Pivotal Platform v2.7. Pivotal works with partners to ensure their tiles are updated to work with the latest versions of Pivotal Platform. For more information about which partner service release compatibility, review the Depends On section of the partners tile download page, see the partners services release documentation in the Pivotal documentation, or contact the partner organization that produces the service tile.

The Product Compatibility Matrix provides an overview of which Pivotal Platform versions support which versions of the most popular service tiles from Pivotal.

Environment Details

You can use the empty table below as a model to print out or adapt for recording and tracking the tile versions that you have deployed in all of your environments.

Sandbox Non-Prod Prod Other…
Pivotal Platform Ops Manager
Pivotal Application Service (PAS)
Pivotal Platform Services MySQL v2
Redis
RabbitMQ
Single Sign-On (SSO)
Spring Cloud Services
Concourse
Pivotal Platform Partner Services New Relic

Upgrade Services Tiles

Upgrade all service tiles to versions that are compatible with Pivotal Platform v2.7. Service tiles are add-on products you install alongside your runtime. For example, MySQL for Pivotal Platform, Pivotal Healthwatch, and RabbitMQ are service tiles.

Do not upgrade runtime tiles, such as PAS, PASW, or Enterprise Pivotal Container Service (Enterprise PKS) at this time.

To check version compatibility, see the Compatibility Matrix and service tile documentation.

Configure BOSH Director

With each release of a new Pivotal Platform version, BOSH Director may require specific updates before upgrading to the new version. For actions to take before upgrading to Pivotal Platform v2.7, see the sections below.

Check Required Machine Specifications

Check the required machine specifications for Ops Manager v2.7. These specifications are specific to your IaaS. If these specifications do not match your existing Ops Manager deployment, modify the values of your Ops Manager VM instance. For example, if the boot disk of your existing Ops Manager deployment is 50 GB and the new Ops Manager deployment requires 100 GB, increase the size of your Ops Manager boot disk to 100 GB.

Configure PAS

With each release of a new Pivotal Platform version, PAS may require specific updates before upgrading to the new version. See the following sections for what action to take before upgrading to Pivotal Platform v2.7:

(Optional) Disable Unused Errands

To save upgrade time, you can disable unused PAS post-deploy errands. For more information, see Post-Deploy Errands in Errands. Only disable these errands if your environment does not need them.

In some cases, if you have previously disabled lifecycle errands for any installed product to reduce deployment time, you may want to re-enable these errands before upgrading. For more information, see Adding and Deleting Products in Adding and Deleting Products.

Configure Diego Cell Garbage Collection

In the App Containers pane of the PAS tile, if Docker images disk cleanup scheduling on Diego Cell VMs is set to Clean up disk space once usage fills disk and the value of Reserved disk space for other jobs is different from the default of 15360, you must specify a new threshold.

If the scheduling is set to Never clean up Diego Cell disk space or Routinely clean up Diego Cell disk space, or the reserved disk space value is set to the default of 15360, no action is necessary. All thresholds will migrate to a sensible value when you upgrade PAS. For more information, see Options for Disk Cleanup in Configuring Diego Cell Disk Cleanup Scheduling in the PAS documentation.

Do Not Edit Syslog Agent Configuration When Upgrading from PAS v2.7.6 or Earlier

If you are upgrading from PAS v2.7.6 or earlier, do not change your Syslog Agent configuration when you stage PAS v2.7.7 or later for the first time. For example, if Syslog Agents are enabled on your current version of PAS, ensure that Syslog Agents are enabled when you stage PAS v2.7.7 or later for the first time.

If you change your Syslog Agent configuration after staging and before the first deploy of PAS v2.7.7 or later, there is a significant period of log loss or log duplication while BOSH enables or disables Syslog Agents. For more information about this log loss or duplication, see Duplicate or Missing Logs in Pivotal Application Service v2.7 Release Notes in the PAS documentation.

You can edit the Syslog Agent configuration in PAS v2.7.7 or later after the first deploy. To edit your Syslog Agent configuration, see Configure System Logging in Configuring PAS in the PAS documentation.

Check OS Compatibility of BOSH-Managed Add-Ons and Tiles

Before upgrading to Pivotal Platform v2.7, operators who have deployed any PCF add-ons such as IPsec for PCF, ClamAV for PCF, or File Integrity Monitoring for PCF and who have deployed or are planning to deploy PASW must modify the add-on manifest to specify a compatible OS stemcell. For more information, see the PASW documentation.

For example, File Integrity Monitor for PCF (FIM) is not supported on Windows. Therefore, the manifest must use an include directive to specify the target OS stemcell of ubuntu-trusty and ubuntu-xenial.

Note: To upgrade to a Xenial stemcell, see the documentation for each add-on and follow the instructions.

To update an add-on manifest:

  1. Locate your existing add-on manifest file. For example, for FIM, locate the fim.yml you uploaded to the Ops Manager VM.

  2. Modify the manifest to include following include directive to your manifest:

      include:
        stemcell:
          - os: ubuntu-xenial
    
  3. Upload the modified manifest file to your PCF deployment. For example instructions, see Installing File Integrity Monitoring on BOSH Director in the Pivotal File Integrity Monitoring documentation.

If you use any other BOSH-managed add-ons in your deployment, you should verify OS compatibility for those component as well. For more information about configuring BOSH add-on manifests, see Addons Block in Director Runtime Config in the BOSH documentation.

Check Backup and Restore External Blobstore Add-On

If you have enabled external blobstore backups for an Azure Blobstore using the blobstore add-on, you must update your runtime configuration to remove the sdk-preview add-on before upgrading. If you do not remove this job, upgrading PAS fails with the error below:

Preparing deployment: Preparing deployment (00:00:01)
  L Error: Colocated job 'azure-blobstore-backup-restorer' is already added to the instance group 'backup-restore'.

After removing this job from your runtime configuration, ensure that the Enable backup and restore checkbox is enabled in the File Storage pane of the PAS tile.

For more information, see Azure Blobstores in Enabling External Blobstore Backups.

Check Certificate Authority Expiration Dates

Depending on the requirements of your deployment, you may need to rotate your certificate authority (CA) certificates. The non-configurable certificates in your deployment expire every two years. You must regenerate and rotate them so that critical components do not face a complete outage.

Note: Pivotal Platform uses SHA-2 certificates and hashes by default. You can convert existing SHA-1 hashes into SHA-2 hashes by rotating your Ops Manager certificates using the procedure described in Rotating Identity Provider SAML Certificates.

To retrieve information about all the RSA and CA certificates for the BOSH Director and other products in your deployment, you can use the GET /api/v0/deployed/certificates?expires_within=TIME request of the Ops Manager API.

In this request, the expires_within parameter is optional. Valid values for the parameter are d for days, w for weeks, m for months, and y for years. For example, to search for certificates expiring within one month, replace TIME with 1m:

$ curl "https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=1m" \
 -X GET \
 -H "Authorization: Bearer UAA_ACCESS_TOKEN"

Where:

  • OPS-MANAGER-FQDN is the fully-qualified domain name (FQDN) of your Ops Manager deployment.
  • UAA-ACCESS-TOKEN is your UAA access token.

For information about regenerating and rotating CA certificates, see Overview of Certificate Rotation.

Check the Capacity of Your Deployment

These sections describe steps for ensuring your deployment has adequate capacity to perform the upgrade.

Confirm Adequate Disk Space

Confirm that you have adequate disk space for your upgrades. You need at least 20 GB of free disk space to upgrade Ops Manager and . If you plan to upgrade other products, the amount of disk space required depends on how many tiles you plan to deploy to your upgraded Pivotal Platform deployment.

To check current persistent disk usage:

  1. Navigate to the Ops Manager Installation Dashboard.

  2. Click the BOSH Director tile.

  3. Select the Status tab and check the value of the PERS. DISK column. If persistent disk usage is higher than 50%:

    1. Select Resource Config.
    2. Increase your persistent disk space to handle the size of the resources. If you do not know how much disk space to allocate, set the value to at least 100 GB.

Check Diego Cell RAM and Disk

Check that Diego Cells have sufficient available RAM and disk capacity to support app containers.

The KPIs that monitor these these resources are are:

  • rep.CapacityRemainingMemory
  • rep.CapacityRemainingDisk

Adjust Diego Cell Limits

If needed, adjust the maximum number of Diego Cells that the platform can upgrade simultaneously, to avoid overloading the other Diego Cells. For more information, see Limit Component Instance Restarts in Configuring PAS for Upgrades in the PAS documentation.

The maximum number of Diego Cells that can update at once, max_in_flight, is 4%. This setting is configured in the BOSH manifest’s Diego Cell definition. For more information, see Prevent Overload in Configuring PAS for Upgrades in the PAS documentation.

For more information about these KPIs, see Diego Cell Metrics in Key Performance Indicators in the PAS documentation.

Review File Storage IOPS and Other Upgrade Limiting Factors

During the Pivotal Platform upgrade process, a large quantity of data is moved around on disk.

To ensure a successful upgrade of your deployment, verify that your underlying PAS file storage is performant enough to handle the upgrade. For more information about the configurations to evaluate, see Configure File Storage in Configuring PAS for Upgrades in the PAS documentation.

In addition to file storage IOPS, consider additional existing deployment factors that can impact overall upgrade duration and performance:

Factor Impact
Network latency Network latency can contribute to how long it takes to move app instance data to new containers.
Number of ASGs A large number of App Security Groups (ASGs) in your deployment can contribute to an increase in app instance container startup time. For more information, see App Security Groups in the PAS documentation.
Number of app instances and app growth A large increase in the number of app instances and average droplet size since the initial deployment can increase the upgrade impact on your system.

For example upgrade-related performance measurements of an existing production deployment, see Upgrade Load Example: Pivotal Web Services During Upgrade in the PAS documentation.

Run BOSH Clean-Up

To clean up old stemcells, releases, orphaned disks, and other resources before upgrade:

  1. Run:

    bosh -e ALIAS clean-up --all
    

    Where ALIAS is your BOSH deployment alias.

This cleanup helps prevent the product and stemcell upload process from exceeding the BOSH Director’s available persistent disk space.

Check the Health of Your Deployment

These sections describe steps for ensuring your deployment is healthy before you perform the upgrade.

Collect Foundation Health Status

For collecting foundation health status, VMware recommends Pivotal Healthwatch, which monitors and alerts on the current health, performance, and capacity of Pivotal Platform. For more information, see the Pivotal Healthwatch documentation.

If you are not using Pivotal Healthwatch, you can do some or all of the following to collect foundation health status:

  • Check your system status.

    • To check the status of your BOSH instances, run:

      bosh -e ALIAS -d DEPLOYMENT-NAME instances --ps
      

      Where:

      • ALIAS is your BOSH deployment alias.
      • DEPLOYMENT-NAME is the name of the BOSH deployment with the instances you want to check.

        bosh instances with the flags --ps, --vitals, or --failing highlights individual job failure.
    • To check the status of your BOSH VMs, run:

      bosh -e ALIAS vms --vitals
      

      Where ALIAS is your BOSH deployment alias.

      This command reveals VMs with high CPU, high memory, high disk utilization, and with state != running.

    • To check the status of your BOSH cloud config, run:

      bosh -e ALIAS -d DEPLOYMENT-NAME cck --report
      

      Where:

      • ALIAS is your BOSH deployment alias.
      • DEPLOYMENT-NAME is the name of the BOSH deployment with the cloud config you want to check.
  • Check the Status tab of each PAS tile for VM CPU, RAM, and disk use levels.

  • Check that Ops Manager persistent disk usage is below 50%. If not, follow the procedure in Confirm Adequate Disk Space above.

  • (Optional) Check the logs for errors before proceeding with the upgrade. For more information, see Viewing Logs in the Command Line Interface in App Logging in PAS in the PAS documentation.

Push and Scale a Test App

Check that a test app can be pushed and scaled horizontally, manually, or through automated testing. This check ensures that the platform supports apps as expected before the upgrade.

Validate MySQL Cluster Health

If you are running PAS MySQL as a cluster, run the mysql-diag tool to validate health of the cluster.

For BOSH CLI v2 instructions, see Running mysql-diag in the PAS documentation.

Review Pending and Recent Changes

To review pending and recent changes:

  1. Confirm there are no outstanding changes in Ops Manager or any other tile. All tiles should be green. If necessary, click Review Pending Changes, then Apply Changes.

  2. After applying changes, click Recent Install Logs to confirm that the changes completed cleanly:

    Cleanup complete
    {"type": "step_finished", "id": "clean_up_bosh.cleaning_up"}
    Exited with 0.
    

Export Your Installation

To export your installation:

  1. In the Ops Manager Installation Dashboard, click the account dropdown and select Settings.

    Upgrade to 1.9

  2. On the Settings page, select Export Installation Settings from the left menu, then click Export Installation Settings.

This exports the current PCF installation with all of its assets.

When you export an installation, the export contains the base VM images, necessary packages, and configuration settings, but does not include releases between upgrades if Ops Manager has already uploaded them to BOSH. When backing up your deployment, you must take this into account by backing up the BOSH blobstore that contains the uploaded releases. BOSH Backup and Restore (BBR) backs up the BOSH blobstore. For more information, see Backing Up Deployments with BBR.

  • The export time depends on the size of the exported file.

  • Some browsers do not provide feedback on the status of the export process and might appear to hang.

Note: Some operating systems automatically unzip the exported installation. If this occurs, create a ZIP file of the unzipped export. Do not start compressing at the “installation” folder level. Instead, start compressing at the level containing the rails_database_dump.postgres file, as shown in the example image below:

Compress

Next Steps

Now that you have completed the Upgrade Preparation Checklist, continue to Upgrading Pivotal Platform.