Upgrade Preparation Checklist for PCF v2.3
- Back Up Your PCF Deployment
- Find Your Decryption Passphrase for Ops Manager
- Review Changes in PCF v2.3
- Validate Installed Buildpacks
- Update Tiles and Add-Ons
- Check Certificate Authority Expiration Dates
- Check the Capacity of Your Deployment
- Check the Health of Your Deployment
- Export Your Installation
- Next Steps
- Complete Survey
Page last updated:
Warning: Pivotal Cloud Foundry (PCF) v2.3 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.
This topic serves as a checklist for preparing to upgrade Pivotal Cloud Foundry (PCF) from v2.2 to v2.3.
This topic contains important preparation steps that you must follow before beginning your upgrade. Failure to follow these instructions may jeopardize your existing deployment data and cause the upgrade to fail.
After completing the steps in this topic, you can continue to Upgrading Pivotal Cloud Foundry.
Warning: Pivotal does not recommend that you skip minor versions when upgrading PCF. Skipping minor versions when upgrading PCF may result in breaking changes. To avoid additional breaking changes, upgrade PCF to the minor version that directly follows your current version of PCF.
Back Up Your PCF Deployment
Pivotal recommends backing up your PCF deployment before upgrading, to restore in the case of failure. To do this, follow the instructions in the Backing Up Pivotal Cloud Foundry with BBR topic.
Find Your Decryption Passphrase for Ops Manager
To complete the Ops Manager upgrade, you must have your Ops Manager decryption passphrase. You defined this decryption passphrase during the initial installation of Ops Manager.
Review Changes in PCF v2.3
Review each of the following links to understand the changes in the new release, such as new features, known issues, and breaking changes.
- Release Notes
- Known Issues
- Ops Manager v2.3 Known Issues
- PAS v2.3 Known Issues
- The following Pivotal Knowledge Base (KB) articles describe known issues for PCF v2.3:
- Diego releases out of sync while upgrading PASW to version 2.3.5 or above
- PAS backup-prepare job renamed to backup-restore may cause automation failures
- Buildpacks are missing or not installed in PAS 2.3
- Errands in upgrade to PCF or PAS 2.3 are failing as deploying on a stack is not supported by the Buildpack
- PCF upgrade fails with StacklessAndStackfulMatchingBuildpacksExistError
- Diego Cell garden healthcheck fails and becomes unhealthy
- Upgrade to RabbitMQ for PCF tile 1.13 fails with lock against the Mnesia DB
- Missing environment variables when using PAS 2.3+ and the cflinuxfs3 stack
- Breaking Changes
- KPI Changes
- Diego Network Communications:
Validate Installed Buildpacks
PAS v2.2 introduced a stack association feature for buildpacks. You can now have multiple versions of the same buildpack for different stacks. For more information, see cflinuxfs3 Stack and Compatible Buildpacks in the PAS Release Notes.
To avoid errors during upgrade, do the following:
If you have multiple buildpacks with the same name, associate each of those buildpacks with a stack. For more information, see PCF upgrade fails with StacklessAndStackfulMatchingBuildpacksExistError.
Unlock any default buildpacks that do not have an associated stack.
Associate any custom or non-default buildpacks with a stack.
For more information about associating a stack with an existing buildpack, see Managing Stack Association with the cf CLI.
Update Tiles and Add-Ons
The following section describes changes you must make to your product tiles and add-ons before upgrading PCF.
Review Tile Compatibility
Before you upgrade to PCF v2.3, please check whether the service tiles that you currently have deployed on PCF v2.2 are compatible with PCF v2.3.
To check PCF versions supported by a service tile, either from Pivotal or a Pivotal partner:
- Navigate to the tile’s download page on Pivotal Network.
- Select the tile version in the Releases dropdown.
- See the Depends On section under Release Details. For more information, refer to the tile’s release notes.
If the currently-deployed version of a tile is not compatible with PCF v2.3, you must upgrade the tile to a compatible version before you upgrade PCF. You do not need to upgrade tiles that are compatible with both PCF v2.2 and v2.3.
WARNING: Some tiles released with PCF v2.3 require a new Xenial stemcell. Before deploying tiles that use Xenial, you must update any BOSH add-ons used in your deployment to include Xenial stemcells and add a Xenial stemcell to your stemcell library in Ops Manager. For more information, see Check OS Compatibility of BOSH-Managed Add-Ons and Tiles.
Some partner service tiles may be incompatible with PCF v2.3. Pivotal works with partners to ensure their tiles are updated to work with the latest versions of PCF. For more information about which partner service release compatibility, review the Depends On section of the partners tile download page, the partners services release documentation in Pivotal Documentation, or contact the partner organization that produces the service tile.
The Product Compatibility Matrix provides an overview of which PCF versions support which versions of the most popular service tiles from Pivotal.
Environment Details
Pivotal provides the empty table below as a model to print out or adapt for recording and tracking the tile versions that you have deployed in all of your environments.
Sandbox | Non-Prod | Prod | Other… | ||
---|---|---|---|---|---|
Pivotal Cloud Foundry | Ops Manager | ||||
Pivotal Application Service (PAS) | |||||
Pivotal Cloud Foundry Services | MySQL v2 | ||||
Redis | |||||
RabbitMQ | |||||
Single Sign On (SSO) | |||||
Spring Cloud Services | |||||
Concourse | |||||
… | |||||
Pivotal Cloud Foundry Partner Services | New Relic | ||||
… |
Upgrade Services Tiles
Upgrade all service tiles to versions that are compatible with PCF v2.3. Service tiles are add-on products you install alongside your runtime. For example, MySQL for PCF, PCF Healthwatch, and RabbitMQ are service tiles.
Do not upgrade runtime tiles, such as PAS, PAS for Windows (PASW), or Pivotal Container Service (PKS), at this time.
Review the Compatibility Matrix and tile documentation to check version compatibility.
Configure BOSH Director
With each release of a new PCF version, BOSH Director may require specific updates before upgrading to the new version. See the following sections for what action to take before upgrading to PCF v2.3:
If you disabled BOSH DNS, reenable it. In the Director Config pane of the BOSH Director tile, clear the Disable BOSH DNS server for troubleshooting purposes checkbox.
Check the required machine specifications for Ops Manager v2.3. These specifications are specific to your IaaS. If these specifications do not match your existing Ops Manager, modify the values of your Ops Manager VM instance. For example, if the boot disk of your existing Ops Manager is 50 GB and the new Ops Manager requires 100 GB, then increase the size of your Ops Manager boot disk to 100 GB.
Configure PAS
With each release of a new PCF version, PAS may require specific updates before upgrading to the new version. See the following sections for what action to take before upgrading to PCF v2.3:
Install PAS v2.2.4 or Later
In PAS v2.3, traffic between the Gorouter and Cloud Controller is encrypted. To prevent app downtime while upgrading from v2.2 to v2.3, you must be running PAS v2.2.4 or later. These versions of PAS v2.2 contain the configuration router.backends.enable_tls: true
in the Gorouter manifest.
Confirm Cipher Suites
Ensure that the TLS Cipher Suites for Router field contains a cipher suite supported by both Gorouter and CAPI. For details, see Removal of Default Ciphers Causes Downtime when Upgrading from 2.2 to 2.3 and from 2.3 to 2.4.
Enable GrootFS
Note: If GrootFS is already enabled for your deployment, you can skip this step.
You must enable GrootFS in your PAS v2.2 tile before upgrading to PAS v2.3. PAS v2.3 uses GrootFS as the only Garden image plugin and removes the checkbox to enable or disable GrootFS in the Configure Application Containers pane. For more information, see Remove Deprecated Garden Image Plugin Option in the PAS v2.3 release notes.
To enable GrootFS, do the following:
Go to Application Containers in PAS.
Select Enable the GrootFS container image plugin for Garden RunC.
Click Save.
Go to Director Config in the BOSH Director tile.
Select Recreate all VMs.
Click Save.
Return to the Installation Dashboard.
Click Review Pending Changes, then Apply Changes.
Once you have enabled GrootFS, ensure the apps running on your deployment function properly before proceeding with your upgrade. Any issues you experience are most likely to appear directly following the deployment of PAS.
Migrate Internal Databases to Percona MySQL
PAS v2.2 introduced Percona Server as an option for your internal MySQL database and deprecated the older MariaDB option. The Percona Server option is more secure than MariaDB, and later versions of PCF remove the MariaDB entirely. Pivotal recommends that you migrate to Percona Server if you use internal MySQL for your PAS deployment and have not yet migrated.
For more information, see Migrating to Internal Percona MySQL.
Disable Unused Errands
To save upgrade time, you can disable unused PAS post-deploy errands. See the Post-Deploy Errands section of the Errands topic for details. Only disable these errands if your environment does not need them.
In some cases, if you have previously disabled lifecycle errands for any installed product to reduce deployment time, you may want to re-enable these errands before upgrading. For more information, see the Adding and Deleting Products topic.
Check OS Compatibility of BOSH-Managed Add-Ons and Tiles
Some tiles released with PCF v2.3 require stemcells based on the Xenial OS rather than the older Trusty line. Before deploying tiles that use Xenial, you must:
- Update any BOSH add-ons used in your deployment so that they include both Trusty and Xenial stemcells. To do this, follow the Update Manifests for BOSH-Managed Add-Ons directions below.
- For tiles that require Xenial, manually download and import a new Xenial stemcell into the Ops Manager Stemcell Library. See the Tile Patch Releases Use Xenial Stemcells note for details.
Update Manifests for BOSH-Managed Add-Ons
Before upgrading to PCF v2.3, operators who have deployed any PCF add-ons such as IPsec for PCF, ClamAV for PCF, or File Integrity Monitoring for PCF and who have deployed or are planning to deploy Pivotal Application Service for Windows must modify the add-on manifest to specify a compatible OS stemcell.
For example, File Integrity Monitor for PCF (FIM) is not supported on Windows. Therefore, the manifest must use an include
directive to specify the target OS stemcell of ubuntu-trusty
and ubuntu-xenial
.
Note: To upgrade to a Xenial stemcell, see the documentation for each add-on and follow the instructions.
To update an add-on manifest, do the following:
Locate your existing add-on manifest file. For example, for FIM, locate the
fim.yml
you uploaded to the Ops Manager VM.Modify the manifest to include following
include
directive to your manifest:include: stemcell: - os: ubuntu-trusty - os: ubuntu-xenial
Upload the modified manifest file to your PCF deployment. For example instructions, see Create the FIM Manifest.
If you use any other BOSH-managed add-ons in your deployment, you should verify OS compatibility for those component as well. For more information about configuring BOSH add-on manifests, see the BOSH documentation.
Update Runtime Config of ClamAV
If you are using ClamAV for PCF, you must update your runtime config before upgrading to Pivotal Application Service for Windows (PASW) v2.3.
If you upgrade to PASW v2.3 without first updating the runtime config, then your Windows VMs will not be protected by ClamAV.
For more information about how to update your runtime config, see Updating ClamAV Add-on for PCF to Run with PAS for Windows v2.3 in the ClamAV documentation.
Check Backup and Restore External Blobstore Add-On
If you have enabled external blobstore backups for an Azure Blobstore using the Blobstore Add-On, you need to update your runtime configuration to remove the sdk-preview
add-on before upgrading to PCF v2.3. If you do not remove this job, upgrading PAS fails with the error:
Preparing deployment: Preparing deployment (00:00:01) L Error: Colocated job 'azure-blobstore-backup-restorer' is already added to the instance group 'backup-restore'.
After removing this job from your runtime configuration, ensure that the Enable backup and restore checkbox is enabled in the PAS tile > File Storage pane. See External Azure Storage for instructions.
Check Certificate Authority Expiration Dates
Depending on the requirements of your deployment, you may need to rotate your Certificate Authority (CA) certificates. The non-configurable certificates in your deployment expire every two years. You must regenerate and rotate them so that critical components do not face a complete outage.
Note: PCF uses SHA-2 certificates and hashes by default. You can convert existing SHA-1 hashes into SHA-2 hashes by rotating your Ops Manager certificates using the procedure described in the Regenerating and Rotating Non-Configurable TLS/SSL Certificates section of Managing TLS Certificates.
To retrieve information about all the RSA and CA certificates for the BOSH Director and other products in your deployment, you can use the GET /api/v0/deployed/certificates?expires_within=TIME
request of the Ops Manager API.
In this request, the expires_within
parameter is optional. Valid values for the parameter are d
for days, w
for weeks, m
for months, and y
for years. For example, to search for certificates expiring within one month, replace TIME
with 1m
:
$ curl "https://OPS-MAN-FQDN/api/v0/deployed/certificates?expires_within=1m" \ -X GET \ -H "Authorization: Bearer UAA_ACCESS_TOKEN"
For information about regenerating and rotating CA certificates, see Managing TLS Certificates.
Check the Capacity of Your Deployment
The following sections describe steps for ensuring your deployment has adequate capacity to perform the upgrade.
Confirm Adequate Disk Space
Confirm that you have adequate disk space for your upgrades. You need at least 20 GB of free disk space to upgrade PCF Ops Manager and Pivotal Application Service. If you plan to upgrade other products, the amount of disk space required depends on how many tiles you plan to deploy to your upgraded PCF deployment.
To check current persistent disk usage, select the BOSH Director tile from the Installation Dashboard. Select Status and review the value of the PERS. DISK
column. If persistent disk usage is higher than 50%, select Settings > Resource Config, and increase your persistent disk space to handle the size of the resources. If you do not know how much disk space to allocate, set the value to at least 100 GB
.
Check Diego Cell RAM and Disk
Check that Diego cells have sufficient available RAM and disk capacity to support app containers.
The KPIs that monitor these these resources are are:
rep.CapacityRemainingMemory
rep.CapacityRemainingDisk
Adjust Diego Cell Limits
If needed, adjust the maximum number of Diego cells that the platform can upgrade simultaneously, to avoid overloading the other cells. See Managing Diego Cell Limits During Upgrade.
For PCF v1.10 and later, the maximum number of cells that can update at once, `max_in_flight is 4%. This setting is configured in the BOSH manifest’s Diego cell definition. See the Preventing Overload section for details.
Review the Diego Cell Metrics section of the KPI topic for more information about these KPIs.
Review File Storage IOPS and Other Upgrade Limiting Factors
During the PCF upgrade process, a large quantity of data is moved around on disk.
To ensure a successful upgrade of PCF, verify that your underlying PAS file storage is performant enough to handle the upgrade. For more information about the configurations to evaluate, see Upgrade Considerations for Selecting Pivotal Cloud Foundry Storage.
In addition to file storage IOPS, consider additional existing deployment factors that can impact overall upgrade duration and performance:
Factor | Impact |
---|---|
Network latency | Network latency can contribute to how long it takes to move app instance data to new containers. |
Number of ASGs | A large number of Application Security Groups in your deployment can contribute to an increase in app instance container startup time. |
Number of app instances and application growth | A large increase in the number of app instances and average droplet size since the initial deployment can increase the upgrade impact on your system. |
To review example upgrade-related performance measurements of an existing production Cloud Foundry deployment, see the Pivotal Web Services Performance During Upgrade topic.
Run BOSH Clean-Up
Run bosh -e ALIAS clean-up --all
to clean up old stemcells, releases, orphaned disks, and other resources before upgrade. This cleanup helps prevent the product and stemcell upload process from exceeding the BOSH Director’s available persistent disk space.
Check the Health of Your Deployment
The following sections describe steps for ensuring your deployment is healthy before you perform the upgrade.
Collect Foundation Health Status
For collecting foundation health status, Pivotal recommends PCF Healthwatch, which monitors and alerts on the current health, performance, and capacity of PCF. For more information, see the PCF Healthwatch documentation.
If you are not using PCF Healthwatch, you can do some or all of the following to collect foundation health status:
- If your PCF deployment has external metrics monitoring set up, verify that VM CPU, RAM, and disk use levels are within reasonable levels.
- Run BOSH CLI commands to check system status:
bosh -e ALIAS -d DEPLOYMENT_NAME instances --ps
.bosh instances
with the flags--ps
,--vitals
, or--failing
highlights individual job failure.bosh -e ALIAS vms --vitals
. This reveals VMs with high CPU, high memory, high disk utilization, and withstate
!=running
.bosh -e ALIAS -d DEPLOYMENT_NAME cck --report
- Check Ops Manager GUI each PAS/Tiles the status page for CPU/RAM/DISK utilization
- Validate Ops Manager persistent disk usage is below 50%. If not, follow the procedure in Confirm Adequate Disk Space.
(Optional) Check the logs for errors before proceeding with the upgrade. For more information, see Viewing Logs in the Command Line Interface.
Push and Scale a Test App
Check that a test app can be pushed and scaled horizontally, manually or through automated testing. This check ensures that the platform supports apps as expected before the upgrade.
Validate MySQL Cluster Health
If you are running PAS MySQL as a cluster, run the mysql-diag
tool to validate health of the cluster.
See the BOSH CLI v2 instructions in the Running mysql-diag topic.
Review Pending and Recent Changes
Confirm there are no outstanding changes in Ops Manager or any other tile. All tiles should be green. Click Review Pending Changes, then Apply Changes if necessary.
After applying changes, click Recent Install Logs to confirm that the changes completed cleanly:
Cleanup complete {"type": "step_finished", "id": "clean_up_bosh.cleaning_up"} Exited with 0.
Export Your Installation
To export your installation, do the following:
In your Ops Manager v2.2 Installation Dashboard, click the account dropdown and select Settings.
On the Settings screen, select Export Installation Settings from the left menu, then click Export Installation Settings.
This exports the current PCF installation with all of its assets.
When you export an installation, the export contains the base VM images, necessary packages, and configuration settings, but does not include releases between upgrades if Ops Manager has already uploaded them to BOSH. When backing up PCF, you must take this into account by backing up the BOSH blobstore that contains the uploaded releases. BOSH Backup and Restore (BBR) backs up the BOSH blobstore. For more information, see Backing Up Pivotal Cloud Foundry with BBR.
- The export time depends on the size of the exported file.
- Some browsers do not provide feedback on the status of the export process and might appear to hang.
Note: Some operating systems automatically unzip the exported installation. If this occurs, create a ZIP file of the unzipped export. Do not start compressing at the “installation” folder level. Instead, start compressing at the level containing the config.yml
file:
WARNING: If you fail to perform the remedial steps for this issue, this upgrade process may corrupt your existing usage data.
Next Steps
Now that you have completed the Upgrade Preparation Checklist for PCF v2.3, continue to Upgrading Pivotal Cloud Foundry.
Complete Survey
Please take some time to help us improve this document by completing the Upgrade Checklist Survey.