PCF v2.2 Breaking Changes

This topic describes the breaking changes you need to be aware of when upgrading to Pivotal Cloud Foundry (PCF) v2.2. For more information about important preparation steps you must follow before beginning an upgrade, see Upgrading Pivotal Cloud Foundry.

Pivotal Cloud Foundry

PKS v1.3 Does Not Support Ops Manager v2.2

PKS v1.3 only supports Ops Manager v2.3 and v2.4. Do not use PKS v1.3 with any other version of Ops Manager.

Tile Patch Releases Use Xenial Stemcells

Ubuntu 14.04 LTS (Trusty) stemcells are reaching end-of-support in April 2019. Starting in September 2018, Pivotal and its partners will begin releasing product tile versions for PCF that require Ubuntu 16.04 LTS (Xenial) stemcells instead of Trusty stemcells. Using Xenial stemcells is necessary to avoid exposure to security vulnerabilities. A PCF deployment may include some products running on Trusty stemcells and others running on Xenial.

Before deploying any tiles that require a Xenial stemcell, you must manually download and import a new Xenial stemcell into the Ops Manager Stemcell Library. As a consequence of using Xenial stemcells, any automation you have set up to update the existing tile using Trusty stemcells may break.

Before you upgrade PCF, always verify that any tiles that you have installed, either from Pivotal or a partner, are compatible with the version of PCF you are deploying and its supported stemcells. For more information on checking PCF and stemcell compatibility, see Tile Compatibility in the Upgrade Checklist.

To find out which stemcell version is used by a product tile version, check the release notes or the Pivotal Network download page for the tile.

Pivotal Application Service (PAS)

Downtime when Upgrading from v2.1 to v2.2.7 or Later

Upgrading directly from PAS v2.1.x to v2.2.7 or later may cause significant app downtime. If you are upgrading from v2.1.x, Pivotal recommends that you upgrade to v2.2.6 or an earlier patch release of v2.2.x. Once you are on v2.2.6 or an earlier patch release of v2.2.x, you can then upgrade to v2.2.7.

A fix for this issue is planned for v2.2.12.

For more information, see the following article in the Pivotal Knowledge Base: How to avoid app downtime while upgrading from PAS v2.1.x to v2.2.7.

CredHub Database Cannot be External on GCP

Log Aggregation Systems May Parse Diego Timestamps Incorrectly

The timestamps in the Diego component logs are now in a format compatible with RFC 3339.

This timestamp format is enabled by default for new PAS v2.2 deployments. Upgrades from earlier PAS versions retain the previous component log format with Unix epoch timestamps.

You can enable the new timestamp format in your PAS tile configuration. Before enabling the new timestamp format for Diego logs, ensure that your log aggregation system anticipates the timestamp format change.

If you experience issues, you can disable RFC 3339 format in the PAS tile. For more information about this PCF v2.2 feature, see PCF v2.2 Feature Highlights and New Format for Timestamps in Diego Component Logs in Pivotal Application Service v2.2 Release Notes.

SSH Proxy May Restrict Client Access

The SSH proxy now accepts a narrower range of ciphers, MACs, and key exchanges. This feature may have complications if you use a SSH client other than cf ssh to connect. For more information, see Application SSH Access without cf CLI.

Temporary Downtime when Migrating Internal System Databases

If you use internal system databases, migrating the databases to the new option of Internal Databases - MySQL - Percona XtraDB Cluster causes temporary downtime of PAS system functions. Migrating the database does not interrupt apps running on PAS, but does temporarily prevent you from pushing new ones. For more information, see Migrating to Internal Percona MySQL.

PAS v2.2 deprecates the MariaDB Galera internal database option, and Pivotal plans to remove MariaDB Galera as a database option in PAS v2.3. To continue using internal system databases in future versions of PAS, you need to migrate them to Percona while you are running PAS v2.2, or else when you upgrade to PAS v2.3.

This change has no effect if you have External Databases selected in the PAS tile Databases pane.

App Autoscaler Fails When Loggregator’s Log Cache Is Disabled

App Autoscaler relies on API endpoints in Loggregator’s Log Cache component. If you disable Log Cache, App Autoscaler will fail. For more information about Log Cache, see Loggregator Introduces Log Cache.

Read-only Volume Mounts

We back-ported a fix from NFS 1.3.1 to NFS 1.2.1 for an incompatibility between our NFS Volume release and Diego’s container runtime, garden. But, because the fix was in the NFS Service Broker, and service bindings created by old versions of this broker won’t get migrated during upgrade, existing NFS service bindings that specify read-only mounts will still exhibit the incompatibility.

As a result, customers upgrading from versions containing nfs-volume-release < 1.2.1 that have NFS services bound read-only to their applications will see that their applications crash after upgrade.

To fix this condition, customers should unbind the service, rebind it, and then restage the application.

Alternately, customers wishing to avoid application down time can temporarily re-bind their applications as read/write before upgrading, and then swtich to read-only afterwards.

Diego Cell Garden Healthcheck Fails and Becomes Unhealthy

You may observe the following error in the Diego Cell Garden logs:

cfnetworking: cni up failed: add network list failed: initialize net out: creating chain: iptables call: running [/var/vcap/packages/iptables/sbin/iptables -t filter -N netout--executor-healthcheck --wait]: exit status 1: iptables: Chain already exists.

This is a symptom of a failed Diego healthcheck, which can cause the Diego cell to become unhealthy. If you encounter this error, see the following Pivotal Knowledge Base article to understand and resolve the issue: Diego Cell Garden healthcheck fails and becomes unhealthy.

PCF Operations Manager (Ops Manager)

Redeploy All Products After Upgrading to Ops Manager v2.2

When upgrading to PCF v2.2, redeploy all product tiles by clicking Apply Changes in the Ops Manager installation dashboard. A universal redeploy ensures that the updated BOSH DNS release applies consistently to all products.

Do not use the Review Pending Changes button to deploy product tiles individually during the upgrade to PCF v2.2. Instead, redeploy everything and selectively deploy individual product tiles only after you have successfully upgraded to PCF v2.2.

For more information about selective deployment, see (Beta) Selectively Deploy Ops Manager Tiles.

Ops Manager Logs Have Changed Locations

Ops Manager centralized its logs in /var/log/opsmanager so that you can now configure a syslog server in the Ops Manager Settings.

Because of this change for Ops Manager v2.2, you cannot run scripts to fetch Ops Manager logs at their previous locations. For example, you cannot fetch logs from /var/tempest or /var/vcap/sys.

For more information about the syslog feature, see Configure an Ops Manager Syslog Server in the PCF Ops Manager v2.2 Release Notes.

Ops Manager API Does Not Return Secret Properties

For security, Ops Manager API responses from the /api/v0/staged/director/properties endpoint no longer include the following properties:

  • GCS blobstore service key service_account_key (on Google Cloud Services)
  • Health Monitor emailer SMTP password smtp_password
  • Health Monitor PagerDuty service key service_key

This change may break any code that relies on retrieving these properties from a GET /api/v0/staged/director/properties call.

For more information, see the Ops Manager API v2.2 documentation.

PCF Isolation Segment

Log Aggregation Systems May Parse Diego Timestamps Incorrectly

See New Format for Timestamps in Diego Component Logs in PCF Isolation Segment v2.2 Release Notes and breaking changes for Pivotal Application Service (PAS) for more information.