PCF v2.3 Breaking Changes

Page last updated:

Warning: Pivotal Cloud Foundry (PCF) v2.3 is no longer supported because it has reached the End of General Support (EOGS) phase. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes the breaking changes you need to be aware of when upgrading to Pivotal Cloud Foundry (PCF) v2.3. For more information about important preparation steps you must follow before beginning an upgrade, see Upgrading Pivotal Cloud Foundry.

Pivotal Application Service (PAS)

Read-Only Volume Mounts

A fix from NFS v1.3.1 has been backported to NFS v1.2.1 to resolve an incompatibility between the NFS volume release and Diego’s container runtime, Garden. Because the fix is in the NFS Service Broker, and service bindings created by old versions of this broker are not migrated during an upgrade, existing NFS service bindings that specify read-only mounts still exhibit the incompatibility. As a result, apps with NFS services bound as read-only crash after upgrading from NFS v1.2.0 or earlier.

To fix this condition, you should unbind the service, rebind it, and restage the app.

Alternately, to avoid app downtime, you can temporarily rebind your apps as read/write before upgrading and switch to read-only afterward.

Mutual TLS App Identity Verification Disables CF SSH and TCP Routing

When you enable the Router and applications use mutual TLS to verify each other’s identity option in PAS v2.3, app containers accept incoming communication only from the Gorouter. This disables cf ssh and TCP routing.

App Downtime When Upgrading from v2.2.3 or Earlier

In PAS v2.3, traffic between the Gorouter and Cloud Controller is encrypted. To prevent app downtime while upgrading from v2.2 to v2.3, you must be running PAS v2.2.4 or later. These versions of PAS v2.2 contain the configuration router.backends.enable_tls: true in the Gorouter manifest.

Removal of Default Ciphers Causes Downtime when Upgrading from 2.2 to 2.3 and from 2.3 to 2.4

This section describes how removing default ciphers can cause downtime and how to avoid or resolve this issue.

In PAS v2.3, communication between Gorouter and CAPI is encrypted using TLS. CAPI uses hardcoded cipher suites for this communication and Gorouter uses cipher suites configured by the operator in the TLS Cipher Suites for Router field in the PAS Networking pane. By default, this field contains the cipher suites required for communication with CAPI.

If you modify the default cipher suites supported by Gorouter using the TLS Cipher Suites for Router field and do not include either of the cipher suites supported by both Gorouter and CAPI, the TLS handshake between Gorouter and CAPI fails. This causes control plane downtime, during which users cannot do things such as push apps and create routes.

To resolve this issue, ensure one of the recommended cipher suites is always in the TLS Cipher Suites for Router field:

  • ECDHE-RSA-AES128-GCM-SHA256 or

backup-prepare Instance Group Has Been Renamed to backup_restore

The name of the backup-prepare instance group has been changed to backup_restore. This may break some external tools that were setting the properties for this instance group through the Ops Manager API. See the article PAS backup-prepare job renamed to backup-restore may cause automation failures in the Pivotal Knowledge Base for details.

Newly-Enabled GrootFS May Require Higher Disk Quotas for Docker Apps

In PCF v2.3, you cannot opt out of GrootFS. Upgrades to PCF v2.3 will fail if GrootFS is disabled in PCF v2.2.

If you did not enable GrootFS prior to PAS v2.3, you may encounter issues with apps deployed with a Docker image. GrootFS fixes a bug in which disk quotas were not accurately calculated. However, this bug fix may cause Docker images that previously adhered to a quota to now exceed the same quota size. See Disk quota calculation in the garden-runc-release repository.

Diego Cell Garden Healthcheck Fails and Becomes Unhealthy

You may observe the following error in the Diego Cell Garden logs:

cfnetworking: cni up failed: add network list failed: initialize net out: creating chain: iptables call: running [/var/vcap/packages/iptables/sbin/iptables -t filter -N netout--executor-healthcheck --wait]: exit status 1: iptables: Chain already exists.

This is a symptom of a failed Diego healthcheck, which can cause the Diego cell to become unhealthy. If you encounter this error, see the following Pivotal Knowledge Base article to understand and resolve the issue: Diego Cell Garden healthcheck fails and becomes unhealthy.

App Autoscaler v1 API Endpoints are Unsupported

The App Autoscaler v1 API is no longer supported. If you are using any v1 API functionality, you must update to the v2 API.

NSX Container Plugin v2.4.1 is Not Compatible with PAS v2.3

PAS v2.3 is not compatible with VMware NSX Container Plugin v2.4.1.

To upgrade the NSX Container Plugin v2.4.0 to v2.4.1, first upgrade to PAS v2.4 or later.

For more information, see NSX Container Plugin 2.4.1 Release Notes in the VMware documentation.

PCF Operations Manager (Ops Manager)

Updates for Xenial Stemcell Support

Starting in PCF v2.3, Ops Manager uses a Xenial stemcell. Before you upgrade to PCF v2.3, you must do the following:

  1. If you are using any of the BOSH add-ons for PCF, you must update the add-on and its configuration to include the Ubuntu Xenial 16.04 stemcell. The following add-on versions support the Xenial stemcell:

  2. Verify that any tiles that you have installed in your current foundation, either from Pivotal or a partner, are compatible with PCF v2.3. For more information on checking PCF and stemcell compatibility, see Tile Compatibility in the Upgrade Checklist.

    In some cases, you may need to download and import a new Xenial stemcell into Ops Manager v2.2 to upgrade your tiles. To download a Xenial stemcell from Pivotal Network, visit Stemcells for PCF (Ubuntu Xenial).

For information on which PCF tile releases now use Xenial, see Tiles Using Xenial Stemcells in PCF.

BOSH DNS Must be Enabled

In PCF v2.3, you cannot opt out of BOSH DNS. Upgrades to PCF v2.3 will fail if BOSH DNS is disabled in PCF v2.2.

Bosh dns error

Downtime when Upgrading on vSphere with NSX-T

If your PCF deployment is on vSphere and uses NSX-T load balancers, upgrading from PCF v2.2 to PCF v2.3 may cause downtime.

This is due to a new feature in the vSphere CPI that Ops Manager uses, which introduces the capability to manage the full lifecycle of the membership of the NSX-T load balancer.

For more information, see Safely Upgrading PAS 2.2 → 2.3 with NSX-T Load Balancers.

Ops Manager API Does Not Return Secret Properties

For security, Ops Manager API responses from the /api/v0/staged/director/properties endpoint no longer include the following properties:

  • GCS blobstore service key service_account_key (on Google Cloud Services)
  • Health Monitor emailer SMTP password smtp_password
  • Health Monitor PagerDuty service key service_key

This change may break any code that relies on retrieving these properties from a GET /api/v0/staged/director/properties call.

For more information, see the Ops Manager API v2.3 documentation.

Pivotal Application Service for Windows (PASW)

Update Runtime Config of ClamAV for PASW v2.3

If you are using the ClamAV Add-on for PCF, you must update your runtime config before upgrading to PASW v2.3.

If you upgrade to PASW v2.3 without first updating the runtime config, then your Windows VMs will not be protected by ClamAV.

For more information about how to update your runtime config, see Updating ClamAV Add-on for PCF to Run with PAS for Windows v2.3 in the ClamAV documentation.

PCF Isolation Segment

There are no breaking changes for PCF Isolation Segment at this time.