PCF v2.3 Breaking Changes
This topic describes the breaking changes you need to be aware of when upgrading to Pivotal Cloud Foundry (PCF) v2.3. For more information about important preparation steps you must follow before beginning an upgrade, see Upgrading Pivotal Cloud Foundry.
A fix from NFS v1.3.1 has been backported to NFS v1.2.1 to resolve an incompatibility between the NFS volume release and Diego’s container runtime, Garden. Because the fix is in the NFS Service Broker, and service bindings created by old versions of this broker are not migrated during an upgrade, existing NFS service bindings that specify read-only mounts still exhibit the incompatibility. As a result, apps with NFS services bound as read-only crash after upgrading from NFS v1.2.0 or earlier.
To fix this condition, you should unbind the service, rebind it, and restage the app.
Alternately, to avoid app downtime, you can temporarily rebind your apps as read/write before upgrading and switch to read-only afterward.
When you enable the Router and applications use mutual TLS to verify each other’s identity option in PAS v2.3, app containers accept incoming communication only from the Gorouter. This disables
cf ssh and TCP routing.
In PAS v2.3, traffic between the Gorouter and Cloud Controller is encrypted. To prevent app downtime while upgrading from v2.2 to v2.3, you must be running PAS v2.2.4 or later. These versions of PAS v2.2 contain the configuration
router.backends.enable_tls: true in the Gorouter manifest.
This section describes how removing default ciphers can cause downtime and how to avoid or resolve this issue.
In PAS v2.3, communication between Gorouter and CAPI is encrypted using TLS. CAPI uses hardcoded cipher suites for this communication and Gorouter uses cipher suites configured by the operator in the TLS Cipher Suites for Router field in the PAS Networking pane. By default, this field contains the cipher suites required for communication with CAPI.
If you modify the default cipher suites supported by Gorouter using the TLS Cipher Suites for Router field and do not include either of the cipher suites supported by both Gorouter and CAPI, the TLS handshake between Gorouter and CAPI fails. This causes control plane downtime, during which users cannot do things such as push apps and create routes.
To resolve this issue, ensure one of the recommended cipher suites is always in the TLS Cipher Suites for Router field:
The name of the
backup-prepare instance group has been changed to
backup_restore. This may break some external tools that were setting the properties for this instance group through the Ops Manager API. See the article PAS backup-prepare job renamed to backup-restore may cause automation failures in the Pivotal Knowledge Base for details.
In PCF v2.3, you cannot opt out of GrootFS. Upgrades to PCF v2.3 will fail if GrootFS is disabled in PCF v2.2.
If you did not enable GrootFS prior to PAS v2.3, you may encounter issues with apps deployed with a Docker image. GrootFS fixes a bug in which disk quotas were not accurately calculated. However, this bug fix may cause Docker images that previously adhered to a quota to now exceed the same quota size. See Disk quota calculation in the
You may observe the following error in the Diego Cell Garden logs:
cfnetworking: cni up failed: add network list failed: initialize net out: creating chain: iptables call: running [/var/vcap/packages/iptables/sbin/iptables -t filter -N netout--executor-healthcheck --wait]: exit status 1: iptables: Chain already exists.
This is a symptom of a failed Diego healthcheck, which can cause the Diego cell to become unhealthy. If you encounter this error, see the following Pivotal Knowledge Base article to understand and resolve the issue: Diego Cell Garden healthcheck fails and becomes unhealthy.
Starting in PCF v2.3, Ops Manager uses a Xenial stemcell. Before you upgrade to PCF v2.3, you must do the following:
If you are using any of the BOSH add-ons for PCF, you must update the add-on and its configuration to include the Ubuntu Xenial 16.04 stemcell. The following add-on versions support the Xenial stemcell:
- ClamAV Add-on for PCF v1.4.28 and later. For information on how to update this add-on, see Updating ClamAV Add-on for PCF to Run with Xenial Stemcells.
- File Integrity Monitoring (FIM) Add-on for PCF v1.2.22 and later. For information on how to update this add-on, see Updating FIM Add-on for PCF to Run with Xenial Stemcells.
- IPSec Add-on for PCF v1.9.9 and later. For information on how to update this add-on, see Updating IPsec Add-on for PCF to Run with Xenial Stemcells.
You must apply these updates before you deploy any tiles that use the Xenial stemcell and before you upgrade to PCF v2.3.
Verify that any tiles that you have installed in your current foundation, either from Pivotal or a partner, are compatible with PCF v2.3. For more information on checking PCF and stemcell compatibility, see Tile Compatibility in the Upgrade Checklist.
In some cases, you may need to download and import a new Xenial stemcell into Ops Manager v2.2 to upgrade your tiles. To download a Xenial stemcell from Pivotal Network, visit Stemcells for PCF (Ubuntu Xenial).
For information on which PCF tile releases now use Xenial, see Tiles Using Xenial Stemcells in PCF.
In PCF v2.3, you cannot opt out of BOSH DNS. Upgrades to PCF v2.3 will fail if BOSH DNS is disabled in PCF v2.2.
If your PCF deployment is on vSphere and uses NSX-T load balancers, upgrading from PCF v2.2 to PCF v2.3 may cause downtime.
This is due to a new feature in the vSphere CPI that Ops Manager uses, which introduces the capability to manage the full lifecycle of the membership of the NSX-T load balancer.
For more information, see Safely Upgrading PAS 2.2 → 2.3 with NSX-T Load Balancers.
For security, Ops Manager API responses from the
/api/v0/staged/director/properties endpoint no longer include the following properties:
- GCS blobstore service key
service_account_key(on Google Cloud Services)
- Health Monitor emailer SMTP password
- Health Monitor PagerDuty service key
This change may break any code that relies on retrieving these properties from a
GET /api/v0/staged/director/properties call.
For more information, see the Ops Manager API v2.3 documentation.
If you are using the ClamAV Add-on for PCF, you must update your runtime config before upgrading to PASW v2.3.
If you upgrade to PASW v2.3 without first updating the runtime config, then your Windows VMs will not be protected by ClamAV.
For more information about how to update your runtime config, see Updating ClamAV Add-on for PCF to Run with PAS for Windows v2.3 in the ClamAV documentation.
There are no breaking changes for PCF Isolation Segment at this time.