PCF v2.0 Breaking Changes

Page last updated:

Warning: Pivotal Cloud Foundry (PCF) v2.0 is no longer supported because it has reached the End of General Support (EOGS) phase. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes the breaking changes you need to be aware of when upgrading to Pivotal Cloud Foundry (PCF) v2.0. For more information about important preparation steps you must follow before beginning an upgrade, see Upgrading Pivotal Cloud Foundry.

Pivotal Application Service (PAS)

Temporary Downtime Due to Connection Failure between HAProxy and Gorouter

PCF v2.0 introduces mutual TLS authentication between HAProxy and the Gorouter. During upgrade to v2.0, the HAProxy component upgrades before the Gorouter, and in the few minutes between when HAProxy upgrades and the Gorouter upgrades, the two components cannot communicate. This is because the Gorouter cannot yet validate the client certificate sent by HAProxy, causing a TLS handshake failure. If you are using HAProxy, this may cause temporary outage for PCF apps, but in the situations described below, you can avoid downtime with the following upgrade procedure:

  1. Change the Router behavior for Client Certificate Validation setting to Router does not request client certificates…
  2. Upgrade (Apply Changes).
  3. Change your Router behavior… setting back to its previous setting.
  4. Upgrade again to restore previous router setting.

The above procedure avoids downtime under the following conditions:

  • Your Configure support for the X-Forwarded-Client-Cert header setting is TLS terminated for the first time at infrastructure load balancer and your Router behavior… setting is Router requests… or Router requires… client certificates.

  • Your Configure support for the X-Forwarded-Client-Cert header setting is either TLS terminated for the first time at HAProxy or TLS terminated for the first time at the Router, and you do not forward certificates to client apps (because no apps on the platform require mTLS).

If your Configure support for the X-Forwarded-Client-Cert header setting is TLS terminated for the first time at HAProxy and you forward certificates to client apps because they require mTLS, there is currently no path for a zero downtime upgrade. Pivotal is working on a patch to enable one.

If you still want to upgrade to PCF v2.0 and can allow a few minutes of downtime, you can follow the procedure above. Apps that use the XFCC header become inaccessible until the second upgrade step.

SSO Tile Requires Upgrade

If you have the Single Sign-On (SSO) tile installed and you are upgrading to PCF v2.0, you must upgrade the SSO tile to v1.5.3 and configure the Apps Manager errand before installing PCF v2.0. To properly prepare for upgrading to PCF 2.0 with the SSO tile, see Upgrade SSO Service Tile resource instance between PCF 1.12 and PCF 2.0 in the Pivotal Knowledge Base.

WARNING: You must upgrade PCF Operations Manager (Ops Manager) before upgrading PAS (formerly Pivotal Elastic Runtime). Ops Manager v2.0 is required when upgrading from Elastic Runtime v1.12 to a PCF PAS v2.0 deployment that has the SSO v1.5.3 tile installed.

Internal Pivotal Application Service Credentials

Several internal credentials that PAS uses for inter-component communication are now generated and stored in BOSH CredHub instead of Ops Manager. If you want to access these credentials, you must use the CredHub CLI or the Ops Manager API instead of accessing the Credentials tab of the PAS tile.

For a list of the credentials migrated to BOSH CredHub, see the Migration of Internal Credentials to BOSH CredHub section of the Pivotal Application Service v2.0 Release Notes topic.

If you opt out of the BOSH DNS feature, your PCF deployment cannot support Secure Service Instance Credentials feature.

Upgrading with SAML and SHA1 as the Signature Algorithm

PAS does not support SHA1 to sign outbound SAML requests.

If you use SHA1 with PCF v1.12, PAS automatically migrates your SAML signature algorithm to SHA256. If you are using SAML in PCF v1.12 through Elastic Runtime or the SSO tile and if your Identity Provider validates the signature of the SAML authentication request, see SHA Signature Support for SAML requests in PAS tile in PCF 2.0 in the Pivotal Knowledge Base.

Upgrading from Earlier Versions of Elastic Runtime

You must upgrade first to a version of Elastic Runtime v1.12 to successfully upgrade to PAS v2.0.

If you are upgrading from a PCF deployment that at one point included Elastic Runtime v1.7.16 or earlier, you must perform the remedial steps outlined in App Usage Data and Events Data Become Corrupted After Upgrade or Install before proceeding with this upgrade. If you fail to perform the remedial steps for this issue, this upgrade process may corrupt your existing usage data.

Some Upgrades Require VM Re-Creation

Due to an update to garden-runc-release, Pivotal recommends that you re-create all VMs when you upgrade from the following older versions to current versions of PAS and PCF Isolation Segment (IST) and you do not also update your stemcell:

  • PAS, Small Footprint Runtime, and Elastic Runtime:

    • v2.0.0 - v2.0.6
    • v1.12.0 - v1.12.15
    • v1.11.0 - v1.11.27
    • v1.10.x, 1.9.x, 1.8.x
  • PCF Isolation Segment:

    • v2.0.0 - v2.0.5
    • v1.12.0 - v1.12.14
    • v1.11.0 - v1.11.25
    • v1.10.x

VMs re-create automatically when you update your stemcell. Otherwise, you can re-create all VMs for both PAS and IST by enabling the Recreate All VMs checkbox on the Director Config pane in the PAS tile.

Configuring Router Behavior for Client Certificates while Deploying PAS

For some deployment configurations, you must configure how PCF handles x-forwarded-client-cert (XFCC) HTTP headers based on where TLS is terminated for the first time in your deployment. In the Router behavior for Client Certificates field, you cannot select the Router does not request client certificates option.

Read-only Volume Mounts

We back-ported a fix from NFS 1.3.1 to NFS 1.2.1 for an incompatibility between our NFS Volume release and Diego’s container runtime, garden. But, because the fix was in the NFS Service Broker, and service bindings created by old versions of this broker won’t get migrated during upgrade, existing NFS service bindings that specify read-only mounts will still exhibit the incompatibility.

As a result, customers upgrading from versions containing nfs-volume-release < 1.2.1 that have NFS services bound read-only to their applications will see that their applications crash after upgrade.

To fix this condition, customers should unbind the service, rebind it, and then restage the application.

Alternately, customers wishing to avoid application down time can temporarily re-bind their applications as read/write before upgrading, and then swtich to read-only afterwards.

Diego Cell Garden Healthcheck Fails and Becomes Unhealthy

You may observe the following error in the Diego Cell Garden logs:

cfnetworking: cni up failed: add network list failed: initialize net out: creating chain: iptables call: running [/var/vcap/packages/iptables/sbin/iptables -t filter -N netout--executor-healthcheck --wait]: exit status 1: iptables: Chain already exists.

This is a symptom of a failed Diego healthcheck, which can cause the Diego cell to become unhealthy. If you encounter this error, see the following Pivotal Knowledge Base article to understand and resolve the issue: Diego Cell Garden healthcheck fails and becomes unhealthy.

Ops Manager

BOSH CLI Renamed

Similar to previous Ops Manager versions, the Ops Manager virtual machine includes both versions of the BOSH CLI. In Ops Manager v2.0, both versions of the BOSH CLI have been renamed.

If you used BOSH CLI v2 in earlier versions of Ops Manager, you ran commands using bosh2. In Ops Manager v2.0.0, run the same commands using bosh. For example, see the following table to compare the changes to the bosh vms command:

BOSH CLI Version PCF v1.12 PCF v2.0
BOSH CLI v1 bosh vms bosh-old vms
BOSH CLI v2 bosh2 -e MY-ENV vms bosh -e MY-ENV vms

Many BOSH CLI v1 commands are incompatible with the BOSH Director. Pivotal recommends using BOSH CLI v2 commands for compatibility with future versions of PCF.

Missing Stemcell Causes Failure to Deploy

In PCF v1.12 and earlier, the BOSH Director may delete stemcells required by errands. This causes upgrades from these versions to fail with Error: Stemcell doesn't exist. To prevent this error, do the following before you click Apply changes in Ops Manager to upgrade:

  1. Download a current stemcell from Pivotal Network.

  2. Upload the stemcell in the Stemcell pane of a given product. You can also manually run bosh upload-stemcell with the BOSH CLI.

See the Pivotal Knowledge Base article Deploy fails with Error: Stemcell doesn’t exist for details.

This known issue has been fixed in Ops Manager v2.0 and later, but it affects upgrades from v1.12 to v2.0.

Disabling BOSH DNS for Ops Manager Director

Do not disable BOSH DNS without consulting Pivotal Support.

For more information about disabling BOSH DNS, see Disabling or Opting Out of BOSH DNS in PCF in the Pivotal Knowledge Base.

current_record Property

The current_record property is now reserved. You can no longer create a new property named current_record in a collection record.

Azure Stack Support is in Beta for Ops Manager

Operators can deploy Ops Manager v2.0 to Microsoft Azure in their own local datacenter using Azure Stack. Azure Stack support is in beta for Ops Manager v2.0 and should not be used in production.

Using Azure Stack in production deployments may result in unpredictable behavior.

PCF Isolation Segment

Configuring Router Behavior for Client Certificates While Installing PCF Isolation Segment

For some deployment configurations, you must configure how PCF handles x-forwarded-client-cert (XFCC) HTTP headers based on where TLS is terminated for the first time in your deployment. In the Router behavior for Client Certificates field, you cannot select the Router does not request client certificates option.