PCF v2.4 Breaking Changes

Page last updated:

Warning: Pivotal Cloud Foundry (PCF) v2.4 is no longer supported because it has reached the End of General Support (EOGS) phase. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes the breaking changes you need to be aware of when upgrading to Pivotal Cloud Foundry (PCF) v2.4. For more information about important preparation steps you must follow before beginning an upgrade, see Upgrading Pivotal Cloud Foundry.

Pivotal Application Service (PAS)

See the following PAS breaking changes:

Gorouter Always Verifies App Identity

Before upgrading to PAS v2.4, you must secure the Gorouter with TLS or mutual TLS for PAS and Isolation Segment tiles. See the following sections that describe different aspects of this breaking change.

Configuration Change Required

In PAS v2.3 and earlier, you can have insecure routing without TLS. The options for Router application identity verification include the following:

  • Router does not verify application identity
  • Router uses TLS to verify application identity (Default)
  • Router and applications use mutual TLS to verify each other’s identity (incompatible with TCP Routing and SSH access to app containers)

In PAS v2.4, the options are only as follows:

  • Router uses TLS to verify application identity (Default)
  • Router and applications use mutual TLS to verify each other’s identity (incompatible with TCP Routing and SSH access to app containers)

If you do not select either the TLS or mTLS option in PAS v2.3 before upgrading to PAS v2.4, your upgrade will fail.

For further instructions about configuring PAS prior to upgrading, see Configure Gorouter with TLS in the Upgrade Preparation Checklist for PCF 2.4.

Post-Upgrade Operational Impact

For information about the operational impact of this feature on capacity and memory usage, see Increased Resiliency, Consistency, and Security for HTTP Routing in the PAS v2.1 Release Notes.

Impact on Other Features

App identity verification impacts other features in the following ways:

  • Toggling SSL certificate verification
    • TLS and mTLS do not work if the Disable SSL certificate verification for this environment checkbox is selected.
  • TCP Routing
    • If you have mutual TLS app identity verification enabled, app containers accept incoming communication only from the Gorouter. This disables TCP routing.

Removal of Default Ciphers Causes Downtime when Upgrading from 2.2 to 2.3 and from 2.3 to 2.4

This section describes how removing default ciphers can cause downtime and how to avoid or resolve this issue.

In PAS v2.3, communication between Gorouter and CAPI is encrypted using TLS. CAPI uses hardcoded cipher suites for this communication and Gorouter uses cipher suites configured by the operator in the TLS Cipher Suites for Router field in the PAS Networking pane. By default, this field contains the cipher suites required for communication with CAPI.

If you modify the default cipher suites supported by Gorouter using the TLS Cipher Suites for Router field and do not include either of the cipher suites supported by both Gorouter and CAPI, the TLS handshake between Gorouter and CAPI fails. This causes control plane downtime, during which users cannot do things such as push apps and create routes.

To resolve this issue, ensure one of the recommended cipher suites is always in the TLS Cipher Suites for Router field:

  • ECDHE-RSA-AES128-GCM-SHA256 or
  • ECDHE-RSA-AES256-GCM-SHA384

Internal MySQL Databases Run Only on Percona

In PAS v2.2 and v2.3, internal MySQL databases could run on either MariaDB or Percona servers. PAS v2.4 no longer uses MariaDB, so you must migrate internal MariaDB databases to Percona and redeploy PAS v2.3 before upgrading to PAS v2.4.

To migrate your databases to Percona, see Migrating to Internal Percona MySQL.

Removed Configuration for Internal MySQL Load Balancer

In PAS 2.3 and earlier, Operators who wanted a HA internal MySQL database for PAS platform apps could configure an internal load balancer in front of the MySQL proxies. This option was configured with the MySQL Proxy IPs and MySQL Service Hostname fields in the Internal MySQL pane of the PAS tile.

Starting in PAS v2.4, platform apps connect to the internal MySQL DB using BOSH DNS instead. Operators are no longer required to specify an LB in the Internal MySQL pane.

After upgrading to PAS 2.4 and re-running the PAS errands that push apps, you can delete the internal MySQL load balancer as PAS no longer uses it.

Changed “deployment” Field Value for PAS Metrics

If you have scripts that rely on metrics with cf as the value for deployment, your scripts may break. PAS v2.4 replaces the cf value in metrics with cf-GUID, which corresponds to the BOSH deployment name of your PAS tile. This feature is enabled by default in new deployments of PAS 2.4, but disabled by default for PAS deployments upgrading to PAS 2.4.

To resolve this issue, you can re-enable the cf value by selecting Use “cf” as deployment name in emitted metrics instead of unique name in the Advanced Features pane of your PAS tile. If this option is toggled and PCF Healthwatch is installed, enable the PCF Healthwatch Push Monitoring Components Errand to ensure correct PCF Healthwatch configuration.

Pivotal does not recommend re-enabling the cf value as a permanent solution. If you can resolve script dependencies, resolve them and leave the default setting.

HAProxy Defaults to Zero Instances

PAS and PCF Isolation Segment (IST) v2.4 do not deploy the optional HAProxy component by default. Previously, IST deployed three instances of HAProxy by default and PAS deployed one.

This is a breaking change if you are using HAProxy with the Automatic setting for your instance count. On upgrading to PAS or IST v2.4, you will have zero HAProxy instances unless you specify an instance count other than the Automatic setting. For example, if you have the instance count set to Automatic: 3 in IST, modify the instance count to 3 in the Resource Config pane before you upgrade.

Updated Configuration for UAA and CredHub Databases

If your PAS v2.3 tile is configured to use Internal MySQL for its UAA or CredHub databases, PAS modifies this configuration to PAS database when upgrading to v2.4.

In the PAS v2.4 tile, you can configure CredHub and UAA to use the same external database as other PAS components. If you select External Databases - (e.g., AWS RDS) on the Databases pane of the PAS tile, select PAS database on the CredHub or UAA panes to use the same external database.

To use a separate external database for runtime CredHub or UAA, select Other external database on the CredHub or UAA pane of the PAS tile and provide a username and password for the database.

Note: If you use GCP, you cannot use an external database for CredHub. For more information, see CredHub Database Cannot Be External on GCP.

Default External CredHub Does Not Deploy on GCP

Runtime CredHub is enabled by default with an instance count of two. Previously, its default instance count was zero.

On GCP, a CredHub instance count greater than zero causes a deploy error when your PAS system databases are external and CredHub is configured to use the system database location. Deploying PAS on GCP generates an Unable to render jobs for instance group 'credhub' error because the CredHub database cannot be external on GCP. For more information, see CredHub Database Cannot Be External on GCP.

To avoid this error on GCP while using external PAS system databases, include the following PAS configuration changes before you click Apply Changes to deploy:

  • Set the CredHub instance count in the PAS tile Resource Config pane to zero.
  • Paste dummy values into the CredHub pane > Encryption Keys > Name and Key fields.

To store credentials securely in an external CredHub database on other IaaSes, select a database for CredHub in the PAS tile CredHub pane and leave the CredHub instance count at its default value, 2. For more information about selecting a database for CredHub, see the Updated Configuration for UAA and CredHub Databases breaking change.

cflinuxfs3 is the Default Stack

PAS v2.3 introduced the cflinuxfs3 stack alongside the cflinuxfs2 default. In PAS v2.4, cflinuxfs3 is the default stack. This means that newly-created Linux apps use the cflinuxfs3 stack instead of cflinuxfs2, unless specified otherwise with the -s flag. Existing apps continue to use their current stack.

You can modify your default stack in the Cloud Controller pane of the PAS tile. For more information, see Cloud Controller.

If you have not already, Pivotal recommends migrating your apps to cflinuxfs3 as cflinuxfs2 is based on Ubuntu 14.04 (Trusty Tahr), which reaches end of general support (EOGS) in April 2019. After that date, it will no longer receive security updates. For more information, see cflinuxfs3 Stack and Compatible Buildpacks in the PAS v2.3 release notes.

Improved Grootfs Garbage Collection

Operators can configure container filesystem garbage collection based on the disk usage of other jobs in their deployment rather than a garbage collection threshold. PAS and IST v2.4 replace the Clean up disk-space once threshold is reached option with Clean up disk-space once usage fills disk in the Application Containers pane.

To understand how this field is handled during upgrade, see the following table:

Field Selected in v2.3 Upgrade Notes
Never clean up Cell disk-space When you upgrade to v2.4, this setting remains the same.
Routinely clean up Cell disk-space When you upgrade to v2.4, this setting remains the same.
Clean up disk-space once threshold is reached If this field is set to the default value in v2.3, it will be migrated to the default value of the new Clean up disk-space once usage fills disk field upon upgrade to v2.4. If this field has another value besides the default, you must either enter the default or select one of the other options to proceed with your upgrade.

Increased CPU on File Storage VM

To improve reliability, PAS v2.4 increases the default and minimum CPU core count from one to two for the optional internal File Storage VM.

If you are using the internal File Storage VM with a VM type that has a single CPU core, PAS will migrate it to a VM type with two CPU cores during the upgrade.

Log Cache API Changes and Autoscaler Downtime

PAS v2.4 uses log-cache API v2, which moves /v1 endpoints to /api/v1.

One of the apps that depends on the log-cache API is Autoscaler. If you are upgrading from PAS v2.3.1, Autoscaler will be intermittently unusable during the upgrade. However, v2.3.2 includes the log-cache API v2, so there is no downtime when upgrading from v2.3.2 or later.

Consul Server Removed

The instance count for the Consul server is zero. This value cannot be configured. The .consul_server.instances resource configuration is also removed.

As a result, you must update any automations that rely on the .consul_server.instances resource configuration.

PAS tiles that depend on the Consul BOSH link should continue to function properly. However, Pivotal recommends removing any dependencies on the Consul BOSH link from PAS tiles.

PCF Ops Manager

See the following PCF Ops Manager breaking changes:

Excluded Recursors API Field Has Moved

The excluded_recursors field is now grouped under dns_configuration instead of director_configuration when you use the GET /api/v0/staged/director/properties API endpoint.

This API endpoint fetches the BOSH Director, IaaS, and security properties. If you have scripts that rely on the GET output of this endpoint, they may break because of this moved field.

For more information about the API endpoint, see Staged BOSH Director in the Ops Manager API documentation.

Ops Manager Removes Support for RELP

Reliable Event Logging Protocols (RELPs) are not in common use by PCF customers. Ops Manager v2.4 removes RELP support to refine engineering focus and streamline future development.

Pivotal recommends that RELP users switch to TCP.

PAS for Windows

See the following PAS for Windows breaking changes:

Changed “deployment” Field Value in PAS for Windows Metrics

In new installations of v2.4, metrics emitted from PAS for Windows are now tagged with the BOSH deployment name of your PAS for Windows tile. Previously, the cf tag was used as the deployment name in PAS for Windows metrics. For more information, see Uniquely Identify Metrics from PAS for Windows in PAS for Windows v2.4 Release Notes.

If you have scripts that rely on metrics with cf as the value for deployment, your scripts may break. To resolve this issue, you can re-enable the cf value in the Advanced Features pane of your PAS for Windows tile. Pivotal does not recommend re-enabling the cf value as a permanent solution.

For upgrades to PAS for Windows v2.4, the cf value is enabled by default.

MySQL for PCF

MySQL for PCF v1 is not compatible with PCF v2.4. For more information, see Upgrade MySQL for PCF from v1 to v2.