LATEST VERSION: 2.1 - CHANGELOG

Pivotal Cloud Foundry v1.12

• v2.1
• v2.0
• v1.11
• v1.10
• v1.9
• v1.8

Download Ask for Help PCF Knowledge Base PDF

Pivotal Elastic Runtime v1.12 Release Notes

Pivotal Cloud Foundry is certified by the Cloud Foundry Foundation for 2018.

Read more about the certified provider program and the requirements of providers.

Releases

1.12.24

• [Security Fix] Bump cflinuxfs2 to version 1.210.0:
  • USN-3643-1
• Update grootfs checkbox to indicate the recreating VMs is recommended
• Bump capi to version 1.40.54
  • Updated azure fog gems to improve reliability when using an azure blobstore
• Bump cf-networking to version 1.4.3
• Bump nats to version 24
  • Bump go to 1.10.1
• Bump push-apps-manager-release to version 662.0.34
  • Usage report page takes into account renamed spaces
  • Fix bug that causes app to crash on app page settings tab
• Bump java-offline-buildpack to version 4.12

1.12.23

• [Security Fix] Bump stemcell to v3468.42:
  • USN-3641
  • USN-3631-2
  • USN-3628-1
  • USN-3625-1
  • USN-3624-1
• [Security Fix] Bump cflinuxfs2-release to v1.201.0:
  • USN-3628-1
  • USN-3625-1
  • USN-3624-1
  • USN-3622-1
• [Feature Improvement] Bump routing-release to v0.163.14 to enable operator to disable logging of client IPs, in compliance with the EU General Data Protection Regulation (GDPR).
• [Feature Improvement] Bump apps-manager-release to v662.0.33:
  • When binding a service instance, notify the user to restage their app from the CLI.
  • When logged-in user can see no apps, show “No results” instead of “Loading…” in the app search.
• [Bug Fix] Provide the Ops Manager root CA certificate and any other operator-provided trusted certificates to all containers in the /etc/cf-system-certificates directory.
• [Bug Fix] Bump loggregator-release to v96.2 to prevent Traffic Controller from failing when consul DNS is stopped first during a BOSH stop or restart.
• Bump mysql-monitoring-release to v8.18.0.
• Bumps the following buildpacks:
  • Nodejs-offline-buildpack to v1.6.23.
  • Php-offline-buildpack to v4.3.54.
  • Python-offline-buildpack to v1.6.15.
  • Ruby-offline-buildpack to v1.7.18.

1.12.22

• [Security Fix] Bumps garden-release to v1.13.1 for CVE-2018-1277.
• [Bug Fix] When upgrading from Elastic Runtime v1.11 to v.12, the Enable secure communication between Diego and Cloud Controller option in the Cloud Controller tab should be disabled by default, instead of enabled by default.
• [Bug Fix] Bumps autoscaling-release to v96.2 to use CF CLI v6.36.1.
• [Bug Fix] Bumps capi-release to v1.40.53 to prevent duplicate app usage events.
• [Feature Improvement] Bumps diego-release to v1.25.13 to add cell and instance identifiers in the container lifecycle logs.
• [Feature Improvement] Bumps apps-manager-release to v662.0.32:
  • Introduce custom memory limit setting for Apps Manager and invitation apps.
  • Show full page error when critical env vars are not set.
  • App last push time now reflects time of most recent ready package.
  • Introduce flag to hide app search bar.
  • App search bar queries apps only when focused.
  • Tell user to re-stage app after binding a service.
• Bumps the following buildpacks:
  • Binary-offine-buildpack to v1.0.18.
  • Dotnet-core-offline-buildpack to v2.0.6.
  • Go-offline-buildpack to v1.8.21.
  • Java-offline-buildpack to v4.10.0.
  • Nodejs-offline-buildpack to v1.6.22.
  • Php-offline-buildpack to v4.3.53.
  • Python-offline-buildpack to v1.6.14.
  • Ruby-offline-buildpack to v1.7.16.
  • Staticfile-offline-buildpack to v1.4.27.

1.12.21

• [Bug Fix] When upgrading from Elastic Runtime v1.11 to v1.12, the Enable secure communication between Diego and Cloud Controller option in the Cloud Controller tab should be disabled by default, instead of enabled by default. Only new installations of Elastic Runtime v1.12 should enable secure communication by default.
• [Security Fix] Bumps cflinuxfs2 to v1.196.0:
  • USN-3611-1
  • USN-3610-1
• [Security Fix] Bumps stemcell to v3468.30:
  • USN-3619-2
  • USN-3611-1
  • USN-3610-1
  • USN-3598-1
  • USN-3586-1
  • USN-3584-1
• [Bug Fix] Bumps syslog-migration-release to v8.0.2:
  • Prevent logs from blackbox from being written to the default syslog log files to prevent logs from being written to the disk 3 additional times.
  • Fix rfc5424 compatibility by ensuring only 1 space occurs between the message and the structured data.
• [Bug Fix] Fixes a bug that caused the Cloud Controller sync job to fail when pushing an app with TCP routing enabled, which causes Diego to not know if its desired state is consistent with Cloud Controller.
• [Feature Improvement] Bumps capi-release to v1.40.52 to improve database connection validation.
• [Feature Improvement] Adds field Custom syslog Configuration to specify custom logging rules in the System Logging tab. For more information, see custom syslog rules.

1.12.20

• [Bug Fix] Bumps capi-release to v1.40.51 to:
  • Prevent app upload from failing when the app has broken symlinks.
  • Fix broken cf ssh for Docker apps.
• [Bug Fix] Bumps cf-mysql-release to v36.11.0. Release Notes
• [Feature Improvement] Bumps mysql-monitoring-release to v8.16.0. Release Notes
• [Feature Improvement] Bumps loggregator-release to v96.0.17 to add stricter app id validation in Traffic Controller.
• [Feature Improvement] The SSO Operator Dashboard now allows plan administrator to send password reset emails.
• [Bug Fix] Bumps push-apps-manager-release to v662.0.28
  • Reintroduce cache busting for js/css files
  • Fixed a bug that would cause apps manager to fail to load when environment variables contained newlines
  • Fix headers for endpoints that we serve
  • Updated the CF CLI that is used to push Apps Manager and Invitations

1.12.19

• [Bug fix] Bumps apps-manager-release to v662.0.25:
  • [IE] Fixes alignment of the app search bar in the header.
  • Fixes a bug that prevented mid-level fetch tasks from being cleared when switching routes and on the 30 second refresh.
  • Fixes a bug that caused marketplace service plans to show “No price available”.
• [Bug fix] Bumps uaa-release to v45.8:
  • Updates JDK version to 8u162.
• [Security Fix] Bumps capi-release to 1.40.49:
  • CVE-2018-1266: Fixes random number guessing exploit.
  • Fixes buildpack pagination.

1.12.18

• [Feature Improvment] Bumps garden-runc-release to v1.12.1:
  • Includes fix for bug where users’ files could go missing in docker-based applications.
• [Bug fix] Bumps routing-release to 0.163.13:
  • Removes backends on any error to prevent 502 errors from being returned to clients.
  • Updates golang to v1.9.4.
• [Bug Fix] Removes unneeded persistent disk from diego brain vms.

1.12.17

• [Feature Improvement] Bumps apps-manager to 662.0.24, which uses nginx and the staticfile buildpack.
• [Bug Fix] Bumps capi-release to version 1.40.47:
  • API no longer loads all users into an array in memory.
• [Bug Fix] Cloud controller is configured to set cc.diego.pid_limit to 0 (unlimited) so that application instances which created many threads do not crash. The previous limit was defaulting to 1024.

1.12.16

Note: it is recommended that you re-create all VMs when upgrading to this release, due to the update to garden-runc-release. This will happen automatically if you are updating your stemcell. If not, you can check the “Recreate All VMs” checkbox on the Ops Manager Director > Director Config tab.

• [Security Fix] Bumps stemcell from version 3468.21 to version 3468.25 to address issues:
  • USN-3582-2
• [Security Fix] Bumps cflinuxfs2-release from v181.0 to v1.188.0 to address issues:
  • USN-3577-1
  • USN-3569-1
  • USN-3554-1
  • USN-3547-1
  • USN-3543-1
  • USN-3540-2
  • USN-3538-1
• [Feature Improvement] Bumps garden-runc-release to v1.11.1 which includes grootfs root filesystem by default.
• [Feature Improvement] Patches cloud controller so users with admin_read_only scope can view stats for apps, which is needed by the cf v3-apps command.
• [Bug Fix] Patches cloud controller nginx http upload module to fix issue where incorrect initialization of the upload path could cause segmentation faults.

1.12.15

• [Security Fix] Patches routing-release for CVE-2018-1221.
• [Bug Fix] Bumps push-usage-service to increase memory footprint, to avoid occasional crashes that some users were seeing.
• [Bug Fix] Enables privileged containers to support upgrading from ERT 1.11 with apps that specify privileged containers.
• [Bug Fix] Fix to ensure that Diego rep will always exit during evacuation, even if Garden destroy hangs during evacuation.
• [Bug Fix] Patches syslog to prevent duplication from blackbox log forwarding.
• [Feature Improvements] Bump mysql-backup-release to v2 in recognition of the fact that v1.38.0 required TLS. See other changes here
• [Feature Improvements] New option in the Networking page to allow operators to enable Gorouter support for the PROXY protocol. This is disabled by default.
• [Feature Improvement] Enable Garden debug_listen_address to listen on a local interface.
• [Feature Improvement] Adds credentials for Healthwatch alerts.

1.12.14

• [Security Fix] Bumps apps-manager-release to v662.0.22 to fix vulnerability that allowed arbitrary file access on server.
• [Bug Fix] Patches diego-release to allow HTTP-based health check on an HTTP endpoint that expects TLS-terminated traffic.
• [Bug Fix] Bumps java-offline-buildpack to v4.8 to address an issue with multiple java-offline-buildpacks being included, which may cause deployments to have different versions of java-offline-buildpack installed.
• Bump buildpacks to latest versions, including:
  • dotnet-core-offline-buildpack to v2.0.1.
  • go-offline-buildpack to v1.8.16.
  • java-offline-buildpack to v4.8.
  • nodejs-offline-buildpack to v1.6.15.
  • php-offline-buildpack to v4.3.48.
  • python-offline-buildpack to v1.6.7.
  • ruby-offline-buildpack to v1.7.11.
  • staticfile-offline-buildpack to v1.4.21.

1.12.13

• [Security Fix] Bumps stemcell to version 3468.21 to address issues:
  • USN-3534-1
  • USN-3540-2
• [Security Fix] Bumps cflinuxfs2-release to v1.181.0 to address issues:
  • USN-3532-1
  • USN-3534-1
  • USN-3535-1
• [Security Fix] Bumps apps-manager-release to v662.0.19
  • Adds new security headers: ‘Strict-Transport-Security’, 'X-Content-Type-Options’, and 'X-XSS-Protection’
• [Security Fix] Patches capi-release to fix issue where refresh tokens are not accepted where access tokens are required.
  • CVE-2018-1195
• [Bug Fix] Bumps mysql-monitoring-release to v8.14.0
• [Bug Fix] Patches capi-release to use delayed job queue to know when a job is in progress
• [Feature Improvement] Bumps syslog-migration-release to v8.0.1 and add a checkbox for log file forwarding through TCP to work around the Truncated Syslog Messages issue.
  • NOTE: Using TCP instead of the default UDP configuration may have a negative impact on performance.

1.12.12

• [Bug Fix] Bumps uaa-release to v45.7.
• [Bug Fix] Patch to allow the BBS to maintain its lock when the MySQL VMs are being upgraded.
• [Bug Fix] Bumps apps-manager-release to v662.0.18 to resolve a number of issues:
  • If instance health is not loaded, do not render row drawer on app status table.
  • When deleting apps, use capi v3 endpoint.
  • Fixed bug where using Docker would crash Apps Manager because of non-existent buildpack info.
  • For app threads tab, handle when there are no app instances.
  • Fixed download of Spring threads on IE.
  • Hide native select dropdown on IE and Firefox.
  • Display formatted cost with all currencies instead of just USD in plan summary.
  • Fixed wiring issue that causes the flyout to always believe non-basic services were not allowed.
  • Fixed select vs upgrade your account button when coming from app services tab panel header.
  • Load app health after scaling.
  • Updated git and buildpack text to match accessibility standards.
  • Show v3 app scaling events on the app page event panel.
  • Load events after scaling app.
  • When a call to /cloudfoundryapplication fails, do not continue to check if the app is a spring app.
  • Add clickjacking protection, while still allowing Apps Manager to load singular.
  • Long org names in the navbar org dropdown are ellipsified.
  • When checking env variables, do not throw if user does not have permission.
  • Space members tab should show all members in the org even if they are not permitted to the space.
  • Fixed 404 page footer in IE.
  • Fixed styling in accounting report download button.
  • Fetch all routes for spaces instead of just the first page.
• [Bug Fix] Adds missing default domain streaming-mysql-backup-tool to mysql-backup certificate. Note: if you installed 1.12.10 or 1.12.11, you will have to rotate certificates. See this KB article for more details: Pivotal Application Service Backup and Restore fails due to Missing Streaming mysql-backup-tool Domain
• [Bug Fix] Bumps pivotal-account-release to v1.8.2 to fix bug that prevented errands from running more than once.
• [Feature Improvement] The SAML 'Entity Id Override’ field has been moved from the Authentication and Enterprise SSO tab to the UAA tab in Ops Manager, to accompany the other SAML fields in the UAA tab.

1.12.11

This release introduces a bug that causes BBR backups to fail due to a missing default domain in the mysql-backup certificate. We recommend skipping this release and upgrading to 1.12.12 or higher, which resolves this issue. See the corresponding Knowledge Base for more information.

• [Security Fix] Bumps stemcell version to 3445.22 for USN-3544-2 and USN-3544-4

1.12.10

This release introduces a bug that causes BBR backups to fail due to a missing default domain in the mysql-backup certificate. We recommend skipping this release and upgrading to 1.12.12 or higher, which resolves this issue. See the corresponding Knowledge Base for more information.

• [Security Fix] Bumps cflinuxfs2-release to v1.176.0 for USN-3513-1.
• [Bug Fix] Resolves an issue in container-networking where a component in the same network with mTLS can cause an sql injection on the DeleteEntry database handler.
• [Bug Fix] Resolves a bug where task states are not updated when droplets are deleted.
• [Feature Improvement] Ops Manager now allows operators to specify an Azure environment name other than the default 'AzureCloud’. The option is in tab File Storage, under the External Azure Storage in the Environment field.
• [Feature Improvement] Bumps mysql-monitoring-release to v8.13.0 to add disk usage metrics as a percentage.
• [Feature Improvement] Bumps mysql-backup-release to v1.38.0 which enables mutual TLS between the backup node and server.
• [Feature] Bumps garden-runc-release to v1.10.0:
  • It is now possible to specify a ProcessSpec.Image. Processes can now have their own filesystem view.
  • Limitation: It is only possible to use ProcessSpec.Image and ProcessSpec.OverrideContainerLimits with unprivileged containers.
    This will be fixed in future releases.
  • Limitation: APIs such as BulkMetrics and Process.Signal may not work immediately after container.Run(ProcessSpec) returns for processes with Image and/or OverrideContainerLimits specified. This will be fixed in future releases.
  • Reduced log volume in BulkMetrics for large environments.
  • Correctly declares that bundles it creates are OCI Runtime Spec version 1.0.0 compliant.

1.12.9

• [Security Fix] Bumps stemcell version to 3445.19 for USN-3509-2.
• [Security Fix] Bumps cflinuxfs2-release to v1.171.0 to resolve several security vulnerabilities:
  • USN-3489-1: Berkeley DB vulnerability
  • USN-3496-1: Python vulnerability
  • USN-3496-3: Python vulnerability
  • USN-3498-1: curl vulnerabilities
  • USN-3501-1: libxcursor vulnerability
• [Bug Fix] Bumps apps-manager-release to v662.0.17 to resolve some bugs:
  • Long org names in the navbar org dropdown are ellipsified.
  • Fix the look of the select component in Firefox.
  • Fix a page crash that could occur when refreshing an app page as a space auditor.
  • Improved the resiliency of the Apps Manager server when a proxy error occurs.
  • Show all org and space members in the space members table on the org/space page members tabs.
• [Bug Fix] Bumps cf-mysql-release to v36.10.0 to finalize a fix for configuration and management of syslog. Release Notes
• [Bug Fix] Bumps mysql-monitoring-release to v8.12.0 to finalize a fix for configuration and management of syslog.
• [Bug Fix] Operators can now optionally disable Router Access logs. This will prevent the Router local disk from becoming filled when the Routers are experiencing increased incoming traffic.
• [Feature Improvement] Operators can now specify the mutual TLS certificate validation behavior for the Router. The Router will request certificates by default and validate them if provided. Operators can optionally configure the Router not to request certificates or to require them with every request.

WARNING: Requests to the platform will fail upon upgrade if your load balancer is configured with client certificates and Gorouter does not have the certificate authority. To mitigate this issue, select Router does not request client certificates for Router behavior for Client Certificate Validation in the Networking pane.

• [Feature Improvement] Operators can now override their SAML Entity ID when configuration SAML as an Identity Provider.

1.12.8

• [Security Fix] Bumps apps-manager-release to v662.0.16 to resolve a number of issues:
  • Upgrades to nodejs v8.0 to resolve a number of security issues.
  • When viewing a Spring App’s Threads tab, and there are no running instances, there is now text to convey this.
  • Fix downloading of Spring threads in Internet Explorer.
  • Fix appearance of select inputs in Internet Explorer.
  • Format service plan costs according to supported currencies in Apps Manager configuration on the space page, services tab
  • Fix bug where paid plans would not be allowed when trying to add a service from the space or app page.
  • When scaling an app, show updated app health more quickly.
  • Show app scaling events in the events panel on the app page.
  • Change color of buildpack text to meet accessibility standards.
  • Prevent Apps Manager from being rendered in an iframe.
• [Security Fix] Bumps buildpack releases versions to pick up security and bug fixes:
  • binary-buildpack v1.0.15
  • dotnet-core-buildpack v1.0.30
  • go-buildpack v1.8.13
  • java-buildpack v4.6
  • nodejs-buildpack v1.6.10
  • php-buildpack v4.3.43
  • python-buildpack v1.6.1
  • ruby-buildpack v1.7.5
  • staticfile-buildpack v1.4.18
• [Security Fix] Bumps the stemcell to v3445.17 to resolve the following security issues:
  • USN-3457-1: curl vulnerability
  • USN-3458-1: ICU vulnerability
  • USN-3464-1: Wget vulnerabilities
  • USN-3469-2: Linux kernel (Xenial HWE) vulnerabilities
  • USN-3475-1: OpenSSL vulnerabilities
  • USN-3478-1: Perl vulnerabilities
  • USN-3485-2: Linux kernel (Xenial HWE) vulnerabilities
• [Security Fix] Bumps cflinuxfs2-release to v1.168.0 to resolve USN-3478-1: Perl vulnerabilities.
• [Security Fix] Patches Cloud Controller to prevent users from being able to create a private subdomain of a route in an organization they do not have access to.
• [Bug Fix] Reverts the previous patche release change to the SAML Entity ID field. The field is once again using http for its URL scheme.
• [Improvement] The custom branding fields for the square logo and favicon are now separate fields.

1.12.7

This release has been pulled due to a regression introduced in the SAML identity provider interface. Please upgrade to 1.12.8 or higher to resolve this issue with the SAML entityID.

• [Security Fix] Bumps cflinuxfs2-release to v1.166.0 to resolve USN-3475-1. Release Notes
• [Bug Fix] Bumps cf-mysql-release to v36.9.0 to resolve an issue where IPsec causes mariadb_ctrl to be left in an Execution Failed state. Release Notes
• [Security Fix] Bumps usage-service-release to v663.0.6 to hide sensitive credential information when the Usage Service deployment errand is run.
• [Security Fix] Bumps grootfs-release to v0.30.0 to resolve CVE-2017-14388. Release Notes.
• [Bug Fix] Changes the scheme for the SAML Entity ID from http to https.

1.12.6

• [Security Fix] Bumps the stemcell to v3445.16 to resolve several security vulnerabilities:
  • USN-3424-1
  • USN-3432-1
  • USN-3434-1
  • USN-3441-1
  • USN-3444-2
• [Security Fix] Bumps the cflinuxfs2-release to v1.165.0 to resolve several security vulnerabilities:
  • USN-3457-1
  • USN-3458-1
  • USN-3464-1
• [Bug Fix] Bumps uaa-release to v45.4 to prevent a denial of service attack against the token revocation endpoint.
• [Bug Fix] Patches loggregator-release to remove the totalReceivedMessageCount metric from the v2 API.
• The logging level for the Cloud Controller, Cloud Controller Worker, and Cloud Controller Clock has been lowered from debug to info. This should help reduce log volume while still logging detailed Cloud Controller information.
• [Bug Fix] Garden is now configured to destroy containers on start. This setting will cause the garden process to remove any containers that are already running when it starts. That action will prevent issues where containers that should no longer be running are left up to run.
• The Router has now been configured to automatically validate and trust certificates issued by the Diego Instance Identity CA.
• The SAML Signature Algorithm field is now configurable outside of the Identity Provider option for SAML. This means that deployments using a non-SAML identity provider can still configure their SAML settings for SSO.
• Bumps the usage-service-release to v663.0.5 to enable resource usage configurations for the usage service. In a future release, this service will be configured to consume less resources on the platform.
• Operators can now opt-in to allowing remote administrator access to the internal MySQL database. This was previously enabled by default in releases prior to 1.11.3. In that release, cf-mysql was bumped to v36. That release brought a number of security improvments, including the ability to prevent remote administrator access to the database. Unfortunately, this was a feature that some operators had come to rely upon. The Elastic Runtime will now allow those operators to enable the feature on a selective basis.

1.12.5

• [Security Fix] Bumps cflinuxfs2-release to v1.161.0 to resolve multiple security issues. Release Notes
• [Bug Fix] Bumps consul-release to v181 to ensure encrypt key rotation only occurs when the key changes.
• [Bug Fix] Resolves an issue with the CloudController drain script that caused failures when running bosh stop. The drain script will now ensure that it prints the exit status of the script to indicate success.

1.12.4

This release includes the new Small Footprint Elastic Runtime. This new product reorganizes the components in the Elastic Runtime into a much smaller deployment. Operators can use the Small Footprint Elastic Runtime to deploy a working Cloud Foundry installation in as few as 4 VMs. See Getting Started with Small Footprint Runtime for more details.

• [Security Fix] Bumps cflinuxfs2-release to v1.158.0 to resolve multiple security issues. Release Notes
• [Bug Fix] Bumps usage-service-release to v663.0.4 to resolve instability caused by low memory constraints.
• [Bug Fix] Bumps apps-manager-release to v662.0.14 to resolve an issue where Docker applications would crash Apps Manager.

1.12.3

• [Bug Fix] Bumps scalable-syslog-release to v11. Release Notes.
• [Bug Fix] Bumps usage-service-release to v663.0.3 to resolve an issue that prevented users from using a custom CA-signed certificate.
• [Security Improvement] Bumps garden-runc-release to v1.9.4. Release Notes.
• [Bug Fix] Bumps uaa-release to v45.3. Release Notes.

1.12.2

• [Security Fix] Bumps cflinuxfs2-release to v1.156.0 to resolve multiple security issues. Release Notes
• [Security Fix] Resolves an issue with an incorrect Host header being set on incoming requests through the Router CVE Notice.
• [Security Fix] Bumps cf-mysql-release to v36.6 to patch vulnerabilities in Bundler and RubyGems CVE-2016-7954 CVE-2017-0902
• [Security Fix] Resolves a remote code execution security vulnerability when the zip program is executed by the Cloud Controller.
• [Bug Fix] Bumps haproxy-boshrelease to v8.4.1 to resolve an issue with certificate/key concatenation Release Notes.
• [Bug Fix] Bumps apps-manager-release to v662.0.12 to patch the following:
  • AppsManager will now show all Application Security Group rules.
  • Fixes a bug that prevented arbitrary schema parameters from working when provisioning a new service from the flyout component.
  • When a space has zero members, a message is displayed indicating that the space has no members.
  • When creating a new org, the current user is added to that org as a user and an org manager.
• [Bug Fix] Resolves a bug caused by a missing “selector” option on the “Networking” tab. Operators who had previous chosen their Networking Point of Entry as a non-TLS external Load Balancer would experience this as a tile that looked fully configured, but could not be deployed due to an OpsMan configuration issue.
• [Stability Improvement] Changes the default Router Max Connections Per Backend from 0, or unlimited, to 500. This change prevents an unresponsive app from consuming all the router file descriptors. In some cases, this may impact the performance of existing apps and you may need to raise the setting. For guidance, see the documentation about the Max Connections Per Backend field. For example, if your PCF deployment is on GCP, see Step 6: Configure Networking in Deploying Elastic Runtime on GCP.
• [Feature] Operators can now configure a “Frontend Idle Timeout” for the Router and HAProxy. The default timeout is 900 seconds.
• [Feature] Bumps diego-release to v1.25.3 to include support for Azure MySQL Release Notes.
• [Feature] Patches cf-networking-release to include support for Azure MySQL
• Bumps buildpacks to the following versions:
  • dotnet-core-buildpack
  • nodejs-buildpack
  • php-buildpack
  • python-buildpack
  • ruby-buildpack
  • staticfile-buildpack
• [Stability Improvement] Changes the default for Galera MySQL state snapshot transfers (SST). Automatic SST is now enabled by default. Operators can disable this feature by visiting the “Internal MySQL” tab and selecting the Prevent node auto re-join checkbox.

1.12.1

• [Security Fix] Bumps stemcell to v3445.11 to address USN-3420-2.
• [Security Fix] Bumps cflinuxfs-release to v1.155.0 to address USN-3415-1.

1.12.0

How to Upgrade

The procedure for upgrading to Pivotal Cloud Foundry (PCF) Elastic Runtime v1.12 is documented in the Upgrading Pivotal Cloud Foundry topic.

When upgrading to v1.12, be aware of the following upgrade considerations:

• You must upgrade first to a version of Elastic Runtime v1.11.x to successfully upgrade to v1.12.

• If your existing PCF v1.11.x installation includes both PCF Runtime for Windows and MySQL for PCF v1.x, you must upgrade to MySQL for PCF v1.10.3 or later before you upgrade to PCF Elastic Runtime v1.12. For instructions on how to upgrade MySQL for PCF, see the MySQL for PCF documentation.

• Some partner service tiles may be incompatible with PCF v1.12. Pivotal is working with partners to ensure their tiles are being updated to work with the latest versions of PCF.
  
  For information about which partner service releases are currently compatible with PCF v1.12, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.

New Features in Elastic Runtime v1.12.0

This section describes new features of the release.

Multiple Buildpack App Support

Developers can now push apps that take advantage of multiple buildpacks simultaneously. You can upgrade apps to use multiple buildpacks with the cf v3-push command. This makes binaries, libraries, and language modules provided by all specified buildpacks available to the app. The final buildpack specified controls how the app starts.

Multiple buildpack support adds flexibility to the Cloud Foundry app development model. You can now use the official Cloud Foundry buildpacks together to support polyglot (multiple language) apps. Additionally, you can specify custom buildpacks before official Cloud Foundry buildpacks to supply dependencies that previously had to be provided with apps.

Granular cf push Commands and Procfile Support

This release includes experimental commands that provide developers with the ability to better orchestrate app deployment workflows. Additionally, developers can supply a Procfile with their app to run multiple long-lived processes using a single codebase. For more information, see Using Experimental cf CLI Commands.

Migration of Internal Credentials to CredHub

Internal credentials, the secret and simple_credentials that Elastic Runtime uses for inter-component communication, are generated and stored in CredHub instead of Ops Manager. This is part of an ongoing effort to migrate all credentials to CredHub, which will reduce the amount of places credentials are stored, aid in credential rotation, and increase security.

If you want to access the following credentials, you must use the CredHub CLI or the Ops Manager API instead of the Credentials tab of the Elastic Runtime tile. For instructions on how to to retrieve Elastic Runtime credentials, see Retrieving Credentials from Your Deployment.

• .mysql.autoscale_credentials
• .mysql.ccdb_credentials
• .mysql.diag_agent_credentials
• .mysql.diegodb_credentials
• .mysql.locketdb_credentials
• .mysql.monitordb_credentials
• .mysql.mysql_backup_server_credentials
• .mysql.mysql_bootstrap_credentials
• .mysql.networkpolicyserverdb_credentials
• .mysql.nfsvolume_credentials
• .mysql.notifications_credentials
• .mysql.pivotal_account_credentials
• .mysql.routingdb_credentials
• .mysql.silkdb_credentials
• .mysql.uaadb_credentials
• .nfsbrokerpush.nfs_broker_push_credentials
• .cloud_controller.bulk_api_credentials
• .cloud_controller.internal_api_user_credentials
• .cloud_controller.staging_upload_credentials
• .mysql.app_usage_credentials
• .mysql.cluster_health_user
• .mysql.galera_sidecar_user
• .mysql.mysql_admin_credentials
• .mysql_proxy.dashboard_credentials
• .nfs_server.blobstore_credentials
• .router.status_credentials

Introducing GrootFS

GrootFS is the new container image management plugin for Garden-runC. It helps with the filesystem isolation of Garden-runC containers, image caching, and disk quota enforcement. GrootFS replaces the previous built-in functionality, which used an obsolete layer filesystem (AUFS) that lacks support from the Linux Kernel community. Additionally, GrootFS uses OCI standards for container images.

For more information about GrootFS in PCF, see the following topics:

• Understanding Container Security
• Component: Garden

Application Instance Identity Credentials

The instance identity system in Diego provides each app container with a PEM-encoded X.509 certificate and PKCS #1 RSA private key. The values of the environment variables CF_INSTANCE_CERT and CF_INSTANCE_KEY contain the absolute paths to the certificate and private key files. The validity period is 3 years for the Instance Identity root and 2 years for the intermediate CA certificates.

See the App Instance Container Identity Credentials section for more information.

Simplified TLS Configuration

The point of entry options on the Elastic Runtime Networking pane have been restructured to be more understandable and flexible. Operators no longer need to configure the Router or HAProxy separately as both components are configured using the same options. This includes the following changes:

• Gorouter and HAProxy always listen for TLS requests. You provide an SSL certificate for both Gorouter and HAProxy using a single field.
• HAProxy forwards all requests to Gorouter over TLS by default. You can optionally disable this feature.
• You can configure the minimum version of TLS for Gorouter and HAProxy with a single field.
• You can provide a list of CAs to HAProxy for it to validate the Gorouter certificate.
• You can optionally disable the HTTP listener for both Gorouter and HAProxy with a single checkbox.
• You can specify TLS cipher suites for HAProxy and Gorouter independently.

See the Elastic Runtime installation instructions for your IaaS for more information.

Mutual TLS Headers on Inbound Application Traffic

Gorouter can now forward the X-Forwarded-Client-Cert header to app instances when provided. Alternatively, operators can configure Gorouter to forward the header only when the mutual TLS connection from the client can be validated. Additionally, operators may now configure Gorouter to overwrite the XFCC header with the client certificate received in mTLS handshakes.

This configuration is available in the Networking pane under Configure the CF Router support for the X-Forwarded-Client-Cert header. See the Elastic Runtime installation instructions for your IaaS for more information.

Secure Communication Between Cloud Controller and Diego

In previous versions of PCF, the Diego Brain VM ran the Cloud Controller Bridge component, which translated Cloud Controller requests into Diego API commands. The Cloud Controller Bridge conveyed communications between the Cloud Controller and Diego over plaintext HTTP. In PCF v1.12, the Cloud Controller and Diego communicate directly via secure TLS protocol. This change streamlines and secures internal communications, and removes the Cloud Controller Bridge.

Securing this communication path will require a second deployment after completing your upgrade to PCF v1.12. Follow the steps in our upgrade guide to secure your PCF installation.

Secure Communication Between Diego and Loggregator

Diego Cells now use the Metron API v2. This gRPC-based API supports mutual TLS authentication and secures the communication path between the Diego rep and Loggregator.

Scaling Loggregator

As part of this release, the Loggregator team has provided guidelines for scaling the Loggregator system. For more information, see Scaling Loggregator and Scaling Nozzles and Operator Guidebook.

HAProxy Release

This release removes the old HAProxy job, which was the last remaining component from cf-release. It now uses the newly incubated haproxy-boshrelease. This replacement allows Elastic Runtime to expose new HAProxy features, such as request filtering.

HAProxy Request Filtering

If your PCF deployment uses HAProxy and you want it to receive traffic only from specific sources, you can use the Protected Domains and Trusted CIDRs fields in the Networking Pane of the Elastic Runtime tile. A key use case for this feature is when a deployment must only allow requests to the system domain from a private network or VPN. See the Elastic Runtime installation instructions for your IaaS for more information.

Container-to-Container Networking Updates

Container-to-container networking is now always enabled. The commands are integrated with the cf CLI and now include the option to specify a port range when adding and removing policies. See Create Policies for Container-to-Container Networking.

Support for Logging All App Traffic

Operators can enable logging of all accepted and denied packets due to ASGs or container-to-container networking policies. This provides more visibility into app traffic, including denied traffic.

Operators configure this global logging in the Networking pane of the Elastic Runtime tile under the Log traffic for all accepted/denied application packets field. See the App Traffic Logging section for more information.

Orphaned Blob Cleanup

The Cloud Controller now scans the blobstore on a regular interval to identify and remove orphaned blobs. For more information, see the Blobstore section of the Cloud Controller topic.

Router Sharding Mode

This release includes support for router sharding between the Elastic Runtime and Isolation Segment tiles. Operators can choose to have the Elastic Runtime tile routers only acknowledge requests from apps deployed within the its Cells, or reject requests for any isolation segment.

You can configure this feature using the following fields:

• Elastic Runtime tile: Routers reject requests for Isolation Segments checkbox
• Isolation Segment tile: Router Sharding Mode selector

See the Elastic Runtime installation instructions for your IaaS for more information.

Gorouter Max Connection Configuration

Operators can limit the number of app instance connections to the backend using the Max Connections Per Backend field in the Networking pane of the Elastic Runtime tile. This field can help prevent malicious apps from consuming all available Gorouter resources. See the Elastic Runtime installation instructions for your IaaS for more information.

Authenticating with Google Container Registry (GCR) to Push Docker Images

For PCF v1.12 and later, Pivotal recommends authenticating with GCR using the procedure documented in the following section: Push a Docker Image from Google Container Registry (GCR). The alternative authentication mechanism provided by GCR passes a short lived (12 hours) access token to PCF. This enables PCF to pull images from GCR during the initial cf push, but subsequent restage, push, or rescheduling operations fail once the access token expires.

NFSV3 Volume Services with LDAP

Operators can now configure LDAP for NFSv3 volume services. Using LDAP secures the NFSV3 volume service by preventing a developer from binding to an NFS share using an arbitrary UID and potentially gaining access to sensitive data stored by another user or app. If you enable LDAP support, developers must provide credentials for any user they wish to bind as. See Enabling NFS Volume Services.

Metrics for MySQL

The internal MySQL job included in Elastic Runtime now emits metrics. See the Elastic Runtime MySQL KPIs.

CloudFormation Template Improvements

This release includes an improved CloudFormation template file available with the Elastic Runtime tile on Pivotal Network. The new template creates three availability zones, a load balancer for TCP routing, and the Ops Manager VM. For updated installation instructions, see Installing PCF on AWS Using CloudFormation.

Diego Cell Max-in-Flight Default

This release lowers the default max-in-flight percentage on Diego Cells to 4%. Previously, this value was set to 10%, which can cause the following issues in larger environments:

• Many simultaneous VM creates/deletes and BOSH blob updates placing significant stress on the underlying infrastructure, especially on vSphere which has a greater probability of being under-provisioned.
• Cells that are draining are no longer available for allocation, resulting in a 10% decrease in total memory and disk capacity during deployment. This can cause deployments to no longer have sufficient total capacity to run all work, or to have insufficient headroom to place larger workloads successfully.

Operators can still use the Ops Manager API to configure this setting to fit their needs. For more information about this property, see Managing Diego Cell Limits During Upgrade.

Removal of etcd

This release removes the etcd server VMs from the PCF deployment. Operators must ensure they are deploying service tiles that are known to be compatible with PCF Elastic Runtime 1.12.

Removal of Postgres

This release removes the legacy Postgres databases for the Cloud Controller and UAA. If your deployment was originally installed before PCF 1.6 and still uses Postgres, you must contact your dedicated Support Engineer or Platform Architect for assistance migrating your Cloud Controller and UAA databases to MySQL. They will have access to the PostgreSQL-to-MySQL Migrator tool and instructions on Pivotal Network.

If you do not migrate to MySQL before upgrading to Elastic Runtime 1.12, the upgrade will fail.

Apps Manager: In-context Service Creation

Developers can create services without leaving the app or space view for an accelerated workflow.

Apps Manager: Service Configuration Parameter Discovery

When creating a new service in Apps Manager, developers can discover additional parameter options as fields, or a JSON editor that enables them to define the parameters.

Known Issues

Disabling HTTP for Routers Causes Failures for Routes Bound to Internal Route Services

In all versions of Elastic Runtime v1.12, disabling the HTTP listener for Gorouter and HAProxy will cause clients to receive 502 responses for requests to routes that are bound to a route services that are run as apps on the platform. Route services that are run externally are not impacted.

To support route services run as apps on ERT, HTTP must remain enabled.

Cells Using Grootfs Fail to Run Privileged-Container Apps

In Elastic Runtime v1.12.0 through v1.12.14, Diego cells using the new Grootfs component fail to run application instances whose internal specifications required a privileged container. This defect affects buildpack-based applications last started on Elastic Runtime v1.8 or earlier.

To resolve this issue, take any one of the following actions:

• Deploy Elastic Runtime v1.12.15 or later,
• Redeploy Elastic Runtime with Grootfs disabled, or
• Manually restart any apps that are affected.

Docker App Disk Quota Failures

In PCF v1.11 and earlier, Grootfs code underestimated Docker image sizes when calculating disk quota usage. PCF v1.12 corrected this error, so after you upgrade, Docker images may exceed their existing disk quotas. This causes apps they contain to fail with the error applying disk limits: disk limit is smaller than volume size.

To fix this error, increase your Docker disk quotas to levels that more accurately reflect reality.

Manual CredHub Restart Required During an Elastic Runtime Redeploy

In Elastic Runtime v1.12.0, the BOSH Backup and Restore (BBR) script does not restart the CredHub process. When following the Restoring Pivotal Cloud Foundry from Backup with BBR procedure, the Elastic Runtime redeploy fails after clicking Apply Changes since CredHub requires a restart.

To work around this issue, manually restart the CredHub process on the BOSH Director by running monit restart credhub, then click Apply Changes.

For more information, see the corresponding Knowledge Base article.

Lack of Autoscaler Scaling

You cannot scale the Autoscaler job to greater than one instance.

Elastic Runtime Forwards High Volume of DEBUG Log Messages

Elastic Runtime forwards a high volume of DEBUG syslog messages from UAA and other system components to an external service.

Note: For information about remediating this issue in PAS v2.0, see PAS Forwards High Volume of DEBUG Log Messages in Pivotal Application Service v2.0 Release Notes.

Truncated Syslog Messages

Note: This issue is remediated when you select the Use TCP for file forwarding local transport option. For more information, see System Logging.

If the total length of a syslog message transported locally from a PCF system component (for example, the Cloud Controller or a Diego cell) is greater than 1,024 bytes, the packet is truncated before it reaches RSYSLOG installed on every BOSH VM instance.

The truncation is caused by the following:

• In PCF v1.12, a job writes log messages to a file in the /var/vcap/sys/log directory, and then syslog-migration-release forwards the messages to RSYSLOG. For reading log files from the /var/vcap/sys/log directory into RSYSLOG, the release uses blackbox. Because blackbox is configured to send log messages over UDP, it causes the underlying library to respect the message length restrictions of RFC 3164 and truncate packets. For more information, see syslog Message Parts in the RFC 3164 documentation.

• Prior to switching to syslog-migration-release in PCF v1.11, when a job generated a log message, it typically wrote the message in two locations: to the /var/vcap/sys/log directory and to RSYSLOG. For writing log messages directly to RSYSLOG, jobs used logger, an Ubuntu utility.

  In PCF v1.12, RSYSLOG receives two copies of each log message: one is from blackbox, and one is from the logger utility. Log messages sent through logger may be truncated as explained below:

  • If jobs are using the default version of logger installed on the stemcell, logs longer than 1 KB are truncated because the utility has a hard-coded message length limit.
  • If jobs are using a newer version of logger without this restriction or other tool to communicate with RSYSLOG over UDP, the truncation may not happen.

As mentioned above, jobs write system logs to the /var/vcap/sys/log directory. You can download full log lines from the directory files using Ops Manager.

Read-Only Volume Mounts Display as “rw”

Due to an underlying kernel defect, read-only volume mounts display as "mode": "rw" when you view the VCAP_SERVICES environment variable for your app.

For more information about binding a volume service, see Using an External File System (Volume Services).

Restore from Automated Backup of Internal MySQL Not Supported

If you configure PAS to use Internal MySQL, ensure that you select Disable Automated Backups of MySQL under the Automated Backups Configuration field. Pivotal does not support restoring the internal MySQL database from a full backup because it degrades the Galera MySQL cluster.

To back up and restore the internal MySQL database, you must use BOSH Backup and Restore (BBR). See Backing Up and Restoring Pivotal Cloud Foundry for information on using BBR.

For more information on this issue, see the following Pivotal Knowledge Base article: Restore from PAS Automated Database Backup is Not Supported in 1.11 and later.

About Advanced Features

The Advanced Features section of the Elastic Runtime tile includes new functionality that may have certain constraints.

Although these features are fully supported, Pivotal recommends caution when using them in production.