Pivotal Elastic Runtime v1.11 Release Notes

Page last updated:

Warning: Pivotal Cloud Foundry (PCF) v1.11 is no longer supported because it has reached the End of General Support (EOGS) phase. To stay up to date with the latest software and security updates, upgrade to a supported version.

Pivotal Cloud Foundry is certified by the Cloud Foundry Foundation for 2020.

Read more about the certified provider program and the requirements of providers.

Releases

1.11.37

• [Security Fix] Bump UAA for [CVE-2018-11047(https://www.cloudfoundry.org/blog/cve-2018-11047/)

• Bump uaa to version 45.11

1.11.36

• [Security Fix] Bump UAA for CVE 2018-11041

• Bump uaa to version 45.10

1.11.35

• [Security Fix] Bump diego to version 1.23.8
  • CVE-2018-1265

1.11.34

• [Security Fix] Bump stemcell to v3468.42:
  • USN-3641
  • USN-3631-2
  • USN-3628-1
  • USN-3625-1
  • USN-3624-1

1.11.33

• [Security Fix] Bumps stemcell to v3468.30:
  • USN-3619-2
  • USN-3611-1
  • USN-3610-1
  • USN-3598-1
  • USN-3586-1
  • USN-3584-1
• [Bug Fix] Bumps syslog-migration-release to v4.0.2:
  • Prevent logs from blackbox from being written to the default syslog log files to prevent logs from being written to the disk 3 additional times.
  • Fix rfc5424 compatibility by ensuring only 1 space occurs between the message and the structured data.
• [Bug Fix] Bumps loggregator-release to v89.0.27 to fix Traffic Controller resource leaks when connections are slow.
• [Feature Improvement] Adds field Custom syslog Configuration to specify custom logging rules in the System Logging tab. For more information, see custom syslog rules.

1.11.32

• [Bug Fix] Bumps capi-release to v1.28.56 to prevent app upload from failing when the app has broken symlinks.
• [Feature Improvement] Bumps loggregator-release to v89.0.26 to add stricter app id validation in Traffic Controller.
• [Bug Fix] Bumps apps-manager-release to v661.1.47:
  • Reintroduces cache busting for js/css files.
  • Fixes a bug that cause Apps Manager to fail to load when environment variables contains newlines.
  • Fixes endpoint headers.
  • Updates the CF CLI that is used to push Apps Manager and invitations.

1.11.31

• [Bug fix] Bumps apps-manager-release to v661.1.44.
  • [IE] Fix alignment of the app search bar in the header.
  • Fixes a bug that prevented mid-level fetch tasks from being cleared when switching routes and on the 30 second refresh.
  • Fixes a bug that caused marketplace service plans to show “No price available”.
  • Fixes occasional error when running usage service errand that would cause errand to fail because .cf directory was being pushed unnecessarily.
• [Bug fix] Bumps uaa-release to v45.8:
  • Updates JDK version to 8u162.
• [Security Fix] Bumps capi-release to v1.28.55:
  • CVE-2018-1266: Fixes random number guessing exploit.
  • Fixes buildpack pagination.

1.11.30

• [Bug fix] Bumps routing-release to v0.160.24:
  • Removes backends on any error to prevent 502 errors from being returned to clients.
  • Updates golang to v1.9.4.

1.11.29

• [Feature Improvement] Bumps apps-manager to v661.1.42, which uses nginx and the staticfile buildpack.
• [Bug Fix] Bumps capi-release to v1.28.53, which includes fix so that the API no longer loads all users into an array in memory.

1.11.28

Note: it is recommended that you re-create all VMs when upgrading to this release, due to the update to garden-runc-release. This will happen automatically if you are updating your stemcell. If not, you can check the “Recreate All VMs” checkbox on the Ops Manager Director > Director Config tab.

• [Security Fix] Bumps stemcell from version 3468.21 to version 3468.25 to address issues:
  • USN-3582-2
• [Security Fix] Bumps cflinuxfs2-release from v181.0 to v1.188.0 to address issues:
  • USN-3577-1
  • USN-3569-1
  • USN-3554-1
  • USN-3547-1
  • USN-3543-1
  • USN-3540-2
  • USN-3538-1
• [Feature Improvement] Bumps garden-runc-release to v1.11.1.
• [Feature Improvement] Patches cloud controller so users with admin_read_only scope can view stats for apps, which is needed by the cf v3-apps command.
• [Bug Fix] Patches cloud controller nginx http upload module to fix issue where incorrect initialization of the upload path could cause segmentation faults.

1.11.27

• [Security Fix] Patches routing-release for CVE-2018-1221.
• [Bug Fix] Fix to ensure that Diego rep will always exit during evacuation, even if Garden destroy hangs during evacuation.
• [Bug Fix] Patches syslog to prevent duplication from blackbox log forwarding.
• [Feature Improvements] Bumps mysql-backup-release to v2 in recognition of the fact that v1.38.0 required TLS. See other changes here
• [Feature Improvement] Enables Garden debug_listen_address to listen on a local interface.

1.11.26

• Bumps apps-manager-release to v661.1.41:
  • Fixes vulnerability that allowed arbitrary file access on server.
  • Increases memory footprint of usage service app to 1GB.
  • Adds security headers to the Apps Manager JS server (Strict-Transport-Security, X-Content-Type-Options, X-XSS-Protection, Cache-Control, Pragma).
  • Fixes checkbox alignment in the invite members page in Internet Explorer.
• [Bug Fix] Patches diego-release to allow HTTP-based health check on an HTTP endpoint that expects TLS-terminated traffic.
• [Bug Fix] Bumps java-offline-buildpack to v4.8 to address an issue with multiple java-offline-buildpacks being included, which may cause deployments to have different versions of java-offline-buildpack installed.
• Bump buildpacks to latest versions, including:
  • dotnet-core-offline-buildpack to v2.0.1.
  • go-offline-buildpack to v1.8.16.
  • java-offline-buildpack to v4.8.
  • nodejs-offline-buildpack to v1.6.15.
  • php-offline-buildpack to v4.3.48.
  • python-offline-buildpack to v1.6.7.
  • ruby-offline-buildpack to v1.7.11.
  • staticfile-offline-buildpack to v1.4.21.

1.11.25

• [Security Fix] Bumps stemcell to version 3468.21 to address issues:
  • USN-3534-1
  • USN-3540-2
• [Security Fix] Bumps cflinuxfs2-release to v1.181.0 to address issues:
  • USN-3532-1
  • USN-3534-1
  • USN-3535-1
• [Security Fix] Patches capi-release to fix issue where refresh tokens are not accepted where access tokens are required.
  • CVE-2018-1195
• [Feature Improvement] Bump syslog-migration-release to v4.0.1 and add a checkbox for log file forwarding through TCP to work around the Truncated Syslog Messages issue.
  • NOTE: Using TCP instead of the default UDP configuration may have a negative impact on performance.
• [Bug Fix] This release addresses memory usage issues on mysql monitoring node. Mysql-monitoring-release updated to v8.14.0. See the MySQL Monitoring v8.14 Release Notes for more information about this feature improvement.

1.11.24

• [Bug Fix] Bumps uaa-release to v45.7.
• [Bug Fix] Bumps apps-manager-release to v661.1.37 to resolve a number of issues:
  • Hide native select dropdown on Firefox.
  • When a call to /cloudfoundry application fails, do not continue to check if the app is a spring app.
  • Add clickjacking protection, while still allowing Apps Manager to load singular.
  • Long org names in the navbar org dropdown are ellipsified.
  • Fix 404 page footer in IE.
  • Fix styling in accounting report download button.
  • Fetch all routes for spaces instead of just the first page
• [Bug Fix] Patches loggregator-release to ensure traffic controller starts pprof server.
• [Bug Fix] Adds missing default domain streaming-mysql-backup-tool to mysql-backup certificate. Note: if you installed 1.11.22 or 1.11.23, you will have to rotate certificates. See this KB article for more details: Pivotal Application Service Backup and Restore fails due to Missing Streaming mysql-backup-tool Domain
• [Bug Fix] Bumps pivotal-account-release to v1.8.2 to fix bug that prevented errands from running more than once.
• [Feature Improvement] The SAML ‘Entity Id Override’ field has been moved from the Authentication and Enterprise SSO tab to the UAA tab in Ops Manager, to accompany the other SAML fields in the UAA tab.

1.11.23

This release introduces a bug that causes BBR backups to fail due to a missing default domain in the mysql-backup certificate. We recommend skipping this release and upgrading to 1.11.24 or higher, which resolves this issue. See the corresponding Knowledge Base for more information.

• [Security Fix] Bumps stemcell version to 3445.22 for USN-3544-2 and USN-3544-4

1.11.22

This release introduces a bug that causes BBR backups to fail due to a missing default domain in the mysql-backup certificate. We recommend skipping this release and upgrading to 1.11.24 or higher, which resolves this issue. See the corresponding Knowledge Base for more information.

• [Security Fix] Bumps cflinuxfs2-release to v1.176.0 for USN-3513-1.
• [Bug Fix] Resolves an issue in container-networking where a component in the same network with mTLS can cause an sql injection on the DeleteEntry database handler.
• [Bug Fix] Resolves a bug where task states are not updated when droplets are deleted.
• [Bug Fix] Bumps apps-manager-release to v661.1.36 to fix the following issues:
  • Fix bug in deployment errand that prevented logging of stdout and stderr.
  • Fixed appearance of 404 page in IE (footer is now at the bottom of the window).
  • Fixed appearance of Download Zip button on Accounting Report page in IE.
• [Feature Improvement] If an operator configures the stager with a whitelist of insecure Docker registries, the Docker app lifecycle builder now allows insecure communication with those registries.
• [Feature Improvement] Ops Manager now allows operators to specify an Azure environment name other than the default 'AzureCloud’. The option is in tab File Storage, under the External Azure Storage in the Environment field.
• [Feature Improvement] Bumps mysql-monitoring-release to v8.13.0 to add disk usage metrics as a percentage.
• [Feature Improvement] Bumps mysql-backup-release to v1.38.0 which enables mutual TLS between the backup node and server.
• [Feature] Bumps garden-runc-release to v1.10.0:
  • It is now possible to specify a ProcessSpec.Image. Processes can now have their own filesystem view.
  • Limitation: It is only possible to use ProcessSpec.Image and ProcessSpec.OverrideContainerLimits with unprivileged containers.
    This will be fixed in future releases.
  • Limitation: APIs such as BulkMetrics and Process.Signal may not work immediately after container.Run(ProcessSpec) returns for processes with Image and/or OverrideContainerLimits specified. This will be fixed in future releases.
  • Reduced log volume in BulkMetrics for large environments.
  • Correctly declares that bundles it creates are OCI Runtime Spec version 1.0.0 compliant.

1.11.21

• [Security Fix] Bumps stemcell version to 3445.19 for USN-3509-2.
• [Security Fix] Bumps cflinuxfs2-release to v1.171.0 to resolve several security vulnerabilities:
  • USN-3489-1: Berkeley DB vulnerability
  • USN-3496-1: Python vulnerability
  • USN-3496-3: Python vulnerability
  • USN-3498-1: curl vulnerabilities
  • USN-3501-1: libxcursor vulnerability
• [Bug Fix] Bumps apps-manager-release to v661.1.35 to resolve some bugs:
  • When an app fails to identify itself as a spring app, do not check again.
  • Prevent Apps Manager from being used in an iframe.
  • Long org names in the navbar org dropdown are ellipsified.
  • Fix the look of the select component in Firefox.
  • Improved the resiliency of the Apps Manager server when a proxy error occurs.
• [Bug Fix] Bumps cf-mysql-release to v36.10.0 to finalize a fix for configuration and management of syslog. Release Notes
• [Bug Fix] Bumps mysql-monitoring-release to v8.12.0 to finalize a fix for configuration and management of syslog.
• [Bug Fix] Operators can now optionally disable Router Access logs. This will prevent the Router local disk from becoming filled when the Routers are experiencing increased incoming traffic.
• [Feature Improvement] Operators can now specify the mutual TLS certificate validation behavior for the Router. The Router will request certificates by default and validate them if provided. Operators can optionally configure the Router not to request certificates or to require them with every request.

WARNING: Requests to the platform will fail upon upgrade if your load balancer is configured with client certificates and Gorouter does not have the certificate authority. To mitigate this issue, select Router does not request client certificates for Router behavior for Client Certificate Validation in the Networking pane.

• [Feature Improvement] Operators can now override their SAML Entity ID when configuration SAML as an Identity Provider.

1.11.20

• [Security Fix] Bumps apps-manager-release to v660.7.19 to resolve a number of security issues and bug fixes:
  • Upgrades to nodejs v8.0 to resolve a number of security issues.
  • When viewing a Spring App’s Threads tab, and there are no running instances, there is now text to convey this.
  • Fixes downloading of Spring threads on Internet Explorer.
  • Service plan costs are now formatted according to supported currencies when displayed on the Space and Services tabs.
  • Fixes bug when trying to view the org page members tab when there is a user without a username.
  • Changes color of buildpack text to meet accessibility standards.
• [Security Fix] Bumps buildpack releases versions to pick up security and bug fixes:
  • binary-buildpack v1.0.15
  • dotnet-core-buildpack v1.0.30
  • go-buildpack v1.8.13
  • java-buildpack v4.6
  • nodejs-buildpack v1.6.10
  • php-buildpack v4.3.43
  • python-buildpack v1.6.1
  • ruby-buildpack v1.7.5
  • staticfile-buildpack v1.4.18
• [Security Fix] Bumps the stemcell to v3445.17 to resolve the following security issues:
  • USN-3457-1: curl vulnerability
  • USN-3458-1: ICU vulnerability
  • USN-3464-1: Wget vulnerabilities
  • USN-3469-2: Linux kernel (Xenial HWE) vulnerabilities
  • USN-3475-1: OpenSSL vulnerabilities
  • USN-3478-1: Perl vulnerabilities
  • USN-3485-2: Linux kernel (Xenial HWE) vulnerabilities
• [Security Fix] Bumps cflinuxfs2-release to v1.168.0 to resolve USN-3478-1: Perl vulnerabilities.
• [Security Fix] Patches Cloud Controller to prevent users from being able to create a private subdomain of a route in an organization they do not have access to.
• [Bug Fix] Reverts the previous patch release change to the SAML Entity ID field. The field is once again using http for its URL scheme.
• [Improvement] The custom branding fields for the square logo and favicon are now separate fields.

1.11.19

This release has been pulled due to a regression introduced in the SAML identity provider interface. Please upgrade to 1.11.20 or higher to resolve this issue with the SAML entityID.

• [Security Fix] Bumps cflinuxfs2-release to v1.166.0 to resolve USN-3475-1. Release Notes
• [Bug Fix] Bumps cf-mysql-release to v36.9.0 to resolve an issue where IPsec causes mariadb_ctrl to be left in an Execution Failed state. Release Notes
• [Bug Fix] Bumps consul-release to v187 to include a fix for Internationalized Domain Name encoding when specifying an availability-zone-specific service.
• [Bug Fix] Bumps notifications-release to v37 to resolve a race condition in the cf CLI used during the deployment of the notifications service errand. Release Notes
• [Security Fix] Patches Golang components in capi-release to pull in Golang v1.8.3.
• [Bug Fix] Patches loggregator-release to introduce a circuit breaker when the Traffic Controller connects to the Doppler VMs.
• [Bug Fix] Patches loggregator-release to ensure metron agents maintain availability-zone affinity when connecting to Doppler VMs.
• [Bug Fix] Changes the scheme for the SAML Entity ID from http to https.

1.11.18

• [Security Fix] Bumps the stemcell to v3445.16 to resolve several security vulnerabilities:
  • USN-3424-1
  • USN-3432-1
  • USN-3434-1
  • USN-3441-1
  • USN-3444-2
• [Security Fix] Bumps the cflinuxfs2-release to v1.165.0 to resolve several security vulnerabilities:
  • USN-3457-1
  • USN-3458-1
  • USN-3464-1
• [Bug Fix] Bumps uaa-release to v45.4 to prevent a denial of service attack against the token revocation endpoint.
• [Bug Fix] Patches loggregator-release to remove the totalReceivedMessageCount metric from the v2 API.
• The logging level for the Cloud Controller, Cloud Controller Worker, and Cloud Controller Clock has been lowered from debug to info. This should help reduce log volume while still logging detailed Cloud Controller information.
• [Bug Fix] Garden is now configured to destroy containers on start. This setting will cause the garden process to remove any containers that are already running when it starts. That action will prevent issues where containers that should no longer be running are left up to run.
• The SAML Signature Algorithm field is now configurable outside of the Identity Provider option for SAML. This means that deployments using a non-SAML identity provider can still configure their SAML settings for SSO.
• Operators can now opt-in to allowing remote administrator access to the internal MySQL database. This was previously enabled by default in releases prior to 1.11.3. In that release, cf-mysql was bumped to v36. That release brought a number of security improvments, including the ability to prevent remote administrator access to the database. Unfortunately, this was a feature that some operators had come to rely upon. The Elastic Runtime will now allow those operators to enable the feature on a selective basis.

1.11.17

• [Security Fix] Bumps cflinuxfs2-release to v1.161.0 to resolve multiple security issues. Release Notes
• [Bug Fix] Bumps consul-release to v181 to ensure encrypt key rotation only occurs when the key changes.
• [Bug Fix] Patches cf-networking-release to allow deployment against a MySQL database version < 5.7.
• [Security Fix] Ensures Cloud Controller CEF logs are written to disk, and syslog. Logs now show up in /var/vcap/sys/log/cloud_controller_ng/security_events.log.

1.11.16

• [Security Fix] Bumps cflinuxfs2-release to v1.158.0 to resolve multiple security issues. Release Notes

1.11.15

• [Security Improvement] Bumps garden-runc-release to v1.9.4. Release Notes.
• [Bug Fix] Bumps uaa-release to v45.3. Release Notes.
• [Bug Fix] Bumps scalable-syslog-release to v11. Release Notes.
• [Bug Fix] Resolves an issue with S3-compatible stores specifying the N/A region.
• [Feature Improvement] Router now supports setting a Frontend Idle Timeout to maintain an open connection when clients support keep-alive. The default value is 900 seconds.

1.11.14

• [Security Fix] Bumps cflinuxfs2-release to v1.156.0 to resolve multiple security issues. Release Notes
• [Security Fix] Bumps cf-mysql-release to v36.6 to patch vulnerabilities in Bundler and RubyGems CVE-2016-7954 CVE-2017-0902
• [Security Fix] Resolves a remote code execution security vulnerability when the zip program is executed by the Cloud Controller.
• [Security Fix] Resolves an issue with an incorrect Host header being set on incoming requests through the Router CVE Notice.
• [Bug Fix] AppsManager will now show all Application Security Group rules.
• [Bug Fix] Loggregator API counters will now include a correct delta.
• Bumps the following buildpack releases:
  • binary-buildpack
  • dotnet-core-buildpack
  • go-buildpack
  • java-buildpack
  • nodejs-buildpack
  • php-buildpack
  • python-buildpack
  • ruby-buildpack
  • staticfile-buildpack
• [Stability Improvement] Changes the default for Galera MySQL state snapshot transfers (SST). Automatic SST is now enabled by default. Operators can disable this feature by visiting the “Internal MySQL” tab and checking the “Prevent node auto re-join” checkbox.
• Operators can now specify a minimum supported TLS version for the Router and HAProxy.
• The Cipher Suites for the Router and HAProxy are now required fields.

1.11.13

• [Security Fix] Bumps stemcell to v3445.11 to address USN-3420-2.
• [Security Fix] Bumps cflinuxfs-release to v1.155.0 to address USN-3415-1.

1.11.12

• [Bug Fix] Bumps apps-manager-release to v661.1.32 to resolve:
  • Several issues when viewing Apps Manager on Internet Explorer and Microsoft Edge browsers.
  • Communicating directly with Spring apps to retrieve heap dumps.
• [Bug Fix] Patches loggregator to resolve the following:
  • The keep alive handler for websocket connections to TC is closing the connection early.
  • Separates internal cipher suite configurations.
  • When doppler crashes, metrons connected to it get “stuck”.
• [Bug Fix] Disables inadvertent iptables logging when container networking is enabled.
• [Bug Fix] Bumps cf-mysql-release to v36.5 to resolve an issue with Syslog configuration.
• [Bug Fix] Prevents a race condition during upgrade by not allowing syslog configuration to be set in metron.

1.11.11

• [Security Fix] Bumps cflinuxfs2-release to v1.150.0 to resolve USN-3398-1.
• [Bug Fix] Bumps cf-autoscaling-release to v93.1 Release Notes.
• [Bug Fix] Bumps apps-manager-release to v661.1.31, including the following fixes:
  • The square logo will now display correctly in the header.
  • Users can now see shared private domains on the org domains tab.
• [Bug Fix] Fixes a data migration in Cloud Controller to prune duplicate routes when upgrading to this version from a 1.10.x version of the Elastic Runtime.
• [Bug Fix] Resolves an issue where deleting an application would not generate a TASK_STOPPED event for any tasks associated with that application.
• [Feature Improvement] Operators can now configure a “Staging Timeout” to force Cloud Controller to wait for staging of applications that may take a very long time.
• [Feature Improvement] The internal MySQL cluster now emits metrics via the Firehose. You can use cf nozzle to view those metrics as they are emitted. See Elastic Runtime MySQL KPIs for KPIs based on the metrics.

1.11.10

• [Security Fix] Bumps stemcell to v3421.20 to resolve USN-3392-2.
• [Security Fix] Bumps cflinuxfs2-release to v1.147.0 to resolve USN-3387-1 and USN-3388-1.

1.11.9

• [Security Fix] Bumps stemcell to version 3421.19.
• [Bug Fix] Bumps diego-release to v1.23.2 to resolve a number of issues including:
  • improving locket stability during MySQL updates which resolves the known issue Brief Unavailability of Application Management during Internal MySQL Database Updates
  • reporting the correct result of running tasks on Windows cells
  • resolution of a race condition in the process launching code that could cause process failures
  • improvements to the healthcheck error messaging
  • removing the extraneous “cancelled” message in logs when applications crash
  • prefixing process errors with their log source
  • removing exit codes from healthcheck output when an application fails its healthcheck
• [Bug Fix] Cloud Controller will no longer maintain keepalive connections to the Router as it could cause errant 502 responses when making API calls.
• Applications now have access to the certificate provided by the requester via the X-Forwarded-Client-Cert header. Configuration for this feature can be found on the Networking tab.
• The regions listed on the File Storage form and the selector for Internal MySQL backups for S3-compatible blobstores now includes all available S3 regions.
• Automated backup for the internal MySQL instances now includes support for GCP and Azure.

1.11.8

• [Security Fix] Bumps stemcell to v3421.18 to resolve USN-3378-2.
• [Security Fix] Bumps cflinuxfs2 to v1.145.0 to resolve multiple CVEs and USNs. Please see the release notes for more details.

1.11.7

• [Security Improvement] Operators can now choose to disable the recording of command history when running the mysql command on their Internal MySQL VMs. Unchecking the “Allow Command History” checkbox on the “Internal MySQL” tab will disable recording of that history into the .mysql_history file.
• [Bug Fix] Bumps apps-manager-release to v661.1.30. In this version, the dependencies for Apps Manager are now properly vendored to support deployments on internetless environments.
• [Bug Fix] Bumps uaa-release to v45. This version includes a patch to ensure that systems deployed on non-RFC 1918 networks will have their Router IPs automatically added to the whitelist of proxies for the UAA.
• [Bug Fix] The smoke test errand will now correctly use the system organization to deploy its canary applications.
• The components included in routing-release (gorouter, route_registrar, routing-api, tcp_emitter, and tcp_router) have been updated to run on Go v1.8.

1.11.6

• [Security Fix] Bumps apps-manager-release to v661.1.29. This release has the following updates:
  • Bumps nodejs to v6.11.1 to provide security fixes.
  • Includes org_id and org_name in usage report zip file.
  • Ensures Spring Actuator requests happen over HTTPS.
• [Bug Fix] Includes AWS US East (Ohio) region in the region selector for Internal MySQL backups.
• [Bug Fix] Specifies a more consistent cluster identifier, cf, for the Internal MySQL cluster when emitting metrics.
• [Stability Improvement] Reduces the default for cc.droplets.max_staged_droplets_stored from 5 to 2. This will result in reduced blobstore utilization as the Cloud Controller will only keep 2 historic staged application droplets, in addition to the currently running application droplet. Garbage collection of expired droplets will occur the next time an application is staged.

1.11.5

• [Security Fix] Provides a fix for CVE-2017-8033. This security vulnerability would have allowed attackers to escalate their privileges by pushing an application that could modify the files on the Cloud Controller VM.
• [Security Fix] The Router will now validate the UAA token issuer field. This will prevent users with valid tokens belonging to an Identity Zone other than the default zone from escalating their privileges when making requests against system components.
• [Bug Fix] Resolves an issue with UAA SAML Service Provider Key Password quoting.
• [Stability Improvement] Operators can now configure the Cluster Probe Timeout for their Internal MySQL cluster. This property controls the maximum time a new MySQL node will search for an existing cluster before starting its own. Higher values for this property will help to maintain cluster quorum on slower or more loaded infrastructures.
• Bumps nfs-volume-release to v1.0.6. Release Notes
• Bumps the following buildpacks to their latest version:
  • dotnet-core-buildpack Release Notes
  • go-buildpack Release Notes
  • java-buildpack Release Notes
  • nodejs-buildpack Release Notes
  • php-buildpack Release Notes
  • python-buildpack Release Notes
  • ruby-buildpack Release Notes
  • staticfile-buildpack Release Notes
• Sets the default max-in-flight value for the Diego Cells to 4%. Operators can still use the Ops Manager API to configure this setting to fit their needs. The max-in-flight percentage for the Diego Cell job in the Elastic Runtime has been set to 10% since 1.9, but we’ve seen especially in larger environments that having the percentage this high can cause some problems:
  • Many simultaneous VM creates/deletes and BOSH blob updates can place significant stress on the underlying infrastructure, especially on vSphere which has a greater probability of being under-provisioned.
  • The cells that are draining are no longer available for allocation, resulting in a 10% decrease in total memory and disk capacity during the deployment. This can cause deployments to no longer have sufficient total capacity to run all the work, or to have insufficient headroom to place larger workloads successfully.

1.11.4

• [Security Fix] Cloud Controller will now validate the UAA token issuer field. This will prevent users with valid tokens belonging to an Identity Zone other than the default zone from escalating their privileges when making requests against system components.
• [Security Fix] Provides a fix for CVE-2017-8035. This security vulnerability would have allowed arbitrary files on the Cloud Controller VM to be downloaded by external API users.

1.11.3

• [Bug Fix] Bumps cf-backup-and-restore-release to v0.0.5. [Release Notes]
• [Security Improvement] Bumps cf-mysql-release to v36. [Release Notes]
• Bumps service-backup-release to v18.1.2. [Release Notes]
• [Bug Fix] Bumps garden-runc-release to v1.9.0. [Release Notes] Resolves known issue with private Docker image registries.
• [Bug Fix] Adds the network.write group to UAA. This new UAA group will allow operators to give users with the SpaceDeveloper role the ability to manage Container Networking access policy.
• [Improvement] Allows operators to scale their etcd instance count to 0.

  Do not scale the number of etcd instances to 0 until you have completed your upgrade to PCF Elastic Runtime 1.11.
  
  Scaling the etcd instances to 0 before performing the upgrade will result in significant downtime on both the management and application tiers.

1.11.2

• [Security Fix] Bumps cflinuxfs2-rootfs to 1.33.0. [Release Notes]
• [Bug Fix] Bumps nfs-volume-release to v1.0.5. [Release Notes]
• Bumps mysql-monitoring-release to v8.3.0. [Release Notes]
• Bumps cf-mysql-release to v35.1.
• Configuring external system database will no longer scale down the internal mysql components. This allows a mixed database configuration where the system database is external but the UAA database may use internal MySQL. Starting in ERT 1.11.0, the UAA database is configured independently on the UAA form. MySQL instance counts for existing installations will not be affected by this change. Going forward, you will need to manually set the instance count to 0 for all MySQL VMs (MySQL Server, MySQL Proxy, and MySQL Monitor) if they are not in use.

1.11.1

• [Security Fix] Bumps stemcell to 3421.9.
• [Security Fix] Bumps uaa-release to v41. [Release Notes]
• [Security Fix] Bumps cflinuxfs2 to v1.126.0. [Release Notes]
• Bumps the java buildpack to v3.17. [Release Notes]
• Bumps diego-release to v1.18.1. [Release Notes]
• [Bug Fix] Patches Cloud Controller to prevent a DB error when a staging response contains too many environment variables.
• [Feature] Container-to-container networking will log iptables rules in the kernel log on each Diego Cell.
• [Feature] All errands except smoke tests will default to run “when-changed”. Smoke tests will default to run every time.
• Removes unnecessary persistent disk from Cloud Controller VM.
• [Bug Fix] Resolves known issue with infrequent Diego cell rep crashes.

1.11.0

How to Upgrade

The procedure for upgrading to Pivotal Cloud Foundry Elastic Runtime v1.11 is documented in the Upgrading Pivotal Cloud Foundry topic.

When upgrading to v1.11, be aware of the following upgrade considerations:

• You must upgrade first to a version of Elastic Runtime v1.10.x in order to successfully upgrade to v1.11.
• Some partner service tiles may be incompatible with PCF v1.11. Pivotal is working with partners to ensure their tiles are being updated to work with the latest versions of PCF.
  
  For information about which partner service releases are currently compatible with PCF v1.11, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.

About Advanced Features

The Advanced Features section of the Elastic Runtime tile includes new functionality that may have certain constraints.

Although these features are fully supported, Pivotal recommends caution when using them in production.

New Features in Elastic Runtime v1.11

This section describes new features of the release.

Apps Manager 508 Compliance

Apps Manager has improved its compliance with Section 508 of the Rehabilitation Act. Customers can now receive an approved Voluntary Product Accessibility Template (VPAT) for Apps Manager.

Container-to-Container Networking

The Container-to-Container Networking feature in Elastic Runtime is out of beta. For more information, see the Container-to-Container Networking topic.

Volume Services

General support for volume services inside of application containers is out of beta. Additionally, the Elastic Runtime ships with an NFS Volume Service Broker.

For more information, see Using an External File System (Volume Services) and Enabling NFS Volume Services.

Increased Application Maximum Disk Quotas

For applications that require large amounts of disk, the maximum disk quota has been raised to 20 GB. This will allow application developers to specify a disk size up to 20 GB for their applications.

Operators should take note that applications with large disk settings can be difficult for Diego to find a placement for. If there is not enough disk on any of the Diego Cells, developers will see an “insufficient resources” error message. Ensuring your Diego Cells provide enough disk to accommodate these much larger application instances will be an important part of your resource planning.

Override DNS Servers

By default, containers use the same DNS servers as the host. To override the DNS servers used by the containers of an isolation segment, enter a comma-separated list of servers in the DNS Servers field of the Application Containers section of the tile.

Highly-Available Default Configuration

The Resource Configuration defaults have been changed to streamline the path to a production-ready deployment. The default instance count for each VM type in the deployment has been set to ensure an operator can deploy a highly-available Cloud Foundry out of the box.

This change will result in larger resource requirements for a default Elastic Runtime deployment. Operators should take this into account when deploying environments on constrained resources.

Component BOSH Releases are SHA2-checksummed

BOSH now supports releases that include a SHA2-checksum of their contents. All the releases included in the Elastic Runtime have been checksummed with SHA2 for improved data integrity in the face of hash collisions.

BOSH Backup and Restore

The Elastic Runtime now includes support for the new BOSH Backup and Restore tool (bbr). Using bbr allow an operator to take a snapshot of their internal blobstore and databases.

Private Docker Registries

Application developers can now push Docker-based applications with images that are hosted on a private registry.

PCF can use private Docker repositories from various different Docker registries:

• GCR can be used to host private Docker images for PCF. However, the passwords GCR issues are essentially access tokens which expire within an hour. This means that the initial application push will be successful but the application may later crash as PCF will not be able to contact GCR.

Common Syslog Formatting and Syslog Over TLS

The Elastic Runtime now emits BOSH component logs in a common syslog format following RFC5424. Additionally, syslog over TLS is now supported to allow operators to deliver their platform logs securely to their syslog aggregator.

UAA Supports SQL Server Database

Operators can now independently configure their UAA service to store its data in an external SQL Server database. You can configure the UAA database in the UAA pane of the Elastic Runtime tile.

Note: UAA database configuration has been fully separated intovthe UAA tab to allow its independent configuration. Upon upgrade, settings willvbe migrated from the existing installation. Going forward, you must change both the UAA and System Database forms to keep their configuration in sync.

Improved Distributed Lock Management

In older versions of the Elastic Runtime, components that needed to maintain a distributed lock could leverage Consul to handle the locking mechanism. In v1.11, that locking mechanism has been replaced with a new service called locket. By switching to locket, the Elastic Runtime will be moving onto a more stable footing for its distributed locking needs.

UAA User Passwords Never Expire

As of v1.11, Elastic Runtime no longer provides the Number of Months Before Password Expires field in the Authentication and Enterprise SSO section. Instead, it sets the value to zero, which means that UAA user passwords for deployments using internal UAA never expire.

If you have set a non-zero password expiry, Elastic Runtime will set it to zero when you upgrade from v1.10 to v1.11.

Redirect URI Validation for OAuth Clients

Starting from v1.11, UAA validates the redirect URIs of OAuth clients to prevent open redirect attacks. A redirect URI is now a mandatory parameter for OAuth clients that use the authorization_code and implicit grant types to acquire an access token. For more information, see Known Issues.

Scalable Syslog

Loggregator now uses the Scalable Syslog BOSH release to support streaming app logs to syslog-compatible aggregation or analytics services. This new delivery method has the following benefits:

• Decouples syslog drains from the Firehose
• Prevents log loss in the Firehose that previously resulted from syslog drains competing for the resources used to deliver logs
• Provides high message reliability for syslog drains
• Maintains the UX of binding an app to a syslog drain service through Cloud Foundry user-provided services (CUPS)

For more information about Scalable Syslog in PCF, see Overview of the Loggregator System.

Known Issues

This section lists known issues for PCF Elastic Runtime.

Brief Unavailability of Application Management During Internal MySQL Database Updates

Fixed in ERT v1.11.9.

This issue applies only to Elastic Runtime deployments using an internal MySQL database with more than one MySQL instance. When updating this MySQL database deployment, some application management operations such as cf push may become unavailable for up to 15 seconds, as may the Routing API. Application instances will continue to run, however, and their routes will remain registered. This period of unavailability results from certain Elastic Runtime components losing their leader-election locks and being able to reclaim the lock only after the 15-second lock expiration time. This availability issue will be fixed in a forthcoming patch release.

App Autoscaler Panic

The App Autoscaler service can panic due to an issue with leader election. To avoid this issue, operators must manually scale the App Autoscaler service down to one instance before starting the upgrade to PCF v1.11. For more information, see the Scale Down App Autoscaler section of the Upgrading Pivotal Cloud Foundry topic.

If you fail to complete this procedure prior to the upgrade and the App Autoscaler service panics, follow the instructions in the Autoscaler failing on SQL query Knowledge Base article to restore the service.

Loggregator Metrics

Reverse Log Proxy (RLP) and Traffic Controller emit metrics tagged with a hard-coded deployment name. Because the components do not allow the deployment name to be configured, Metron cannot properly tag the metrics. This affects the following metrics:

• loggregator.rlp.ingress
• loggregator.rlp.dropped
• loggregator.rlp.egress
• loggregator.trafficcontroller.ingress

A patch is planned to fix this issue and apply the deployment name consistently.

Elastic Runtime Forwards High Volume of DEBUG Log Messages

Elastic Runtime forwards a high volume of DEBUG syslog messages from UAA and other system components to an external service.

Note: For information about remediating this issue in PAS v2.0, see PAS Forwards High Volume of DEBUG Log Messages in Pivotal Application Service v2.0 Release Notes.

Truncated Syslog Messages

Note: This issue is remediated when you select the Use TCP for file forwarding local transport option. For more information, see System Logging.

If the total length of a syslog message transported locally from a PCF system component (for example, the Cloud Controller or a Diego cell) is greater than 1,024 bytes, the packet is truncated before it reaches RSYSLOG installed on every BOSH VM instance.

The truncation is caused by the following:

• In PCF v1.11, a job writes log messages to a file in the /var/vcap/sys/log directory, and then syslog-migration-release forwards the messages to RSYSLOG. For reading log files from the /var/vcap/sys/log directory into RSYSLOG, the release uses blackbox. Because blackbox is configured to send log messages over UDP, it causes the underlying library to respect the message length restrictions of RFC 3164 and truncate packets. For more information, see syslog Message Parts in the RFC 3164 documentation.

• Prior to switching to syslog-migration-release in PCF v1.11, when a job generated a log message, it typically wrote the message in two locations: to the /var/vcap/sys/log directory and to RSYSLOG. For writing log messages directly to RSYSLOG, jobs used logger, an Ubuntu utility.

  In PCF v1.11, RSYSLOG receives two copies of each log message: one is from blackbox, and one is from the logger utility. Log messages sent through logger may be truncated as explained below:

  • If jobs are using the default version of logger installed on the stemcell, logs longer than 1 KB are truncated because the utility has a hard-coded message length limit.
  • If jobs are using a newer version of logger without this restriction or other tool to communicate with RSYSLOG over UDP, the truncation may not happen.

As mentioned above, jobs write system logs to the /var/vcap/sys/log directory. You can download full log lines from the directory files using Ops Manager.

Mount Location Bug

If you have the VOLUME ["/some/data/dir"] directive in your image, ensure your "mount" section in your bind JSON matches what is in the Dockerfile, such as "mount":"/some/data/dir".

See this example Dockerfile on GitHub for reference.

This only works if the path in question is in an already existing root level folder. For example, "/var/some/random/path" works, but "/some/random/path" causes your app to crash at startup. This limitation is caused by an aufs bug.

Private Docker Registry Support for ECR, GCR and JFrog Artifactory

Fixed in ERT v1.11.3.

The private Docker registry support referenced above does not currently support Amazon EC2 Container Registry, Google Cloud Platform Container Registry, or JFrog Artifactory. Future patches will bring support for these registries.

UAA Redirect URIs

A redirect URI is now a mandatory parameter for the authorization_code and implicit grant types, which is required to submit an access request. When using the UAA API or UAAC and declaring OAuth clients in the manifest, make sure at least one redirect URI is specified. Existing OAuth clients that do not provide this information will get an error when accessing the /oauth/authorize endpoint. See the example below:

{
"error": "invalid_client",
"error_description": "authorization_code grant type requires at least one redirect URL."
}

Redirect URIs should start with http or https. A subdomain regex is supported only in the first part of the base domain:

• https://*.apps.com is supported.
• https://*.apps.com* is not supported.
• https://*.apps*.com* is not supported.

Failed Upgrade In Large Deployments

Upgrades of large deployments to Elastic Runtime v1.11 may fail due to a timeout while updating PostgreSQL. If the upgrade fails, redeploy.

For more information, see the “Deploying ERT 1.11 fails to due to a timeout while updating PostGreSQL” Knowledge Base article.

Diego Cell Rep Crashes

Fixed in ERT v1.11.1.

In ERT v1.11.0, the Diego cell rep process will infrequently crash as a result of a race condition when buffering health-check logs for application instances. The application instances on that cell will be rescheduled when the rep process is restarted or when the Diego system notices that its presence has expired, typically within 15 seconds of the rep process crashing.

Restore from Automated Backup of Internal MySQL Not Supported

If you configure PAS to use Internal MySQL, ensure that you select Disable Automated Backups of MySQL under the Automated Backups Configuration field. Pivotal does not support restoring the internal MySQL database from a full backup because it degrades the Galera MySQL cluster.

To back up and restore the internal MySQL database, you must use BOSH Backup and Restore (BBR). See Backing Up and Restoring Pivotal Cloud Foundry for information on using BBR.

For more information on this issue, see the following Pivotal Knowledge Base article: Restore from PAS Automated Database Backup is Not Supported in 1.11 and later.

Read-Only Volume Mounts Display as “rw”

Due to an underlying kernel defect, read-only volume mounts display as "mode": "rw" when you view the VCAP_SERVICES environment variable for your app.

For more information about binding a volume service, see Using an External File System (Volume Services).