LATEST VERSION: 1.11 - CHANGELOG

Pivotal Cloud Foundry v1.11

• v1.10
• v1.9
• v1.8
• v1.7
• v1.6
• v1.5
• v1.4
• v1.3
• v1.2

Download Ask for Help PCF Knowledge Base PDF

Pivotal Elastic Runtime v1.11 Release Notes

Pivotal Cloud Foundry is certified by the Cloud Foundry Foundation for 2017.

Read more about the certified provider program and the requirements of providers.

Releases

1.11.8

• [Security Fix] Bumps stemcell to v3421.18 to resolve USN-3378-2.
• [Security Fix] Bumps cflinuxfs2 to v1.145.0 to resolve multiple CVEs and USNs. Please see the release notes for more details.

1.11.7

• [Security Improvement] Operators can now choose to disable the recording of command history when running the mysql command on their Internal MySQL VMs. Unchecking the “Allow Command History” checkbox on the “Internal MySQL” tab will disable recording of that history into the .mysql_history file.
• [Bug Fix] Bumps apps-manager-release to v661.1.30. In this version, the dependencies for Apps Manager are now properly vendored to support deployments on internetless environments.
• [Bug Fix] Bumps uaa-release to v45. This version includes a patch to ensure that systems deployed on non-RFC 1918 networks will have their Router IPs automatically added to the whitelist of proxies for the UAA.
• [Bug Fix] The smoke test errand will now correctly use the system organization to deploy its canary applications.
• The components included in routing-release (gorouter, route_registrar, routing-api, tcp_emitter, and tcp_router) have been updated to run on Go v1.8.

1.11.6

• [Security Fix] Bumps apps-manager-release to v661.1.29. This release has the following updates:
  • Bumps nodejs to v6.11.1 to provide security fixes.
  • Includes org_id and org_name in usage report zip file.
  • Ensures Spring Actuator requests happen over HTTPS.
• [Bug Fix] Includes AWS US East (Ohio) region in the region selector for Internal MySQL backups.
• [Bug Fix] Specifies a more consistent cluster identifier, cf, for the Internal MySQL cluster when emitting metrics.
• [Stability Improvement] Reduces the default for cc.droplets.max_staged_droplets_stored from 5 to 2. This will result in reduced blobstore utilization as the Cloud Controller will only keep 2 historic staged application droplets, in addition to the currently running application droplet. Garbage collection of expired droplets will occur the next time an application is staged.

1.11.5

• [Security Fix] Provides a fix for CVE-2017-8033. This security vulnerability would have allowed attackers to escalate their privileges by pushing an application that could modify the files on the Cloud Controller VM.
• [Security Fix] The Router will now validate the UAA token issuer field. This will prevent users with valid tokens belonging to an Identity Zone other than the default zone from escalating their privileges when making requests against system components.
• [Bug Fix] Resolves an issue with UAA SAML Service Provider Key Password quoting.
• [Stability Improvement] Operators can now configure the Cluster Probe Timeout for their Internal MySQL cluster. This property controls the maximum time a new MySQL node will search for an existing cluster before starting its own. Higher values for this property will help to maintain cluster quorum on slower or more loaded infrastructures.
• Bumps nfs-volume-release to v1.0.6. Release Notes
• Bumps the following buildpacks to their latest version:
  • dotnet-core-buildpack Release Notes
  • go-buildpack Release Notes
  • java-buildpack Release Notes
  • nodejs-buildpack Release Notes
  • php-buildpack Release Notes
  • python-buildpack Release Notes
  • ruby-buildpack Release Notes
  • staticfile-buildpack Release Notes
• Sets the default max-in-flight value for the Diego Cells to 4%. Operators can still use the Ops Manager API to configure this setting to fit their needs. The max-in-flight percentage for the Diego Cell job in the Elastic Runtime has been set to 10% since 1.9, but we’ve seen especially in larger environments that having the percentage this high can cause some problems:
  • Many simultaneous VM creates/deletes and BOSH blob updates can place significant stress on the underlying infrastructure, especially on vSphere which has a greater probability of being under-provisioned.
  • The cells that are draining are no longer available for allocation, resulting in a 10% decrease in total memory and disk capacity during the deployment. This can cause deployments to no longer have sufficient total capacity to run all the work, or to have insufficient headroom to place larger workloads successfully.

1.11.4

• [Security Fix] Cloud Controller will now validate the UAA token issuer field. This will prevent users with valid tokens belonging to an Identity Zone other than the default zone from escalating their privileges when making requests against system components.
• [Security Fix] Provides a fix for CVE-2017-8035. This security vulnerability would have allowed arbitrary files on the Cloud Controller VM to be downloaded by external API users.

1.11.3

• [Bug Fix] Bumps cf-backup-and-restore-release to v0.0.5. [Release Notes]
• [Security Improvement] Bumps cf-mysql-release to v36. [Release Notes]
• Bumps service-backup-release to v18.1.2. [Release Notes]
• [Bug Fix] Bumps garden-runc-release to v1.9.0. [Release Notes] Resolves known issue with private docker image registries.
• [Bug Fix] Adds the network.write group to UAA. This new UAA group will allow operators to give users with the SpaceDeveloper role the ability to manage Container Networking access policy.
• [Improvement] Allows operators to scale their etcd instance count to 0.

  Do not scale the number of etcd instances to 0 until you have completed your upgrade to PCF Elastic Runtime 1.11.
  
  Scaling the etcd instances to 0 before performing the upgrade will result in significant downtime on both the management and application tiers.

1.11.2

• [Security Fix] Bumps cflinuxfs2-rootfs to 1.33.0. [Release Notes]
• [Bug Fix] Bumps nfs-volume-release to v1.0.5. [Release Notes]
• Bumps mysql-monitoring-release to v8.3.0. [Release Notes]
• Bumps cf-mysql-release to v35.1.
• Configuring external system database will no longer scale down the internal mysql components. This allows a mixed database configuration where the system database is external but the UAA database may use internal MySQL. Starting in ERT 1.11.0, the UAA database is configured independently on the UAA form. MySQL instance counts for existing installations will not be affected by this change. Going forward, you will need to manually set the instance count to 0 for all MySQL VMs (MySQL Server, MySQL Proxy, and MySQL Monitor) if they are not in use.

1.11.1

• [Security Fix] Bumps stemcell to 3421.9.
• [Security Fix] Bumps uaa-release to v41. [Release Notes]
• [Security Fix] Bumps cflinuxfs2 to v1.126.0. [Release Notes]
• Bumps the java buildpack to v3.17. [Release Notes]
• Bumps diego-release to v1.18.1. [Release Notes]
• [Bug Fix] Patches Cloud Controller to prevent a DB error when a staging response contains too many environment variables.
• [Feature] Container-to-container networking will log iptables rules in the kernel log on each Diego Cell.
• [Feature] All errands except smoke tests will default to run “when-changed”. Smoke tests will default to run every time.
• Removes unnecessary persistent disk from Cloud Controller VM.
• [Bug Fix] Resolves known issue with infrequent Diego cell rep crashes.

1.11.0

How to Upgrade

The procedure for upgrading to Pivotal Cloud Foundry Elastic Runtime v1.11 is documented in the Upgrading Pivotal Cloud Foundry topic.

When upgrading to v1.11, be aware of the following upgrade considerations:

• You must upgrade first to a version of Elastic Runtime v1.10.x in order to successfully upgrade to v1.11.
• Some partner service tiles may be incompatible with PCF v1.11. Pivotal is working with partners to ensure their tiles are being updated to work with the latest versions of PCF.
  
  For information about which partner service releases are currently compatible with PCF v1.11, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.

About Advanced Features

The Advanced Features section of the Elastic Runtime tile includes new functionality that may have certain constraints.

Although these features are fully supported, Pivotal recommends caution when using them in production.

New Features in Elastic Runtime v1.11.0

This section describes new features of the release.

Apps Manager 508 Compliance

Apps Manager has improved its compliance with Section 508 of the Rehabilitation Act. Customers can now receive an approved Voluntary Product Accessibility Template (VPAT) for Apps Manager.

Container-to-Container Networking

The Container-to-Container Networking feature in Elastic Runtime is out of beta. For more information, see the Container-to-Container Networking topic.

Volume Services

General support for volume services inside of application containers is out of beta. Additionally, the Elastic Runtime ships with an NFS Volume Service Broker.

For more information, see Using an External File System (Volume Services) and Enabling NFS Volume Services.

Increased Application Maximum Disk Quotas

For applications that require large amounts of disk, the maximum disk quota has been raised to 20GB. This will allow application developers to specify a disk size up to 20GB for their applications.

Operators should take note that applications with large disk settings can be difficult for Diego to find a placement for. If there is not enough disk on any of the Diego Cells, developers will see an “insufficient resources” error message. Ensuring your Diego Cells provide enough disk to accommodate these much larger application instances will be an important part of your resource planning.

Override DNS Servers

By default, containers use the same DNS servers as the host. To override the DNS servers used by the containers of an isolation segment, enter a comma-separated list of servers in the DNS Servers field of the Application Containers section of the tile.

Highly-Available Default Configuration

The Resource Configuration defaults have been changed to streamline the path to a production-ready deployment. The default instance count for each VM type in the deployment has been set to ensure an operator can deploy a highly-available Cloud Foundry out of the box.

This change will result in larger resource requirements for a default Elastic Runtime deployment. Operators should take this into account when deploying environments on constrained resources.

Component BOSH releases are SHA2-checksummed

BOSH now supports releases that include a SHA2-checksum of their contents. All the releases included in the Elastic Runtime have been checksummed with SHA2 for improved data integrity in the face of hash collisions.

BOSH Backup and Restore

The Elastic Runtime now includes support for the new BOSH Backup and Restore tool (bbr). Using bbr allow an operator to take a snapshot of their internal blobstore and databases.

Private Docker Registries

Application developers can now push Docker-based applications with images that are hosted on a private registry.

PCF can use private Docker repositories from various different Docker registries:

* GCR can be used to host private Docker images for PCF. However, the passwords GCR issues are essentially access tokens which expire within an hour. This means that the initial application push will be successful but the application may later crash as PCF will not be able to contact GCR.

Common Syslog Formatting and Syslog over TLS

The Elastic Runtime now emits BOSH component logs in a common syslog format following RFC5424. Additionally, syslog over TLS is now supported to allow operators to deliver their platform logs securely to their syslog aggregator.

UAA supports SQL Server database

Operators can now independently configure their UAA service to store its data in an external SQL Server database. You can configure the UAA database in the UAA pane of the Elastic Runtime tile.

Important Note: UAA database configuration has been fully separated into the UAA tab to allow its independent configuration. Upon upgrade, settings will be migrated from the existing installation. Going forward, you must change both the UAA and System Database forms to keep their configuration in sync.

Improved Distributed Lock Management

In older versions of the Elastic Runtime, components that needed to maintain a distributed lock could leverage Consul to handle the locking mechanism. In 1.11, that locking mechanism has been replaced with a new service called locket. By switching to locket, the Elastic Runtime will be moving onto a more stable footing for its distributed locking needs.

UAA User Passwords Never Expire

As of v1.11, Elastic Runtime no longer provides the Number of Months Before Password Expires field in the Authentication and Enterprise SSO section. Instead, it sets the value to zero, which means that UAA user passwords for deployments using internal UAA never expire.

If you have set a non-zero password expiry, Elastic Runtime will set it to zero when you upgrade from v1.10 to v1.11.

Redirect URI Validation for OAuth Clients

Starting from v1.11, UAA validates the redirect URIs of OAuth clients to prevent open redirect attacks. A redirect URI is now a mandatory parameter for OAuth clients that use the authorization_code and implicit grant types to acquire an access token. For more information, see Known Issues.

Scalable Syslog

Loggregator now uses the Scalable Syslog BOSH release to support streaming app logs to syslog-compatible aggregation or analytics services. This new delivery method has the following benefits:

• Decouples syslog drains from the Firehose
• Prevents log loss in the Firehose that previously resulted from syslog drains competing for the resources used to deliver logs
• Provides high message reliability for syslog drains
• Maintains the UX of binding an app to a syslog drain service through Cloud Foundry user-provided services (CUPS)

For more information about Scalable Syslog in PCF, see Overview of the Loggregator System.

Known Issues

This section lists known issues for PCF Elastic Runtime.

Brief Unavailability of Application Management during Internal MySQL Database Updates

This issue applies only to Elastic Runtime deployments using an internal MySQL database with more than one MySQL instance. When updating this MySQL database deployment, some application management operations such as cf push may become unavailable for up to 15 seconds, as may the Routing API. Application instances will continue to run, however, and their routes will remain registered. This period of unavailability results from certain Elastic Runtime components losing their leader-election locks and being able to reclaim the lock only after the 15-second lock expiration time. This availability issue will be fixed in a forthcoming patch release.

App Autoscaler Panic

The App Autoscaler service can panic due to an issue with leader election. To avoid this issue, operators must manually scale the App Autoscaler service down to one instance before starting the upgrade to PCF v1.11. For more information, see the Scale Down App Autoscaler section of the Upgrading Pivotal Cloud Foundry topic.

If you fail to complete this procedure prior to the upgrade and the App Autoscaler service panics, follow the instructions in the Autoscaler failing on SQL query Knowledge Base article to restore the service.

Loggregator Metrics

Reverse Log Proxy (RLP) and Traffic Controller emit metrics tagged with a hard-coded deployment name. Because the components do not allow the deployment name to be configured, Metron cannot properly tag the metrics. This affects the following metrics:

• loggregator.rlp.ingress
• loggregator.rlp.dropped
• loggregator.rlp.egress
• loggregator.trafficcontroller.ingress

A patch is planned to fix this issue and apply the deployment name consistently.

Mount Location Bug

If you have the VOLUME ["/some/data/dir"] directive in your image, ensure your "mount" section in your bind JSON matches what is in the Dockerfile, such as "mount":"/some/data/dir".

See this example Dockerfile on GitHub for reference.

This only works if the path in question is in an already existing root level folder. For example, "/var/some/random/path" works, but "/some/random/path" causes your app to crash at startup. This limitation is caused by an aufs bug.

Private Docker Registry Support for ECR, GCR and JFrog Artifactory

Fixed in ERT 1.11.3.

The private Docker registry support referenced above does not currently support Amazon EC2 Container Registry, Google Cloud Platform Container Registry, or JFrog Artifactory. Future patches will bring support for these registries.

UAA Redirect URIs

A redirect URI is now a mandatory parameter for the authorization_code and implicit grant types, which is required to submit an access request. When using the UAA API or UAAC and declaring OAuth clients in the manifest, make sure at least one redirect URI is specified. Existing OAuth clients that do not provide this information will get an error when accessing the /oauth/authorize endpoint. See the example below:

{
"error": "invalid_client",
"error_description": "authorization_code grant type requires at least one redirect URL."
}

Redirect URIs should start with http or https. A subdomain regex is supported only in the first part of the base domain:

• https://*.apps.com is supported.
• https://*.apps.com* is not supported.
• https://*.apps*.com* is not supported.

Failed Upgrade In Large Deployments

Upgrades of large deployments to Elastic Runtime v1.11 may fail due to a timeout while updating PostgreSQL. If the upgrade fails, redeploy.

For more information, see the “Deploying ERT 1.11 fails to due to a timeout while updating PostGreSQL” Knowledge Base article.

Diego Cell Rep Crashes

Fixed in ERT 1.11.1.

In ERT 1.11.0, the Diego cell rep process will infrequently crash as a result of a race condition when buffering health-check logs for application instances. The application instances on that cell will be rescheduled when the rep process is restarted or when the Diego system notices that its presence has expired, typically within 15 seconds of the rep process crashing.