Pivotal Application Service v2.8 Release Notes

Page last updated:

Warning: Pivotal Application Service (PAS) v2.8 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic contains release notes for Pivotal Application Service (PAS) v2.8.

For the feature highlights of this release, read the blog post Any company can become a software-driven organization. The new release of Tanzu Application Service gives you the blueprint or see New Features in PAS v2.8.

Pivotal Platform is certified by the Cloud Foundry Foundation for 2022.

For more information about the Cloud Foundry Certified Provider Program, see How Do I Become a Certified Provider? on the Cloud Foundry website.

Because VMware uses the Percona Distribution for MySQL, expect a time lag between Oracle releasing a MySQL patch and VMware releasing PAS containing that patch.

Releases

2.8.31

Release Date: 12/21/2021

• [Security Fix] Fix uncontrolled recursion related to Log4j (CVE-2021-45105)
• Bump credhub to version 2.9.8 which has Log4j 2.17.0
• Bump java-offline-buildpack to version 4.47
• Bump uaa to version 74.5.30 which has Log4j 2.17.0

2.8.30

Release Date: 12/16/2021

• [Security Fix] Fix remote code execution vulnerability related to Log4j (CVE-2021-45046)
• Bump credhub to version 2.9.7 which has Log4j 2.16.0
• Bump java-offline-buildpack to version 4.45
• Bump php-offline-buildpack to version 4.4.53
• Bump uaa to version 74.5.29 which has Log4j 2.16.0

2.8.29

Release Date: 12/15/2021

• [Security Fix] UAA and CredHub - Fix remote code execution vulnerability related to Log4j CVE-2021-44228
• [Security Fix] Gorouter built with Go 1.16.7 to address CVE-2021-36221
• [Bug Fix] Diego - Envoy 1.19 should use original TCP connection pool, so that it can accept more than 1024 downstream connections
• Bump binary-offline-buildpack to version 1.0.39
• Bump bpm to version 1.1.13
• Bump cf-networking to version 2.42.0
• Bump cflinuxfs3 to version 0.251.0
• Bump credhub to version 2.9.6 which has Log4j 2.15.0
• Bump diego to version 2.54.0
• Bump dotnet-core-offline-buildpack to version 2.3.32
• Bump go-offline-buildpack to version 1.9.34
• Bump java-offline-buildpack to version 4.44
• Bump log-cache to version 2.6.18
• Bump metric-registrar to version 1.1.9
• Bump nginx-offline-buildpack to version 1.1.30
• Bump nodejs-offline-buildpack to version 1.7.57
• Bump php-offline-buildpack to version 4.4.52
• Bump pxc to version 0.37.0
• Bump python-offline-buildpack to version 1.7.43
• Bump r-offline-buildpack to version 1.1.20
• Bump ruby-offline-buildpack to version 1.8.42
• Bump silk to version 2.38.0
• Bump staticfile-offline-buildpack to version 1.5.24
• Bump system-metrics-scraper to version 2.0.14
• Bump uaa to version 74.5.28 which has Log4j 2.15.0

2.8.28

Release Date: 06/22/2021

• [Security Fix] Bump some dependencies to resolve security vulnerabilities
• Bump bpm to version 1.1.12
• Bump cf-autoscaling to version 238
• Bump cf-networking to version 2.37.0
• Bump cflinuxfs3 to version 0.241.0
• Bump dotnet-core-offline-buildpack to version 2.3.29
• Bump java-offline-buildpack to version 4.39
• Bump loggregator-agent to version 5.2.14
• Bump metric-registrar to version 1.1.6
• Bump nginx-offline-buildpack to version 1.1.27
• Bump nodejs-offline-buildpack to version 1.7.53
• Bump php-offline-buildpack to version 4.4.40
• Bump python-offline-buildpack to version 1.7.41
• Bump ruby-offline-buildpack to version 1.8.40
• Bump silk to version 2.37.0
• Bump statsd-injector to version 1.11.16

2.8.27

Release Date: 05/27/2021

• [Feature Improvement] Patch versions can be upgraded without a stemcell upgrade.
• [Feature Improvement] Improve metric-registrar to handle unreachable CC more gracefully, delete smoke-test app in the case of failure, and integrate CUPS caching.
• [Feature Improvement] Adds per request metrics reporting, which makes metric frequency proportional to request frequency.
• [Bug Fix] Fix race condition and prevent network policy MySQL database from being able to get into an invalid state.
• Bump binary-offline-buildpack to version 1.0.38
• Bump bpm to version 1.1.11
• Bump cf-networking to version 2.36.0
• Bump cflinuxfs3 to version 0.238.0
• Bump diego to version 2.50.0
• Bump dotnet-core-offline-buildpack to version 2.3.28
• Bump garden-runc to version 1.19.25
• Bump go-offline-buildpack to version 1.9.31
• Bump metric-registrar to version 1.1.5
• Bump nginx-offline-buildpack to version 1.1.26
• Bump nodejs-offline-buildpack to version 1.7.51
• Bump notifications to version 62
• Bump php-offline-buildpack to version 4.4.39
• Bump push-usage-service-release to version 671.0.20
• Bump pxc to version 0.35.0
• Bump python-offline-buildpack to version 1.7.39
• Bump r-offline-buildpack to version 1.1.17
• Bump ruby-offline-buildpack to version 1.8.39
• Bump silk to version 2.36.0
• Bump staticfile-offline-buildpack to version 1.5.21

2.8.26

Release Date: 03/31/2021

• [Breaking Change] This restores the breaking change originally found in 2.8.24 and temporarily remediated in 2.8.25: Gorouter update to Golang v1.15 introduces stricter transfer-encoding header standards. Stricter header standards break Spring apps that incorrectly set the header. For more information, see Applications on TAS for VMs get 502 chunked response error in the Knowledge Base.
• [Breaking Change] PAS v2.8.26 specifies Java buildpack v4.36. This causes a breaking change if you use Tanzu GemFire for VMs v1.12 or earlier with Tomcat session state caching. For information about how to avoid this breaking change, see Known Issues in the Tanzu GemFire for VMs documentation.
• [Security Fix] Usage Service - Upgrade Nokogiri gem to address CVE-2020-26247
• [Security Fix] Usage Service - Upgrade Rails gem to address CVE-2021-22880
• [Feature Improvement] MySQL binlogs volume is capped at 33% of available disk storage
• Bump ubuntu-xenial stemcell to version 621.115
• Bump cf-cli to version 1.32.0
• Bump cflinuxfs3 to version 0.227.0
• Bump diego to version 2.49.0
• Bump go-offline-buildpack to version 1.9.26
• Bump nfs-volume to version 5.0.12
• Bump nodejs-offline-buildpack to version 1.7.42
• Bump php-offline-buildpack to version 4.4.31
• Bump push-usage-service-release to version 671.0.19
• Bump python-offline-buildpack to version 1.7.30
• Bump routing to version 0.213.0
• Bump ruby-offline-buildpack to version 1.8.31

2.8.25

Release Date: 02/19/2021

• [Breaking Change] PAS v2.8.25 specifies Java buildpack v4.36. This causes a breaking change if you use Tanzu GemFire for VMs v1.12 or earlier with Tomcat session state caching. For information about how to avoid this breaking change, see Known Issues in the Tanzu GemFire for VMs documentation.
• [Temporary Remediation] Gorouter - Emit log, emit metric, and don’t error when an app response contains a duplicate “Transfer-Encoding: chunked” header. This is a stop gap to discover which apps are sending invalid responses. For more information, see Applications on TAS for VMs get 502 chunked response error in the Knowledge Base.
• [Bug Fix] Autoscaling - Reduce RabbitMQ overhead of RabbitMQ based autoscaling rules.
• Bump ubuntu-xenial stemcell to version 621.101
• Bump cf-autoscaling to version 237
• Bump cflinuxfs3 to version 0.223.0
• Bump dotnet-core-offline-buildpack to version 2.3.22
• Bump go-offline-buildpack to version 1.9.25
• Bump java-offline-buildpack to version 4.36
• Bump nginx-offline-buildpack to version 1.1.20
• Bump nodejs-offline-buildpack to version 1.7.41
• Bump php-offline-buildpack to version 4.4.30
• Bump pxc to version 0.33.0
• Bump python-offline-buildpack to version 1.7.29
• Bump r-offline-buildpack to version 1.1.12
• Bump routing to version 0.211.1
• Bump ruby-offline-buildpack to version 1.8.30
• Bump staticfile-offline-buildpack to version 1.5.15
• Bump uaa to version 74.5.22

2.8.24

Release Date: 12/18/2020

• [Breaking Change] Gorouter update to Golang v1.15 introduces stricter transfer-encoding header standards. Stricter header standards break Spring apps that incorrectly set the header. For more information, see Applications on TAS for VMs get 502 chunked response error in the Knowledge Base.
• [Security Fix] Bump garden-runc-release to address CVE-2020-15257
• Bump ubuntu-xenial stemcell to version 621.94
• Bump cf-autoscaling to version 235
• Bump cf-networking to version 2.35.0
• Bump cflinuxfs3 to version 0.216.0
• Bump dotnet-core-offline-buildpack to version 2.3.19
• Bump garden-runc to version 1.19.18
• Bump go-offline-buildpack to version 1.9.23
• Bump java-offline-buildpack to version 4.34
• Bump nginx-offline-buildpack to version 1.1.19
• Bump nodejs-offline-buildpack to version 1.7.37
• Bump php-offline-buildpack to version 4.4.27
• Bump python-offline-buildpack to version 1.7.26
• Bump r-offline-buildpack to version 1.1.11
• Bump routing to version 0.209.0
• Bump ruby-offline-buildpack to version 1.8.27
• Bump silk to version 2.35.0
• Bump staticfile-offline-buildpack to version 1.5.14

2.8.23

Release Date: 11/17/2020

• Bump bosh-system-metrics-forwarder to version 0.0.20
• Bump metric-registrar to version 1.1.4
• Bump pxc to version 0.31.0
• Bump uaa to version 74.5.21

2.8.22 - Withdrawn

Warning: This release has been removed from VMware Tanzu Network due to the Bosh System Metrics Forwarder release v0.0.19 regresssion.

Release Date: 11/04/2020

• [Security Fix] Fix for CVE-2020-5417
• Bump ubuntu-xenial stemcell to version 621.90
• Bump bosh-system-metrics-forwarder to version 0.0.19
• Bump capi to version 1.87.10
• Bump dotnet-core-offline-buildpack to version 2.3.17
• Bump go-offline-buildpack to version 1.9.21
• Bump java-offline-buildpack to version 4.33
• Bump log-cache to version 2.6.17
• Bump metrics-discovery to version 2.0.9
• Bump nginx-offline-buildpack to version 1.1.16
• Bump nodejs-offline-buildpack to version 1.7.32
• Bump php-offline-buildpack to version 4.4.23
• Bump python-offline-buildpack to version 1.7.24
• Bump r-offline-buildpack to version 1.1.10
• Bump ruby-offline-buildpack to version 1.8.26
• Bump staticfile-offline-buildpack to version 1.5.12
• Bump statsd-injector to version 1.11.15

2.8.21

Release Date: 10/26/2020

• [Feature] UAA - Expose configuration options for proxy settings
• [Feature Improvement] Operators can configure how long cloud controller audit events are retained
• [Feature Improvement] Networking: Clarify that drain timeout should be lower than backend request timeout to reduce drain time during deploys
• [Bug Fix] Loggregator Agent Release - Prom Scraper metrics server names match
• Bump ubuntu-xenial stemcell to version 621.89
• Bump cflinuxfs3 to version 0.210.0
• Bump uaa to version 74.5.20

2.8.20

Release Date: 10/19/2020

• [Security Fix] Bump Percona XtraDB Cluster to 5.7.31
• [Bug Fix] ServiceDiscoveryController - Reconnect internal routing metrics to the firehose [Bug Fix] Gorouter performs TLS to backends when “Skip SSL Certificate Verification” is enabled. Stale routes are now pruned.
• Bump ubuntu-xenial stemcell to version 621.87
• Bump cf-networking to version 2.34.0
• Bump cflinuxfs3 to version 0.209.0
• Bump pxc to version 0.30.0
• Bump routing to version 0.208.0
• Bump silk to version 2.34.0

2.8.19

Release Date: 10/09/2020

• [Security Fix] Remove credentials from process name and fix CVE CODEC-134, CODEC-270
• [Bug Fix] Enable users to adjust the timeout for all healthcheck types in Apps Manager
• [Bug Fix] Users can view process based audit events in Apps Manager
• [Bug Fix] Revert Metric-Registrar Bump
• Bump ubuntu-xenial stemcell to version 621.85
• Bump cflinuxfs3 to version 0.208.0
• Bump credhub to version 2.5.13
• Bump dotnet-core-offline-buildpack to version 2.3.15
• Bump go-offline-buildpack to version 1.9.19
• Bump nginx-offline-buildpack to version 1.1.15
• Bump nodejs-offline-buildpack to version 1.7.29
• Bump php-offline-buildpack to version 4.4.22
• Bump push-apps-manager-release to version 671.0.18
• Bump python-offline-buildpack to version 1.7.22
• Bump r-offline-buildpack to version 1.1.9
• Bump ruby-offline-buildpack to version 1.8.25
• Bump staticfile-offline-buildpack to version 1.5.11

2.8.18

Release Date: 09/21/2020

• [Security Fix] Bump Usage Service ruby version to 2.6.6 - CVE-2020-15169 CVE-2020-10933 CVE-2020-10663
• [Security Fix] Update cf-autoscaling’s dependencies to mitigate CVEs
• [Bug Fix] Modify cf-autoscaling’s API to return HTTP status 404 (not found) when not logged in. Previously it returned 401 (unauthorized). The behavior now matches the documentation
• [Bug Fix] Include correct version of push-usage-service-release 671.0.17 - an incorrect version was shipped in TAS v2.8.16 and TAS v2.8.17
• [Feature Improvement] Secure scraping available in Metric Registrar (Reverted in v2.8.19)
• Bump ubuntu-xenial stemcell to version 621.84
• Bump cf-autoscaling to version 233
• Bump metric-registrar to version 1.2.1

2.8.17

Release Date: 09/10/2020

WARNING: This version of TAS includes an incorrect version of Usage Service. We strongly suggest skipping this release and using TAS v2.8.18 or newer instead. If you are already running this release we suggest upgrading. For more information see this kb article.

• [Security Fix] Fix for CVE-2020-5420: Improve Gorouter’s handling of invalid HTTP response codes
• [Feature Improvement] Gorouter aliases /healthz to /health in order to prevent downtime during upgrades
• [Feature Improvement] Allow users to scale memory & disk for web processes in Apps Manager when autoscaling is enabled
• [Bug Fix] Safeguard against null log payloads for apps in Apps Manager
• [Bug Fix] Improve Log Cache Syslog Ingestion Performance
• [Breaking Change] Change noisy Gorouter logs to use debug log level. Replace with more helpful, quieter logs. For more details, see routing-release in GitHub.
• Bump ubuntu-xenial stemcell to version 621.82
• Bump cf-networking to version 2.33.0
• Bump cflinuxfs3 to version 0.204.0
• Bump diego to version 2.48.0
• Bump dotnet-core-offline-buildpack to version 2.3.14
• Bump go-offline-buildpack to version 1.9.17
• Bump log-cache to version 2.6.16
• Bump nfs-volume to version 5.0.10
• Bump nginx-offline-buildpack to version 1.1.14
• Bump nodejs-offline-buildpack to version 1.7.26
• Bump php-offline-buildpack to version 4.4.20
• Bump push-apps-manager-release to version 671.0.17
• Bump python-offline-buildpack to version 1.7.20
• Bump routing to version 0.207.0
• Bump silk to version 2.33.0
• Bump staticfile-offline-buildpack to version 1.5.10

2.8.16

Release Date: 08/24/2020

WARNING: This version of TAS includes an incorrect version of Usage Service. We strongly suggest skipping this release and using TAS v2.8.18 or newer instead. If you are already running this release we suggest upgrading. For more information see this kb article.

• [Security Fix] Fix for CVE-2020-5416: Improve Gorouter’s websocket error handling
• [Bug Fix] loggr-syslog-agent - Fix server alternative name
• [Bug Fix] Fix memory leak in RLP gateway
• [Bug Fix]: Return 502 TLS Handshake error for an unresponsive backend
• [Bug Fix] Fix Usage Service for inactive foundations
• [Bug Fix] Bump garden-runc to v1.19.16
• Bump ubuntu-xenial stemcell to version 621.78
• Bump cflinuxfs3 to version 0.203.0
• Bump garden-runc to version 1.19.16
• Bump go-offline-buildpack to version 1.9.16
• Bump java-offline-buildpack to version 4.32.1
• Bump loggregator to version 106.2.9
• Bump python-offline-buildpack to version 1.7.18
• Bump routing to version 0.205.0
• Bump ruby-offline-buildpack to version 1.8.23

2.8.15

Release Date: 08/07/2020

• [Security Fix] Notifications-ui removes UAA client secret from logs during installation
• [Feature Improvement] Upgrade Percona-XtraDB-Cluster to version 5.7.30-31.43
• [Bug Fix] Fix issue where requests to internal routes could fail due to incorrect case-sensitivity in DNS lookup in the service discovery controller.
• [Bug Fix] System Metrics Scraper/Prom Scraper — Fixes a bug that causes excess log volume and increases scrape interval to reduce metric volume
• Bump ubuntu-xenial stemcell to version 621.77
• Bump cf-cli to version 1.28.0
• Bump cf-networking to version 2.31.0
• Bump cf-smoke-tests to version 40.0.134
• Bump cflinuxfs3 to version 0.202.0
• Bump dotnet-core-offline-buildpack to version 2.3.13
• Bump garden-runc to version 1.19.14
• Bump go-offline-buildpack to version 1.9.15
• Bump nginx-offline-buildpack to version 1.1.12
• Bump nodejs-offline-buildpack to version 1.7.25
• Bump notifications-ui to version 39
• Bump php-offline-buildpack to version 4.4.19
• Bump pxc to version 0.28.0
• Bump python-offline-buildpack to version 1.7.17
• Bump ruby-offline-buildpack to version 1.8.22
• Bump silk to version 2.31.0
• Bump system-metrics-scraper to version 2.0.13

2.8.14

Release Date: 07/16/2020

• [Security Fix] Fix for CVE-2020-15586: Bump golang to version 1.14.5 with a fix in the net/http/httputil package for an issue which could cause the Gorouter to crash if a malicious client sends specially crafted HTTP requests.
• [Feature Improvement] Platform operators can see X-Cf-RouterError response headers in router access logs
• [Feature Improvement] Application developers can successfully deploy a reverse-proxy with support for sticky sessions

• [Feature Improvement] Gorouter provides improved logging when the following error is received: x509: certificate has expired or is not yet valid

• Bump cf-cli to version 1.27.0

• Bump cf-smoke-tests to version 40.0.132

• Bump cflinuxfs3 to version 0.198.0

• Bump dotnet-core-offline-buildpack to version 2.3.12

• Bump go-offline-buildpack to version 1.9.14

• Bump java-offline-buildpack to version 4.31.1

• Bump nginx-offline-buildpack to version 1.1.11

• Bump nodejs-offline-buildpack to version 1.7.24

• Bump php-offline-buildpack to version 4.4.18

• Bump python-offline-buildpack to version 1.7.16

• Bump r-offline-buildpack to version 1.1.7

• Bump routing to version 0.203.0

• Bump ruby-offline-buildpack to version 1.8.21

• Bump staticfile-offline-buildpack to version 1.5.9

2.8.13

Release Date: 07/09/2020

• [Breaking Change] If you use the NSX-T Container Plugin (NCP) tile v3.0.1 or earlier, do not upgrade to this patch. The stemcell in this patch is not compatible with the NCP tile v3.0.1 and causes the openvswitch job to fail when you deploy.
• [Security Fix] Stop logging credentials in Autoscaler app
• [Bug Fix] For sets of logs larger than 4MB, Apps Manager does not make requests to log cache with an invalid log limit
• [Bug Fix] Display correct guid for App subresources in v2 GET response
• [Bug Fix] Fix bug impacting hybrid grant flow with external oauth providers
• [Bug Fix] Restore access to Log Cache service logs
• Bump capi to version 1.87.9
• Bump cf-autoscaling to version 232
• Bump cf-smoke-tests to version 40.0.130
• Bump cflinuxfs3 to version 0.197.0
• Bump log-cache to version 2.6.15
• Bump push-apps-manager-release to version 671.0.15
• Bump uaa to version 74.5.18

2.8.12

Release Date: 06/25/2020

• [Breaking Change] Incorrect HTTP(S) Proxy configuration breaks CredHub interpolation for apps. For more information, see Incorrect HTTP(S) Proxy Configuration Breaks CredHub Interpolation for Apps in PAS v2.8.12 and Later below.
• [Breaking Change] If you use the NSX-T Container Plugin (NCP) tile v3.0.1 or earlier, do not upgrade to this patch. The stemcell in this patch is not compatible with the NCP tile v3.0.1 and causes the openvswitch job to fail when you deploy.
• [Bug Fix] Add a new cache configuration to the NFS service allowing service instances to enable file attribute caching and achieve directory listing performance similar to the nfs-legacy service
• [Bug Fix] Purged and re-seeded AppUsageEvents now contain parent app guid/name
• [Bug Fix] Fix Autoscaler logging to respect the ‘Enable Verbose Logging’ checkbox
• [Bug Fix] Remove invalid characters (such as underscores) in hostnames in outgoing application syslog messages to comply with RFC 5424
• Bump ubuntu-xenial stemcell to version 621.76
• Bump capi to version 1.87.7
• Bump cflinuxfs3 to version 0.195.0
• Bump diego to version 2.47.0
• Bump java-offline-buildpack to version 4.31
• Bump loggregator-agent to version 5.2.10
• Bump nfs-volume to version 5.0.9
• Bump push-usage-service-release to version 671.0.14

2.8.11

Release Date: 06/15/2020

• The option to enable dynamic egress through the PAS UI is removed. To administer App Security Groups (ASGs) for your apps instead of dynamic egress policies, see App Security Groups.
• [Feature Improvement] Service instances can send metrics to logcache over syslog
• [Feature Improvement] Upgrade Bellsoft JDK to version 11.0.7+10
• [Bug Fix] Update App Metrics UAA client to support cloud_controller.admin scope
• [Bug Fix] Usage Service - Backfill missing service name fields in usage reports
• [Bug Fix] Fix issue preventing rolling deployments from working with Windows apps
• [Bug Fix] Gorouter - Drain timeout always uses configured value
• [Bug Fix] Silk - Continue container networking during cell drain
• [Bug Fix] Pass through arbitrary parameters when binding a service to a route in Apps Manager
• [Bug Fix] Prevent click into Apps Manager search bar from erroring out when data is not fully loaded
• [Bug Fix] Improve monitoring metrics for Usage Service
• Bump ubuntu-xenial stemcell to version 621.75
• Bump capi to version 1.87.6
• Bump cf-networking to version 2.30.0
• Bump cflinuxfs3 to version 0.193.0
• Bump push-apps-manager-release to version 671.0.14
• Bump push-usage-service-release to version 671.0.12
• Bump routing to version 0.201.0
• Bump silk to version 2.30.0
• Bump uaa to version 74.5.17

2.8.10

Release Date: 06/03/2020

• [Security Fix] Fix minor CVEs in Credhub server from dependent libraries
• [Feature] Allow egress traffic from apps to addresses on host via host_tcp_services
• [Feature Improvement] HTTP trace requests now respond with a generic error page
• [Bug Fix] Safeguard against unavailable stack traces in Spring and Steeltoe threaddump actuator endpoint in Apps Manager
• [Bug Fix] Update Reverse Log Proxies to fix shutdown issues in Loggregator
• [Bug Fix] Migrate services/intermediate_tls_ca to /services/tls_leaf for Maestro
• Bump cf-smoke-tests to version 40.0.128
• Bump cflinuxfs3 to version 0.189.0
• Bump credhub to version 2.5.12
• Bump go-offline-buildpack to version 1.9.12
• Bump loggregator to version 106.2.8
• Bump nfs-volume to version 5.0.6
• Bump nodejs-offline-buildpack to version 1.7.18
• Bump push-apps-manager-release to version 671.0.11
• Bump uaa to version 74.5.16

2.8.9

Release Date: 05/18/2020

• [Security Fix] Support various CVE impacted components
• [Bug Fix] Fix scheduling issue in loggregator agent by upgrading to Go 1.14.2
• [Bug Fix] Fix issue in App Autoscaler where rules that were based on the HTTP-throughput metric failed to fire. For information about the HTTP Throughput metric, see Default Metrics for Scaling Rules.
• [Bug Fix] Fix issue in App Autoscaler where the Scheduler API returned an error when executes_at was set to a time that was in the past.
• Bump ubuntu-xenial stemcell to version 621.74
• Bump cf-autoscaling to version 230
• Bump cflinuxfs3 to version 0.180.0
• Bump loggregator-agent to version 5.2.9

2.8.8

Release Date: 05/05/2020

• [Security Fix] Update debian packages and source libraries in nfs and mapfs releases
• [Feature Improvement] Improved access logging and bumped versions of Jackson and MariaDB
• [Feature Improvement] Autoscaler only skips certificate validation when no Database CA is provided
• [Bug Fix] Performance and stability improvements in Log Cache
• [Bug Fix] Cloud Controller only checks for bucket presence on startup instead of every call to blobstore
• [Bug Fix] Fix bug that caused Apps Manager to error out on clicking into the search bar
• [Bug Fix] Show full list of jobs for an app in Apps Manager
• Bump ubuntu-xenial stemcell to version 621.71
• Bump capi to version 1.87.4
• Bump cflinuxfs3 to version 0.177.0
• Bump dotnet-core-offline-buildpack to version 2.3.9
• Bump go-offline-buildpack to version 1.9.11
• Bump log-cache to version 2.6.14
• Bump mapfs to version 1.2.4
• Bump nfs-volume to version 5.0.5
• Bump nginx-offline-buildpack to version 1.1.8
• Bump nodejs-offline-buildpack to version 1.7.17
• Bump php-offline-buildpack to version 4.4.13
• Bump push-apps-manager-release to version 671.0.10
• Bump python-offline-buildpack to version 1.7.13
• Bump r-offline-buildpack to version 1.1.4
• Bump ruby-offline-buildpack to version 1.8.17
• Bump smb-volume to version 3.0.1
• Bump staticfile-offline-buildpack to version 1.5.6
• Bump uaa to version 74.5.15

2.8.7

Release Date: 04/22/2020

• [Security Fix] Update netaddr library to prevent misconfigured file permissions in CAPI Release
• [Feature] HAProxy can now be configured with custom certificate authorities
• [Feature Improvement] Autoscaler uses TLS to communicate with its database
• [Bug Fix] Fix Certificates in CredHub KMS Provider Interface
• [Bug Fix] Fix server_name value to use Common Name as metrics_agent_metrics_tls
• Bump ubuntu-xenial stemcell to version 621.64
• Bump capi to version 1.87.3
• Bump cf-cli to version 1.26.0
• Bump cflinuxfs3 to version 0.175.0

2.8.6

Release Date: 04/07/2020

• [Feature] Allow Zero Instances of MySQL Monitor
• [Feature Improvement] Apps Manager Revisions tab only shows the Redeploy button for revisions that can be redeployed
• [Bug Fix] Fix DNS Loggregator Trafficcontroller DNS issue
• [Bug Fix] Bump Tomcat in UAA to fix SAML login issues
• [Bug Fix] garden-runc - bump to latest release in supported versions
• [Bug Fix] Apps now show a status of down instead of crashed
• [Bug Fix] You can now view logs larger than 4 MB in Apps Manager
• [Bug Fix] Honor option to hide service plan prices in Apps Manager services tables
• [Bug Fix] Expose buildpack versions in the Settings tab for the app in Apps Manager
• [Bug Fix] Apps Manager accepts numerical status, such as 500, from the Spring and Steeltoe health actuator endpoints
• [Bug Fix] Fix error when viewing the Settings tab for an app in Apps Manager while a Spring or Steeltoe app is restarting
• [Bug Fix] Gorouter correctly handles control characters in URLs
• [Bug Fix] Ensure usage service correctly considers usage events when installed after PAS
• [Bug Fix] Fix a memory leak and go-routine leak related to having multiple aggregate drains in Loggregator
• [Bug Fix] App developers now receive a 401 when using an expired access token with policy server
• [Bug Fix] Autoscaler smoke test works when router rejects requests on port 80
• [Security fix] Bump backup and restore SDK
• Bump ubuntu-xenial stemcell to version 621.61
• Bump backup-and-restore-sdk to version 1.17.4
• Bump cf-autoscaling to version 226
• Bump cf-networking to version 2.28.0
• Bump cflinuxfs3 to version 0.174.0
• Bump dotnet-core-offline-buildpack to version 2.3.7
• Bump garden-runc to version 1.19.10
• Bump go-offline-buildpack to version 1.9.8
• Bump java-offline-buildpack to version 4.29.1
• Bump loggregator-agent to version 5.2.8
• Bump loggregator to version 106.2.6
• Bump nginx-offline-buildpack to version 1.1.6
• Bump nodejs-offline-buildpack to version 1.7.15
• Bump php-offline-buildpack to version 4.4.9
• Bump push-apps-manager-release to version 671.0.9
• Bump push-usage-service-release to version 671.0.9
• Bump python-offline-buildpack to version 1.7.10
• Bump r-offline-buildpack to version 1.1.2
• Bump routing to version 0.199.0
• Bump ruby-offline-buildpack to version 1.8.14
• Bump silk to version 2.28.0
• Bump staticfile-offline-buildpack to version 1.5.5
• Bump uaa to version 74.5.13

2.8.5

Note: This release includes breaking changes, security fixes, new features, feature improvements, and bug fixes from PAS v2.8.4 and v2.8.5. For the list of changes included in PAS v2.8.4, see 2.8.4 below.

Release Date: 03/13/2020

• [Security Fix] Improve autoscaler HTTP throughput calculation performance and omit DATABASE_URL from logs
• [Bug Fix] Fix bug that prevented usage report in Apps Manager from displaying when only partial data is available
• [Bug Fix] Wrap long resource names in Apps Manager’s usage report and invite members flow
• [Bug Fix] Fix DNS Interaction between Loggregator Agent and Doppler
• Bump ubuntu-xenial stemcell to version 621.59
• Bump cf-autoscaling to version 223
• Bump cf-cli to version 1.25.0
• Bump cflinuxfs3 to version 0.169.0
• Bump dotnet-core-offline-buildpack to version 2.3.6
• Bump go-offline-buildpack to version 1.9.7
• Bump loggregator-agent to version 5.2.7
• Bump loggregator to version 106.2.4
• Bump nginx-offline-buildpack to version 1.1.5
• Bump nodejs-offline-buildpack to version 1.7.13
• Bump php-offline-buildpack to version 4.4.8
• Bump push-apps-manager-release to version 671.0.8
• Bump python-offline-buildpack to version 1.7.8
• Bump ruby-offline-buildpack to version 1.8.12
• Bump staticfile-offline-buildpack to version 1.5.4
• Bump uaa to version 74.5.10

2.8.4 - Do Not Use

Warning: Pivotal advises that you do not upgrade to PAS v2.8.4. Pivotal is investigating issues reported with the stability of PAS v2.8.4. If you have already upgraded to PAS v2.8.4 and are experiencing instability in your PAS deployment, such as cf push failure, contact Support.

Note: PAS v2.8.5 includes the breaking changes, security fixes, new features, feature improvements, and bug fixes listed below. For more changes included in PAS v2.8.5, see 2.8.5 above.

Release Date: 03/02/2020

• [Breaking Change] Autoscaler controls do not appear for apps in Apps Manager. For the workaround, see the known issue Autoscaler Controls Do Not Appear in Apps Manager for Proxied Setups.
• [Breaking Change] PCF Metrics link disappears from Apps Manager. For the workaround, see the known issue PCF Metrics Link Disappears in Apps Manager for Proxied Setups.
• [Security Fix] Stop logging private data in background jobs
• [Security Fix] Fix vulnerabilities CVE-2019-2426, CVE-2019-2449, CVE-2019-2422 in Credhub
• [Security Fix] Fix Vulnerability CVE-2020-5402 in UAA
• [Feature] Support Maestro’s rotation capability by adding Services TLS CA to all App containers
• [Feature Improvement] Add circuit breaker to Logcache so users can push apps even if metrics are unavailable
• [Feature Improvement] Bring bug fixes and improvements in latest routing releases to all supported PAS versions
• [Feature Improvement] The latest routing release adds the gorouter_time field, which logs the total time it takes for a request to travel through Gorouter. Because this changes the access log format, you might need to update your external monitoring configuration. For more information, see About Access Logs.
• [Feature Improvement] Allow users to specify a list of URIs for other foundations that Apps Manager manages
• [Feature Improvement] Introduce read-only capabilities in Apps Manager for users with cloud_controller.global_auditor and cloud_controller.admin_read_only scopes
• [Feature Improvement] Safeguard deletion of spaces in Apps Manager
• [Feature Improvement] Improve the service instance creation flow in Apps Manager to better represent plan costs and features
• [Feature Improvement] Improve performance of app logs loaded in Apps Manager
• [Bug Fix] Fix display of Pivotal logo in footer of Apps Manager for Internet Explorer users
• [Bug Fix] Allow users with usage_service.audit scope to view Usage Report in Apps Manager
• [Bug Fix] Increase responsiveness of Apps Manager sidebar services count
• [Bug Fix] Remove unnecessary suggestion to restage apps when binding the App Autoscaler to apps in Apps Manager
• [Bug Fix] Surface errors from Spring and Steeltoe trace actuator endpoint in Apps Manager
• [Bug Fix] Account for asynchronous Spring and Steeltoe health actuator endpoint responses in Apps Manager
• [Bug Fix] Fix bug that prevented users from navigating beyond first page of app revisions in Apps Manager
• [Bug Fix] Fix issue that caused Apps Manager to show a 404 after renaming an org
• [Bug Fix] Show full space and organization names, regardless of length, in Apps Manager
• [Bug Fix] Use more informative description “Last Update” instead of “Last Push” in Apps Manager
• [Bug Fix] Log only necessary information when auction scoring fails
• [Bug Fix] Fix Race Condition in Loggregator Agent
• [Bug Fix] Cloud Controller no longer tries to connect to Copilot when it is not deployed
• Bump ubuntu-xenial stemcell to version 621.57
• Bump capi to version 1.87.2
• Bump cf-smoke-tests to version 40.0.127
• Bump cflinuxfs3 to version 0.164.0
• Bump credhub to version 2.5.11
• Bump diego to version 2.44.0
• Bump log-cache to version 2.6.12
• Bump loggregator-agent to version 5.2.6
• Bump push-apps-manager-release to version 671.0.6
• Bump routing to version 0.198.0
• Bump uaa to version 74.5.9

2.8.3

Release Date: 02/06/2020

• [Security Fix] CVE-2020-5399 - Use TLS for MySQL database connections in Credhub
• [Feature Improvement] Replace Metric Forwarder integration with Metric Registrar integration in Apps Manager
• [Feature Improvement] The HSM Client Private Key for CredHub can be encrypted.
• [Feature Improvement] Use the Diego logging format for the Garden job
• [Bug Fix] Show spring mappings in Apps Manager for apps using Spring Boot 2.2.x
• [Bug Fix] Add empty state message to Marketplace for orgs without spaces in Apps Manager
• [Bug Fix] Add support for non-ASCII characters in app logs shown in Apps Manager
• [Bug Fix] The Apps Manager bound services list correctly shows the number of bound apps when a table is paginated
• [Bug Fix] Show full app name, regardless of length, in Apps Manager
• [Bug Fix] Allow users with cloud_controller.global_auditor scope to view Cloud Controller resources in Apps Manager
• [Bug Fix] Allow users with cloud_controller.admin_read_only scope to view Cloud Controller resources in Apps Manager, including secrets
• [Bug Fix] When you click the Restage App option, Apps Manager renders the restage app modal.
• [Bug Fix] For apps using Spring Boot 2.2.x, show Spring Health information in Apps Manager
• [Bug Fix] Wait for necessary information to load in Apps Manager before rendering link to recently accessed apps
• [Bug Fix] Disable hostname validation for external DBs in routing-api, silk-controller, and policy-server
• [Bug Fix] HAProxy returns with HTTP/1.1 proto for 504s
• Bump ubuntu-xenial stemcell to version 621.51
• Bump cflinuxfs3 to version 0.160.0
• Bump credhub to version 2.5.10
• Bump dotnet-core-offline-buildpack to version 2.3.4
• Bump go-offline-buildpack to version 1.9.5
• Bump nginx-offline-buildpack to version 1.1.4
• Bump nodejs-offline-buildpack to version 1.7.9
• Bump php-offline-buildpack to version 4.4.6
• Bump push-apps-manager-release to version 671.0.3
• Bump python-offline-buildpack to version 1.7.6
• Bump ruby-offline-buildpack to version 1.8.8

2.8.2

Release Date: 01/16/2020

• [Security Fix] Several security issues were fixed in MySQL USN-4070-1, USN-4195-1
• [Feature] Expose PAS database metrics in the Healthwatch Indicator Protocol dashboard
• [Bug Fix] mapfs - Fix error when appending to a file
• Bump ubuntu-xenial stemcell to version 621.41
• Bump binary-offline-buildpack to version 1.0.36
• Bump cf-cli to version 1.24.0
• Bump cf-smoke-tests to version 40.0.125
• Bump cflinuxfs3 to version 0.153.0
• Bump dotnet-core-offline-buildpack to version 2.3.3
• Bump go-offline-buildpack to version 1.9.4
• Bump nginx-offline-buildpack to version 1.1.3
• Bump nodejs-offline-buildpack to version 1.7.8
• Bump php-offline-buildpack to version 4.4.5
• Bump pxc to version 0.22.0
• Bump python-offline-buildpack to version 1.7.5
• Bump r-offline-buildpack to version 1.1.1
• Bump ruby-offline-buildpack to version 1.8.6
• Bump staticfile-offline-buildpack to version 1.5.3

2.8.1

Release Date: 12/26/2019

• [Security Fix] App Usage Service - Bump Nokogiri to 1.10.5 to fix CVE-2019-13117
• [Security Fix] CVE-2019-17596 - Fix panic upon an attempt to process network traffic containing an invalid DSA public key for syslog release
• [Security Fix] CVE-2019-17596 - Fix panic upon an attempt to process network traffic containing an invalid DSA public key for garden-runc release
• [Security Fix] CVE-2019-17596 - Fix panic upon an attempt to process network traffic containing an invalid DSA public key for loggregator releases
• [Feature] Expose all platform metrics on Prometheus endpoints
• [Feature Improvement] Upgrade nats release to use go 1.13 release
• [Feature Improvement] Notifications service will skip hostname validation for external databases
• [Feature Improvement] Clarify wording of Marketplace URL help text in Apps Manager configuration
• [Feature Improvement] Add doppler.firehose and usage_service.audit to Apps Manager client
• [Feature Improvement] Always enable ssh-proxy TLS to backend instances to ensure widest compatibility mode with PASW and IST
• [Feature Improvement] When users have correct permissions, show bound and bindable services shared from other spaces as bindable for an app in Apps Manager
• [Feature Improvement] When users have correct permissions, show bound and bindable apps from spaces a service instance has been shared to in Apps Manager
• [Bug Fix] Fix bug that prevented users from downloading the Accounting and Usage Service reports through Apps Manager when fields are undefined or null
• [Bug Fix] Prevent new requests from being made when clicking on the currently active tab in Apps Manager
• [Bug Fix] Fix bug that prevented additional resources from populating after user permissions load in Apps Manager
• [Bug Fix] Fix bug preventing multiple service instances without binding names from being bound to apps in Apps Manager
• [Bug Fix] Exclude user provided service instances from org level service instance hours on Usage Report in Apps Manager
• [Bug Fix] Account for malformed git properties in Spring and Steeltoe apps to keep Apps Manager from crashing on render
• [Bug Fix] Fix bug where 'Invalid Date’ was shown in Apps Manager trace tab when using Spring v2.0
• [Bug Fix] Prevent Apps Manager’s revisions tab from crashing out when a deployment is in progress
• [Bug Fix] Move tooltip in the Apps Manager bind services flyout to make text fully visible
• [Bug Fix] Prevent attempts to build a droplet when starting an app through Apps Manager if there is no associated package
• [Bug Fix] Passwords containing commas no longer cause the SMB volume service to crash at startup with a “mount failed” error
• [Bug Fix] All CAPI jobs respect “Maximum disk quota per app”
• Bump ubuntu-xenial stemcell to version 621.29
• Bump cf-smoke-tests to version 40.0.124
• Bump cflinuxfs3 to version 0.151.0
• Bump garden-runc to version 1.19.9
• Bump log-cache to version 2.6.1
• Bump loggregator-agent to version 5.2.1
• Add new release metrics-discovery at version 2.0.2
• Bump nats to version 28
• Bump push-apps-manager-release to version 671.0.2
• Bump push-usage-service-release to version 671.0.8
• Bump pxc to version 0.21.0
• Bump smb-volume to version 2.1.1
• Bump statsd-injector to version 1.11.8
• Bump syslog to version 11.6.1
• Bump system-metrics-scraper to version 2.0.3

2.8.0

• See New Features in PAS v2.8
• See Breaking Changes

How to Upgrade

To upgrade to PAS v2.8, see Upgrading Pivotal Platform.

When upgrading to PAS v2.8, be aware of the following upgrade considerations:

• If you previously used an earlier version of PAS, you must first upgrade to PAS v2.7 to successfully upgrade to PAS v2.8.

• Some partner service tiles may be incompatible with Pivotal Platform v2.8. Pivotal is working with partners to ensure their tiles are updated to work with the latest versions of Pivotal Platform.
  
  For information about which partner service releases are currently compatible with Pivotal Platform v2.8, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.

New Features in PAS v2.8

PAS v2.8 includes the following major features:

Diego Sets Container CPU Weight Property Equal to Container Memory Limit

Diego sets the CPU weight property on the containers it creates to a number equivalent to the container memory limit. This allows Garden to calculate the AbsoluteCPUEntitlement metric, which is the CPU entitlement for the container. With AbsoluteCPUEntitlement, Garden can produce accurate CPU usage metrics that are relative to AbsoluteCPUEntitlement.

For more information about the AbsoluteCPUEntitlement metric, see Diego Container Metrics in Container Metrics. For information about the Cloud Foundry CPU Entitlement Plugin, an experimental plugin that allows you to examine the CPU usage of PAS apps relative to their CPU entitlement, see the Cloud Foundry CPU Entitlement Plugin repository on GitHub.

Spring Cloud Services Configuration in Apps Manager

For Spring Cloud Services (SCS) instances, Apps Manager shows the current status of the SCS Config Server and lets you trigger the Config Server to update app configurations.

For more information, see View and Update Spring Cloud Services Configurations in Managing Apps and Service Instances Using Apps Manager.

View the Active Droplet for an App in Apps Manager

On the Apps Manager Revisions page for an app, you can view which revision version contains the active droplet for the app. The active droplet has a GUID that is equivalent to the current droplet GUID of the app.

The text Deployed (Active) appears in the Status column of the table to indicate that the revision version is active.

For more information about using the Apps Manager UI, see Using Apps Manager.

cf CLI Supports Sidecar Processes

The Cloud Foundry Command-Line Interface (cf CLI) adds support for sidecar processes. You can add a sidecar process to an app process using an app manifest. The cf CLI displays the sidecar process alongside the app process to which it is attached.

For more information about deploying sidecar processes with apps, see Pushing Apps with Sidecar Processes (Beta).

Deploy Sidecar Processes with a Custom Buildpack

As an alternative to using an app manifest, you can deploy a sidecar process for an app with a custom buildpack.

Note: PAS v2.8 does not support using custom buildpacks to push sidecars for Java apps. You can only push sidecars for Java apps by using an app manifest.

For more information, see Sidecar Buildpacks.

Configure Retention Period for Usage Service Data

Usage Service deletes granular data after 365 days by default. You can configure the retention period for granular data in the Advanced Features pane of the PAS tile.

This feature reduces the amount of data in the Usage Service database, which helps prevent data migration issues on very large foundations.

For more information, see Usage Data Retention in Reporting App, Task, and Service Instance Usage.

SMB Volume Services Enabled by Default

SMB volume services are enabled by default. When SMB volume services are enabled, app developers can bind existing SMB shares to apps for shared file access.

To disable SMB volume services in the PAS tile, select App Containers and clear the Enable SMB volume services checkbox, and then select Errands and set the SMB Broker Errand to Off.

For more information, see Enable SMB Volume Services in Enabling Volume Services.

For general information about volume services, see Using an External File System (Volume Services).

NFS Broker Uses CredHub as Backing Store

NFS Broker uses CredHub as its backing store, rather than an internal PAS database. Because BOSH Backup and Restore (BBR) no longer backs up NFS Broker, the nfsbroker-bbr job is removed.

For more information about CredHub, see CredHub.

Enable URL Encoding For UAA Client Credentials

You can disable an optional Client basic auth compatibility mode checkbox in the UAA pane of the PAS tile to require URL encoding for UAA client credentials.

URL encoding is defined by RFC 6749. For more information, see the 2.3.1. Client Password section of RFC 6749.

By default, the Client basic auth compatibility mode checkbox is enabled. When the checkbox is enabled, UAA does not require URL encoding for client IDs and secrets. This represents the default behavior of UAA prior to v74.0.0. For more information, see v74.0.0 in GitHub.

For more information about configuring the Client basic auth compatibility mode checkbox, see Configure UAA in Configuring PAS.

Warning: If you disable the Client basic auth compatibility mode checkbox, URL encoding is required for all UAA client apps in your deployment. To avoid breaking changes, ensure that all client apps support URL encoding before you disable the checkbox.

Support for Pushing Container Images Hosted in AWS ECR

When you push container images hosted in AWS Elastic Container Registry (ECR) with the Cloud Foundry CLI (cf CLI), you can provide the access key ID and secret for an AWS IAM user as a Docker username and password as part of the cf push command. Apps are able to then continuously restart and restage successfully.

This update allows the cf CLI to successfully pull container images hosted in ECR with valid AWS Identity and Access Management (IAM) user credentials.

For more information, see Amazon Elastic Container Registry (ECR) in Deploying an App with Docker.

Mutual TLS Added to Loggregator Endpoints and Components

Mutual TLS is added to the Loggregator, Loggregator Agent, and Log Cache endpoints. It is also added to the Leadership Election job. This provides additional security between these endpoints and metric scrapers.

For more information about Loggregator components, see Loggregator Architecture. For more information about the Leadership Election job and metric scraping, see the System Metrics repository on GitHub.

V2 Firehose Can Be Disabled

You can disable the Loggregator V2 Firehose by deselecting the Enable V2 Firehose checkbox in the System Logging pane of the PAS tile. This shuts down VMs used for the V2 Firehose, such as Dopplers and Reverse Log Proxies. After you disable the V2 Firehose, you can delete these VMs from your deployment to save resources.

Warning: If you disable the V1 or V2 Firehose, you must disable the Smoke Test Errand or the deploy fails. For more information, see Disable the Smoke Test Errand If You Disable the Firehose.

To forward logs and metrics to a syslog endpoint after you disable the Firehose, configure an aggregate log and metric drain for your foundation. For more information about disabling the V2 Firehose and enabling aggregate drains, see Configure System Logging in Configuring PAS.

For more information about the Loggregator Firehose, see Loggregator Architecture.

Aggregate Drain for Metrics and App Logs

You can configure an aggregate log and metric drain for your foundation to allow Syslog Agents to forward all app metrics, app logs, and PAS component VM metrics to one or more syslog endpoints.

This allows you to forward logs and metrics for all apps in your foundation without configuring syslog drains for each app individually.

You can also use an aggregate log and metric drain instead of the Loggregator Firehose. This allows you to disable the Firehose and delete related VMs, such as Dopplers and Reverse Log Proxies. For more information about disabling the Firehose, see V2 Firehose Can Be Disabled.

To enable an aggregate log and metric drain for your foundation, add a comma-separated list of syslog endpoints to the Aggregate log and metric drain destinations field in the System Logging pane of the PAS tile. For more information, see Configure System Logging in Configuring PAS.

PAS Must Use At Least One CredHub VM

In PAS v2.8, you must use at least one CredHub VM. The default number of CredHub instances is increased from 0 to 2. You can configure the number of CredHub VMs PAS uses in the Resource Config pane of the PAS tile.

For high availability, Pivotal recommends that you use at least one CredHub instance per availability zone (AZ). Or, if you have only one AZ, use at least three CredHub instances.

Warning: If you use an external GCP or Azure database for PAS and previously set CredHub instances to 0 in PAS v2.7, you must also disable hostname verification before you upgrade to PAS v2.8. For more information, see Disable Hostname Verification for External CredHub Databases on GCP and Azure in Upgrade Preparation Checklist for Pivotal Platform v2.8.

For more information about CredHub, see CredHub. For more information about configuring VMs, see Configure Resources in Configuring PAS.

Customizable Marketplace URL and Secondary Navigation Links in Apps Manager

In the Custom Branding pane of PAS v2.8, you can customize the Marketplace URL and secondary navigation links that appear in your Apps Manager deployment.

For more information, see Configure Custom Branding and Apps Manager in Configuring PAS.

Agent-Based Syslog Egress Cannot Be Disabled

Agent-based syslog egress is enabled in PAS v2.8.0 and cannot be disabled. Syslog Agents forward logs to Loggregator and syslog drains that you configure. This feature removes the need for VMs dedicated to syslog drains.

For more information, see Agent-Based Syslog Egress Is Enabled by Default in the PAS v2.7 release notes.

Additional Metadata Tags for Metrics and App Logs

In PAS v2.8, six new metadata tags appear in app logs and metrics. The following metadata tags have been added to outgoing syslogs in v2.8:

• app_id
• app_name
• organization_id
• organization_name
• space_id
• space_name

The tags correspond to the internal name and ID of the app and its org and space.

The new metadata tags appear in app logs in the following format:

956 <14>1 2020-03-31T12:11:02.529497+00:00 My-org.My-space.my-app-blue ec3cd4e4-baf9-456d-965a-96bcb2c61a47
[APP/PROC/WEB/0] - [tags@47450 app_id="ec3cd4e4-baf9-456d-965a-96bcb2c61a47" app_name="my-app-blue"
deployment="cf-78e7a9442158adb53366" index="c57b95c6-d79f-4cfb-b7f3-08a770946b7a" instance_id="0"
ip="10.214.110.84" job="diego_cell" organization_id="c23f6fb2-beeb-4d2f-8c1d-693be5bd502c"
organization_name="My-org" origin="rep" process_id="ec3cd4e4-baf9-456d-965a-96bcb2c61a47"
process_instance_id="1cf3854e-29d8-4825-7685-11ec" process_type="web" product="Pivotal Application Service"
source_id="ec3cd4e4-baf9-456d-965a-96bcb2c61a47" source_type="APP/PROC/WEB"
space_id="93e9f375-1567-4ea4-b2a1-ef6fd7e0125b" space_name="My-space"
system_domain="example.mydomain.io"] 2020-03-31 07:11:02.529
INFO [my-app,B2347215-T21118008-COWF/35558169/38311791] --- [http-nio-8080-exec-5]
c.c.b.t.p.s.p.b.BatchPostProcessor : Report file created in 47 ms

For information about the log format, see Metadata Tags for Metrics and App Logs in Pivotal Platform v2.7 Breaking Changes.

For information about a known issue related to the metadata tags, see Metadata Tags Not Updated if App, Org, or Space Name Changed.

Rolling App Deployments Is GA

Rolling app deployments was released as a beta feature in PAS v2.4. This feature allows you to push updates to apps without incurring downtime.

This feature is enabled by default. You can optionally disable it in the Advanced Features pane.

Rolling App Deployments is GA through Apps Manager and v3 CC API. The rolling app deployments feature is still experimental on cf CLI v6. To use the feature as GA through the cf CLI, upgrade to TAS 2.10+

For more information, see Rolling App Deployments.

Jump Upgrade to TAS for VMs v2.11

TAS for VMs v2.11 is a long-term supported (LTS) version of PAS. TAS for VMs v2.11 will be supported through April 2024.

You can jump upgrade directly from previous versions of PAS to the LTS version of TAS for VMs. For more information, see Jump Upgrading to TAS for VMs v2.11.

Breaking Changes

PAS v2.8 includes the following breaking changes:

PAS v2.8.25 and Later Incompatible with Tanzu GemFire for VMs v1.12 and Earlier with Tomcat Session State Caching

PAS v2.8.25 and later specifies Java buildpack v4.36. This causes a breaking change if you use Tanzu GemFire for VMs v1.12 or earlier with Tomcat session state caching. For information about how to avoid this breaking change, see Known Issues in the Tanzu GemFire for VMs documentation.

Incorrect HTTP(S) Proxy Configuration Breaks CredHub Interpolation for Apps in PAS v2.8.12 and Later

In PAS v2.8.12 and later, apps that have an incorrect HTTP(S) Proxy configuration fail to stage or restart due to a CredHub interpolation error.

Before you upgrade to PAS v2.8.12 or later, you must fix the HTTP(S) Proxy configuration of any impacted applications:

1. Determine whether your apps are impacted by following the resolution procedure in Knowledgebase Article 9305.
2. Update all impacted apps to use the recommended proxy settings that are documented in Configuring Proxy Settings for All Apps.
3. Restart modified apps.

PAS v2.8.4 Stability Issues

WARNING: Pivotal advises that you do not upgrade to PAS v2.8.4 at this time. Pivotal is investigating issues reported with the stability of PAS v2.8.4.

If you have already upgraded to PAS v2.8.4 and are experiencing instability in your PAS deployment, such as cf push failure, contact Support.

CredHub Requires At Least One CredHub Instance

PAS v2.8 and later requires at least one CredHub instance. If the number of CredHub instances is set to 0, the deployment fails.

If you scaled the number of CredHub instances to 0 in PAS v2.7, check the number of CredHub instances in the Resource Config pane of the PAS v2.8 tile and increase to one or more instances before you apply changes for the upgrade.

For more information, see PAS Must Use At Least One CredHub VM in Pivotal Application Service v2.8 Release Notes.

Syslog Adapters Are Removed

Syslog Adapters are not supported in PAS v2.8. If you disabled Syslog Agents in PAS v2.7.7, you must enable Syslog Agents before you upgrade to PAS v2.8. If you do not enable Syslog Agents prior to upgrade, the upgrade fails in order to prevent potential log loss.

To enable Syslog Agents on PAS v2.7.7, do one of the following:

• Go to the System Logging pane in the PAS tile and select the Enable syslog egress through Syslog Agents checkbox.

• Set the properties.syslog_agent_enabled property to true.

Additional Metadata Tags for Metrics and App Logs

In PAS v2.8, metadata tags for app logs and metrics appear after the header of each syslog message and before the syslog message text. If your external monitoring depends on a specific format for app logs, you must update the configuration.

The new metadata tags appear in syslogs in the following format:

956 <14>1 2020-03-31T12:11:02.529497+00:00 My-org.My-space.my-app-blue ec3cd4e4-baf9-456d-965a-96bcb2c61a47
[APP/PROC/WEB/0] - [tags@47450 app_id="ec3cd4e4-baf9-456d-965a-96bcb2c61a47" app_name="my-app-blue"
deployment="cf-78e7a9442158adb53366" index="c57b95c6-d79f-4cfb-b7f3-08a770946b7a" instance_id="0"
ip="10.214.110.84" job="diego_cell" organization_id="c23f6fb2-beeb-4d2f-8c1d-693be5bd502c"
organization_name="My-org" origin="rep" process_id="ec3cd4e4-baf9-456d-965a-96bcb2c61a47"
process_instance_id="1cf3854e-29d8-4825-7685-11ec" process_type="web" product="Pivotal Application Service"
source_id="ec3cd4e4-baf9-456d-965a-96bcb2c61a47" source_type="APP/PROC/WEB"
space_id="93e9f375-1567-4ea4-b2a1-ef6fd7e0125b" space_name="My-space"
system_domain="example.mydomain.io"] 2020-03-31 07:11:02.529
INFO [my-app,B2347215-T21118008-COWF/35558169/38311791] --- [http-nio-8080-exec-5]
c.c.b.t.p.s.p.b.BatchPostProcessor : Report file created in 47 ms

For more information, see Human-Friendly Metadata in Identifying the Source Deployment of Metrics.

Agent-Based Syslog Egress Cannot Be Disabled

If you disabled Agent-based syslog egress in PAS v2.7.7 or later, then you must update your external monitoring configuration.

Agent-based syslog egress is always enabled in PAS v2.8.0. Agent-based syslog egress was enabled by default in PAS v2.7.0, but you could optionally disable the egress in PAS v2.7.7 and later. For more information, see Agent-Based Syslog Egress Is Enabled by Default in the Pivotal Application Service v2.7 Release Notes.

Known Issues

PAS v2.8 includes the following known issues:

Rolling App Deployment Does Not Timeout

Rolling app deployments do not properly timeout when the startup timeout is reached. You may experience a rolling app deployment process that hangs indefinitely.

If you experience a hanging rolling app deployment, you can manually terminate the process. For more information about terminating the rolling app deployment process, see the cf CLI v7 procedure in Cancel a Deployment.

Incompatible Stemcell Causes Job Failure

If you use the NSX-T Container Plugin (NCP) tile v3.0.2 or earlier, do not upgrade to PAS 2.8.19. The stemcell in this patch is not compatible with the NCP tile v3.0.2 or earlier and causes the openvswitch job to fail when you deploy.

Duplicate Metrics Appear in the Firehose

PAS v2.8 introduces a System Metrics Agent that sends metrics to the Firehose. These metrics match existing BOSH system metrics, but they use an updated format. In PAS v2.8, metrics appear in both formats. In the Firehose, you see duplicate metrics entries in both the Loggregator format and the System Metrics Agent format.

The following table shows examples of the existing Loggregator metrics format and the new System Metrics Agent metrics format:

The new System Metrics Agent adds about 50 metric envelopes per VM each minute.

The existing BOSH system metrics forwarder emits 2 metric envelopes per VM each minute. Each envelope contains 13 metrics.

Disable the Smoke Test Errand If You Disable the Firehose

If you disable the V1 or V2 Firehose in PAS v2.8, you must also disable the smoke test errand.

If you do not disable the smoke test errand, the deploy fails with an error similar to the following:

[91m[1m[Fail] [0m[90mLoggregator: [0m[0mcf logs [0m[90mlinux [0m[91m[1m[It] can see app messages in the logs [0m
          [37m/var/vcap/packages/smoke_tests/src/github.com/cloudfoundry/cf-smoke-tests/smoke/logging/loggregator_test.go:42[0m
          [1m[91mRan 1 of 2 Specs in 56.171 seconds[0m
          [1m[91mFAIL![0m -- [32m[1m0 Passed[0m | [91m[1m1 Failed[0m | [33m[1m0 Pending[0m | [36m[1m1 Skipped[0m
          --- FAIL: TestSmokeTests (56.17s)
          FAIL
          Ginkgo ran 2 suites in 1m7.050120251s
          Test Suite Failed
Stderr     Error: failed to run job-process: exit status 1 (exit status 1)

To disable the smoke test errand:

1. Navigate to the Errands pane in the PAS tile.

2. For Smoke Test Errand, select Off.

For more information, see Configure Errands in Configuring PAS.

Bosh System Metrics Forwarder Regression in v2.8.22

If you already have Healthwatch installed, upgrading to this patch will cause installation errors.

If you do not have Healthwatch, you are still impacted in downstream monitoring systems (DataDog, Splunk, etc). Those monitoring systems will start firing missing data alerts for metrics for BOSH/TAS recommended KPIs.

To resolve this issue:

If possible, do not upgrade.

If the upgrade has failed:

1. Download the TAS manifest and manually set enabled: true on the bosh_system_metrics_forwarder job.

2. Redeploy with Bosh.

DO NOT apply changes on the TAS tile until the next version is out.

Errors in NFS Volume Service File Append Operations

A defect in the mapfs FUSE driver causes errors to occur in file append operations when you enable the ID mapping feature with NFS in PAS v2.8.0 through PAS v2.8.1.

You enable the ID mapping feature by specifying either the uid or username option in service instance or service bind configurations.

When this issue occurs, appending files within the mounted file system fail with the error File operation not supported. For example, echo hello >> test.txt fails.

This issue is resolved in PAS v2.8.2.

PCF Metrics v1.6.2 and Earlier Not Compatible with PAS v2.8.3 and Later

App Metrics v1.6.2 and earlier is incompatible with the following PAS patch versions:

• PAS v2.5.20 and later
• PAS v2.6.15 and later
• PAS v2.7.9 and later
• PAS v2.8.3 and later

This incompatibility is caused by an update to nodejs-offline-buildpack v1.7.9, which removes support for Node.js 8.x.

If you upgrade to one of the PAS versions above and you are using App Metrics v1.6.2 or earlier, then App Metrics no longer works.

To resolve this issue, upgrade to App Metrics v1.6.3 or later.

For more information, see PCF Metrics v1.6.x is not compatible with PAS 2.5.20+, 2.6.15+, 2.7.9+ & 2.8.3+.

Logs Take a Long Time to Load in Apps Manager

In PAS v2.8.0 through v2.8.3, Apps Manager uses an inefficient method of loading app logs. This method causes the logs page to remain in a loading state for a long time before displaying logs. This issue is resolved in PAS v2.8.4.

Asynchronous Spring and Steeltoe health Endpoint Causes Apps Manager to Crash

This issue only applies to PAS v2.8.3 and earlier.

If your Spring or Steeltoe app uses an asynchronous health actuator endpoint, you might see the error TypeError: Cannot read property 'code' of undefined in the Overview tab for the app. The normal contents of the Overview tab are not displayed.

This does not indicate a problem with the app, and information on the other tabs displays correctly.

Errors Viewing App Logs after Disabling V1 Firehose

If you disable the V1 Firehose and you are using a version of the cf CLI earlier than v6.50, you may encounter errors when you push an app or view the logs for an app. The logs exist but are not visible from the cf CLI.

Running the following commands results in errors:

• cf logs: Timeout trying to connect to NOAA
• cf push: timeout connecting to log server, no log will be shown

Despite the log-related errors, cf push works correctly and pushes the app.

To avoid encountering errors after disabling the Loggregator V1 Firehose, upgrade to cf CLI v6.50 or later.

App Metrics v2.0.0 Is Incompatible with Apps Manager Integration

This issue affects App Metrics v2.0.0.

If the App Metrics v2.0.0 tile is installed on a foundation, then the View in Metrics link on the app Overview tab in Apps Manager does not appear or is broken.

Traffic Controller Can Issue a Large Number of Unnecessary DNS Queries

This issue is resolved in PAS v2.8.6 and later.

In PAS v2.8.5, Loggregator release version 106.2.4 uses an upgraded version of GOLANG GRPC. By default, GRPC attempts to find the hostname q-s0.doppler.NETWORK-NAME.cf-GUID.bosh through BOSH DNS for A, SRV, and TXT records. If this attempt fails or is delayed, GRPC sends DNS lookups to all of the external DNS servers listed in /etc/resolv.conf. This can result in the Loggregator Traffic Controller generating hundreds of requests per second.

Example log message from BOSH DNS server:

[RequestLoggerHandler] 2020/03/27 12:06:04 WARN - handlers.DiscoveryHandler Request qtype=[TXT] qname=[q-s0.doppler.pas.cf-GUID.bosh.] rcode=SERVFAIL time=17000ns

To work around this issue in PAS v2.8.5, see Loggregator continuously generates hundreds of DNS requests per second to external DNS servers in the Knowledge Base.

App Metrics Route Change Results In “Unexpected error occurrence”

This issue affects you only if you upgrade from App Metrics v2.0.0 to App Metrics v2.0.1 or later.

The route to App Metrics moved from appmetrics.FOUNDATION_SYSTEM_DOMAIN.com in v2.0.0 to metrics.FOUNDATION_SYSTEM_DOMAIN in v2.0.1.

If you have set the Multi-foundation configuration (beta) field of the Apps Manager section in a PAS tile, you must update the metricsUrl field to reflect the route change. If the field is not updated, then clicking View in Metrics on the app Overview tab in Apps Manager results in an Unexpected error occurence message.

Invalid Events from Cloud Controller Purge and Reseed

In PAS v2.8.11 and earlier, the /v2/app_usage_events/destructively_purge_all_and_reseed_started_apps endpoint may generate app events without valid GUIDs. These invalid GUIDs can cause errors with components that consume them when parsing and correlating events. This issue affects Cloud Controller and App Usage Service.

For more information about the API endpoint, see Purge and reseed App Usage Events in the App Usage Events API documentation. For more information about the issue, see App Usage Service startup errors and data inconsistency in the knowledge base.

This issue is resolved in PAS v2.8.12.

App Metrics v2.0 Causes Apps Manager to Log Out on PAS v2.8.3 and Earlier

For deployments with both App Metrics v2.0 and PAS v2.8.3 or earlier installed, viewing any app Overview page in Apps Manager causes the user session to expire, which results in a log out.

For deployments with PAS v2.8.3 and earlier, this issue affects all App Metrics v2.0 patch versions.

This issue is resolved in PAS v2.8.4 and later.

Metadata Tags Not Updated if App, Org, or Space Name Changed

If you rename an app, org, or space, the metadata tags for metrics and app logs do not update to reflect the new names. The original names persist in the metrics and app logs.

To resolve this issue, restart the app. After the app restarts, app logs and metrics use the updated metadata tags.

For more info about these metadata tags, see Additional Metadata Tags for Metrics and App Logs above.

PCF Metrics Link Disappears in Apps Manager for Proxied Setups

This issue affects PAS v2.8.4 and later deployments that have any version of PCF Metrics or App Metrics installed.

If your PAS deployment has restrictive networking policies around request proxying, then the View in PCF Metrics link may not appear in Apps Manager.

To resolve this issue:

1. Using the cf CLI, log in to the system org and system space.
2. Locate the search-server app.

3. Update the no_proxy environment variable for the search-server app to include your system domain.

   cf set-env search-server no_proxy '*.SYSTEM-DOMAIN'
   

   where SYSTEM-DOMAIN is the system domain configured for your PAS deployment. For example:

   cf set-env search-server no_proxy '*.example.com'
   

4. Restage the search-server app.

   cf restage search-server
   

Autoscaler Controls Do Not Appear in Apps Manager for Proxied Setups

This issue affects PAS v2.8.4 and later deployments that manage the Autoscaler service for individual apps in Apps Manager.

If your PAS deployment has restrictive networking policies around request proxying, then Autoscaler controls may not appear for apps within Apps Manager even when the Autoscaler service is enabled for an org.

To resolve this issue:

1. Using the cf CLI, log in to the system org and system space.
2. Locate the search-server app.

3. Update the no_proxy environment variable for the search-server app to include your system domain.

   cf set-env search-server no_proxy '*.SYSTEM-DOMAIN'
   

   where SYSTEM-DOMAIN is the system domain configured for your PAS deployment. For example:

   cf set-env search-server no_proxy '*.example.com'
   

Metric Registrar v1.2.1 Known Issues

The following known issues with Metrics Registrar v1.2.1 affect PAS v2.8.18.

Several issues with the Metric Registrar v1.2.1 release cause problems when upgrading to PAS v2.8.18.

These known issues led to a reversion of Metrics Registrar in PAS v2.8.19. Features introduced in PAS v2.8.18 are not available in later versions of PAS.

Metric Registrar Orchestrator Cannot Connect to Cloud Controller

When the metric_registrar_orchestrator job starts up, it exits if it cannot connect to the Cloud Controller.

This issue can arise if either BOSH DNS or the Cloud Controller fails. For example, if BOSH DNS is being rolled during an update, there may be a log line similar to the following regarding host name resolution:

unable to connect to Cloud Controller: Get \"https://q-s0.cloud-controller.env.deployment.bosh:9227/internal/v4/syslog_drain_urls\": dial tcp: lookup q-s0.cloud-controller.pcfdev802.cf-85ed2c836ae0287e088c.bosh on 10.0.0.6:23: no such host

If Cloud Controller is unavailable, there may be a log line about the Cloud Controller:

unable to connect to Cloud Controller: unexpected status from cloud controller: 503

The failure of metric_registrar_orchestrator to start can block upgrades to PAS v2.8.18.

Metric Registrar Orchestrator Crashes with Unexpected Service URLs

The metric_registrar_orchestrator job parses user-provided services and their associated URLs. It expects all service URLs to have a scheme, and if it encounters one without a scheme, it crashes with the error message:

panic: runtime error: index out of range [1] with length 1

To fix this issue, check user-provided services, and make sure each service has a scheme. For example, if one of the services has the URL my.service.com, update the scheme to http://my.service.com or whichever scheme the service is communicated over.

This issue is resolved in PAS v2.8.19 and later.

Metric Registrar Smoke Tests Fail in Environments that Enable mTLS for App Containers

This issue affects PAS v2.8.18 deployments that have the Gorouter app identity verification property enabled. When enabled, this property configures Gorouter and app containers to use mTLS to verify each other’s identity.

When running the metric_registrar_smoke_test errand, the smoke test fails to detect metrics from a secure-endpoint:

DeploymentValidation
[90m/var/vcap/data/compile/smoke_test/smoke_test/deployment_validation/deployment_validation_test.go:25[0m
  [91m[1mreceives metrics scraped from metric endpoints [It][0m
  [90m/var/vcap/data/compile/smoke_test/smoke_test/deployment_validation/deployment_validation_test.go:48[0m

  [91mMonitor could not detect metrics emitted from the registered endpoint.
  Expected
      <int>: 0
  to be >
      <int>: 0[0m

To resolve this issue:

1. Retrieve the cf manifest:

   bosh -d $CF_DEPLOYMENT manifest > /tmp/cf-manifest.yml
   

2. Update the containers.proxy.verify_subject_alt_name property of the rep job to include the server name of the Metric Registrar secure endpoint scraper:

   verify_subject_alt_name:
       - gorouter.service.cf.internal
       - ssh-proxy.service.cf.internal
       - metric_registrar_endpoint_worker_scrape_tls
   

3. Redploy with the updated manifest:

   bosh -d $CF_DEPLOYMENT deploy /tmp/cf-manifest.yml
   

This issue is resolved in PAS v2.8.19 and later.