Pivotal Application Service v2.8 Release Notes

Page last updated:

Pivotal Platform is certified by the Cloud Foundry Foundation for 2020.

Read more about the certified provider program and the requirements of providers.


How to Upgrade

The procedure for upgrading to Pivotal Application Service (PAS) v2.8 is documented in Upgrading Pivotal Platform.

When upgrading to PAS v2.8, be aware of the following upgrade considerations:

  • If you previously used an earlier version of PAS, you must first upgrade to PAS v2.7 to successfully upgrade to PAS v2.8.

  • Some partner service tiles may be incompatible with Pivotal Platform v2.8. Pivotal is working with partners to ensure their tiles are updated to work with the latest versions of Pivotal Platform.

    For information about which partner service releases are currently compatible with Pivotal Platform v2.8, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.

Releases

2.8.2

Release Date: 01/16/2020

  • [Security Fix] Several security issues were fixed in MySQL USN-4070-1, USN-4195-1
  • [Feature] Expose PAS database metrics in the Healthwatch Indicator Protocol dashboard
  • [Bug Fix] mapfs - Fix error when appending to a file
  • Bump ubuntu-xenial stemcell to version 621.41
  • Bump binary-offline-buildpack to version 1.0.36
  • Bump cf-cli to version 1.24.0
  • Bump cf-smoke-tests to version 40.0.125
  • Bump cflinuxfs3 to version 0.153.0
  • Bump dotnet-core-offline-buildpack to version 2.3.3
  • Bump go-offline-buildpack to version 1.9.4
  • Bump nginx-offline-buildpack to version 1.1.3
  • Bump nodejs-offline-buildpack to version 1.7.8
  • Bump php-offline-buildpack to version 4.4.5
  • Bump pxc to version 0.22.0
  • Bump python-offline-buildpack to version 1.7.5
  • Bump r-offline-buildpack to version 1.1.1
  • Bump ruby-offline-buildpack to version 1.8.6
  • Bump staticfile-offline-buildpack to version 1.5.3
Component Version
ubuntu-xenial stemcell621.41
backup-and-restore-sdk1.17.2
binary-offline-buildpack1.0.36
bosh-dns-aliases0.0.3
bosh-system-metrics-forwarder0.0.18
bpm1.1.5
capi1.87.0
cf-autoscaling222
cf-cli1.24.0
cf-networking2.27.0
cf-smoke-tests40.0.125
cflinuxfs30.153.0
credhub2.5.6
diego2.39.0
dotnet-core-offline-buildpack2.3.3
garden-runc1.19.9
go-offline-buildpack1.9.4
haproxy9.8.0
istio1.3.0
java-offline-buildpack4.26
log-cache2.6.1
loggregator-agent5.2.1
loggregator106.2.0
mapfs1.2.0
metric-registrar1.1.1
metrics-discovery2.0.2
mysql-monitoring9.7.0
nats28
nfs-volume5.0.2
nginx-offline-buildpack1.1.3
nodejs-offline-buildpack1.7.8
notifications-ui36
notifications61
php-offline-buildpack4.4.5
push-apps-manager-release671.0.2
push-usage-service-release671.0.8
pxc0.22.0
python-offline-buildpack1.7.5
r-offline-buildpack1.1.1
routing0.196.0
ruby-offline-buildpack1.8.6
silk2.27.0
smb-volume2.1.1
staticfile-offline-buildpack1.5.3
statsd-injector1.11.8
syslog11.6.1
system-metrics-scraper2.0.3
uaa74.5.5

2.8.1

Release Date: 12/26/2019

  • [Security Fix] App Usage Service - Bump Nokogiri to 1.10.5 to fix CVE-2019-13117
  • [Security Fix] CVE-2019-17596 - Fix panic upon an attempt to process network traffic containing an invalid DSA public key for syslog release
  • [Security Fix] CVE-2019-17596 - Fix panic upon an attempt to process network traffic containing an invalid DSA public key for garden-runc release
  • [Security Fix] CVE-2019-17596 - Fix panic upon an attempt to process network traffic containing an invalid DSA public key for loggregator releases
  • [Feature] Expose all platform metrics on Prometheus endpoints
  • [Feature Improvement] Upgrade nats release to use go 1.13 release
  • [Feature Improvement] Notifications service will skip hostname validation for external databases
  • [Feature Improvement] Clarify wording of marketplace url help text in Apps Manager configuration
  • [Feature Improvement] Add doppler.firehose and usage_service.audit to Apps Manager client
  • [Feature Improvement] Always enable ssh-proxy TLS to backend instances to ensure widest compatibility mode with PASW and IST
  • [Feature Improvement] When users have correct permissions, show bound and bindable services shared from other spaces as bindable for an app in Apps Manager
  • [Feature Improvement] When users have correct permissions, show bound and bindable apps from spaces a service instance has been shared to in Apps Manager
  • [Bug Fix] Fix bug that prevented users from downloading the Accounting and Usage Service reports through Apps Manager when fields are undefined or null
  • [Bug Fix] Prevent new requests from being made when clicking on the currently active tab in Apps Manager
  • [Bug Fix] Fix bug that prevented additional resources from populating after user permissions load in Apps Manager
  • [Bug Fix] Fix bug preventing multiple service instances without binding names from being bound to apps in Apps Manager
  • [Bug Fix] Exclude user provided service instances from org level service instance hours on Usage Report in Apps Manager
  • [Bug Fix] Allow users withusage_service.audit scope to view Usage Report in Apps Manager
  • [Bug Fix] Account for malformed git properties in Spring and Steeltoe apps to keep Apps Manager from crashing on render
  • [Bug Fix] Fix bug where ‘Invalid Date’ was shown in Apps Manager trace tab when using Spring v2.0
  • [Bug Fix] Prevent Apps Manager’s revisions tab from crashing out when a deployment is in progress
  • [Bug Fix] Move tooltip in the Apps Manager bind services flyout to make text fully visible
  • [Bug Fix] Prevent attempts to build a droplet when starting an app through Apps Manager if there is no associated package
  • [Bug Fix] Passwords containing commas no longer cause the SMB volume service to crash at startup with a “mount failed” error
  • [Bug Fix] All CAPI jobs respect “Maximum disk quota per app”
  • Bump ubuntu-xenial stemcell to version 621.29
  • Bump cf-smoke-tests to version 40.0.124
  • Bump cflinuxfs3 to version 0.151.0
  • Bump garden-runc to version 1.19.9
  • Bump log-cache to version 2.6.1
  • Bump loggregator-agent to version 5.2.1
  • Add new release metrics-discovery at version 2.0.2
  • Bump nats to version 28
  • Bump push-apps-manager-release to version 671.0.2
  • Bump push-usage-service-release to version 671.0.8
  • Bump pxc to version 0.21.0
  • Bump smb-volume to version 2.1.1
  • Bump statsd-injector to version 1.11.8
  • Bump syslog to version 11.6.1
  • Bump system-metrics-scraper to version 2.0.3
Component Version
ubuntu-xenial stemcell621.29
backup-and-restore-sdk1.17.2
binary-offline-buildpack1.0.35
bosh-dns-aliases0.0.3
bosh-system-metrics-forwarder0.0.18
bpm1.1.5
capi1.87.0
cf-autoscaling222
cf-cli1.23.0
cf-networking2.27.0
cf-smoke-tests40.0.124
cflinuxfs30.151.0
credhub2.5.6
diego2.39.0
dotnet-core-offline-buildpack2.3.2
garden-runc1.19.9
go-offline-buildpack1.9.3
haproxy9.8.0
istio1.3.0
java-offline-buildpack4.26
log-cache2.6.1
loggregator-agent5.2.1
loggregator106.2.0
mapfs1.2.1
metric-registrar1.1.1
metrics-discovery2.0.2
mysql-monitoring9.7.0
nats28
nfs-volume5.0.2
nginx-offline-buildpack1.1.1
nodejs-offline-buildpack1.7.4
notifications-ui36
notifications61
php-offline-buildpack4.4.2
push-apps-manager-release671.0.2
push-usage-service-release671.0.8
pxc0.21.0
python-offline-buildpack1.7.2
r-offline-buildpack1.1.0
routing0.196.0
ruby-offline-buildpack1.8.2
silk2.27.0
smb-volume2.1.1
staticfile-offline-buildpack1.5.1
statsd-injector1.11.8
syslog11.6.1
system-metrics-scraper2.0.3
uaa74.5.5

2.8.0

Component Version
ubuntu-xenial stemcell621.26
backup-and-restore-sdk1.17.2
binary-offline-buildpack1.0.35
bosh-dns-aliases0.0.3
bosh-system-metrics-forwarder0.0.18
bpm1.1.5
capi1.87.0
cf-autoscaling222
cf-cli1.23.0
cf-networking2.27.0
cf-smoke-tests40.0.123
cflinuxfs30.150.0
credhub2.5.6
diego2.39.0
dotnet-core-offline-buildpack2.3.2
garden-runc1.19.8
go-offline-buildpack1.9.3
haproxy9.8.0
istio1.3.0
java-offline-buildpack4.26
log-cache2.6.0
loggregator-agent5.1.0
loggregator106.2.0
mapfs1.2.1
metric-registrar1.1.1
mysql-monitoring9.7.0
nats27
nfs-volume5.0.2
nginx-offline-buildpack1.1.1
nodejs-offline-buildpack1.7.4
notifications-ui36
notifications61
php-offline-buildpack4.4.2
push-apps-manager-release671.0.1
push-usage-service-release671.0.7
pxc0.20.0
python-offline-buildpack1.7.2
r-offline-buildpack1.1.0
routing0.196.0
ruby-offline-buildpack1.8.2
silk2.27.0
smb-volume2.1.0
staticfile-offline-buildpack1.5.1
statsd-injector1.11.1
syslog11.5.0
system-metrics-scraper2.0.1
uaa74.5.5

New Features in PAS v2.8

Diego Sets Container CPU Weight Property Equal to Container Memory Limit

Diego sets the CPU weight property on the containers it creates to a number equivalent to the container memory limit. This allows Garden to calculate the AbsoluteCPUEntitlement metric, which is the CPU entitlement for the container. With AbsoluteCPUEntitlement, Garden can produce accurate CPU usage metrics that are relative to AbsoluteCPUEntitlement.

For more information about the AbsoluteCPUEntitlement metric, see the Diego Container Metrics section of the Container Metrics topic. For information about the Cloud Foundry CPU Entitlement Plugin, an experimental plugin that allows you to examine the CPU usage of PAS apps relative to their CPU entitlement, see the Cloud Foundry CPU Entitlement Plugin repository on GitHub.

Spring Cloud Services Configuration in Apps Manager

For Spring Cloud Services (SCS) instances, Apps Manager shows the current status of the SCS Config Server and lets you trigger the Config Server to update app configurations.

For more information, see the View and Update Spring Cloud Services Configurations section of the Managing Apps and Service Instances Using Apps Manager topic.

View the Active Droplet for an App in Apps Manager

On the Apps Manager Revisions page for an app, you can view which revision version contains the active droplet for the app. The active droplet has a GUID that is equivalent to the current droplet GUID of the app.

The text Deployed (Active) appears in the Status column of the table to indicate that the revision version is active.

For more information about using the Apps Manager UI, see Using Apps Manager.

Deploy Sidecar Processes with a Buildpack

You can deploy a sidecar process for an app with a buildpack rather than with an app manifest.

For more information, see Sidecar Buildpacks.

cf CLI Supports Sidecar Processes

The Cloud Foundry Command-Line Interface (cf CLI) adds support for sidecar processes. You can add a sidecar process to an app process using an app manifest. The cf CLI displays the sidecar process alongside the app process to which it is attached.

For more information about deploying sidecar processes with apps, see Pushing Apps with Sidecar Processes (Beta).

Configure Retention Period for Usage Service Data

Usage Service deletes granular data after 365 days by default. You can configure the retention period for granular data in the Advanced Features pane of the PAS tile.

This feature reduces the amount of data in the Usage Service database, which helps prevent data migration issues on very large foundations.

For more information, see the Usage Data Retention section of the Reporting App, Task, and Service Instance Usage topic.

SMB Volume Services Enabled by Default

SMB volume services are enabled by default. When SMB volume services are enabled, app developers can bind existing SMB shares to apps for shared file access.

To disable SMB volume services in the PAS tile, select App Containers and clear the Enable SMB volume services checkbox, and then select Errands and set the SMB Broker Errand to Off.

For more information, see the Enable SMB Volume Services section of the Enabling Volume Services topic.

For general information about volume services, see Using an External File System (Volume Services).

NFS Broker Uses CredHub as Backing Store

NFS Broker uses CredHub as its backing store, rather than an internal PAS database. Because BOSH Backup and Restore (BBR) no longer backs up NFS Broker, the nfsbroker-bbr job is removed.

For more information about CredHub, see CredHub.

Enable URL Encoding For UAA Client Credentials

You can disable an optional Client basic auth compatibility mode checkbox in the UAA pane of the PAS tile to require URL encoding for UAA client credentials.

URL encoding is defined by RFC 6749. For more information, see the 2.3.1. Client Password section of RFC 6749.

By default, the Client basic auth compatibility mode checkbox is enabled. When the checkbox is enabled, UAA does not require URL encoding for client IDs and secrets. This represents the default behavior of UAA prior to v74.0.0. For more information, see v74.0.0 in GitHub.

For more information about configuring the Client basic auth compatibility mode checkbox, see the Configure UAA section of the Configuring PAS topic.

Warning: If you disable the Client basic auth compatibility mode checkbox, URL encoding is required for all UAA client apps in your deployment. To avoid breaking changes, ensure that all client apps support URL encoding before you disable the checkbox.

Support for Pushing Container Images Hosted in AWS ECR

When you push container images hosted in AWS Elastic Container Registry (ECR) with the Cloud Foundry CLI (cf CLI), you can provide the access key ID and secret for an AWS IAM user as a Docker username and password as part of the cf push command. Apps are able to then continuously restart and restage successfully.

This update allows the cf CLI to successfully pull container images hosted in ECR with valid AWS Identity and Access Management (IAM) user credentials.

For more information, see the Amazon Elastic Container Registry (ECR) section of the Deploying an App with Docker topic.

Mutual TLS Added to Loggregator Endpoints and Components

Mutual TLS is added to the Loggregator, Loggregator Agent, and Log Cache endpoints. It is also added to the Leadership Election job. This provides additional security between these endpoints and metric scrapers.

For more information about Loggregator components, see Loggregator Architecture. For more information about the Leadership Election job and metric scraping, see the System Metrics repository on GitHub.

V2 Firehose Can Be Disabled

You can disable the Loggregator V2 Firehose by deselecting the Enable V2 Firehose checkbox in the System Logging pane of the PAS tile. This shuts down VMs used for the V2 Firehose, such as Dopplers and Reverse Log Proxies. After you disable the V2 Firehose, you can delete these VMs from your deployment to save resources.

Warning: If you disable the V2 Firehose, you must select the Enable Log Cache syslog ingestion checkbox, or logs and metrics do not appear in Log Cache. Pivotal recommends that you do not disable the Firehose if you are dependent on any of the following:
  • Service tile metrics
  • Pivotal Healthwatch or Pivotal App Metrics
  • Partner log or metric integrations

Warning: If you disable the V1 or V2 Firehose, you must disable the Smoke Test Errand or the deploy fails. For more information, see Disable the Smoke Test Errand If You Disable the Firehose.

To forward logs and metrics to a syslog endpoint after you disable the Firehose, configure an aggregate log and metric drain for your foundation. For more information about disabling the V2 Firehose and enabling aggregate drains, see the Configure System Logging section of the Configuring PAS topic.

For more information about the Loggregator Firehose, see Loggregator Architecture.

Aggregate Drain for Metrics and App Logs

You can configure an aggregate log and metric drain for your foundation to allow Syslog Agents to forward all app metrics, app logs, and PAS component VM metrics to one or more syslog endpoints.

This allows you to forward logs and metrics for all apps in your foundation without configuring syslog drains for each app individually.

You can also use an aggregate log and metric drain instead of the Loggregator Firehose. This allows you to disable the Firehose and delete related VMs, such as Dopplers and Reverse Log Proxies. For more information about disabling the Firehose, see V2 Firehose Can Be Disabled.

To enable an aggregate log and metric drain for your foundation, add a comma-separated list of syslog endpoints to the Aggregate log and metric drain destinations field in the System Logging pane of the PAS tile. For more information, see the Configure System Logging section of the Configuring PAS topic.

PAS Must Use At Least One CredHub VM

In PAS v2.8, you must use at least one CredHub VM. The default number of CredHub instances is increased from 0 to 2. You can configure the number of CredHub VMs PAS uses in the Resource Config pane of the PAS tile.

For high availability, Pivotal recommends that you use at least one CredHub instance per availability zone (AZ). Or, if you have only one AZ, use at least three CredHub instances.

Warning: If you use an external GCP or Azure database for PAS and previously set CredHub instances to 0 in PAS v2.7, you must also disable hostname verification before you upgrade to PAS v2.8. For more information, see Disable Hostname Verification for External CredHub Databases on GCP and Azure in the Upgrade Preparation Checklist for Pivotal Platform v2.8 topic.

For more information about CredHub, see CredHub. For more information about configuring VMs, see the Configure Resources section of the Configuring PAS topic.

In the Custom Branding pane of PAS v2.8, you can customize the marketplace URL and secondary navigation links that appear in your Apps Manager deployment.

For more information, see the Configure Custom Branding and Apps Manager section of the Configuring PAS topic.

Known Issues

Duplicate Metrics Appear in the Firehose

PAS v2.8 introduces a System Metrics Agent that sends metrics to the Firehose. These metrics match existing BOSH system metrics, but they use an updated format. In PAS v2.8, metrics appear in both formats. In the Firehose, you see duplicate metrics entries in both the Loggregator format and the System Metrics Agent format.

The following table shows examples of the existing Loggregator metrics format and the new System Metrics Agent metrics format:

Loggregator Format System Metrics Agent Format
system.healthy system_healthy
system.mem.percent system_mem_percent
system.disk.system.percent system_disk_system_percent
system.disk.ephemeral.percent system_disk_ephemeral_percent
system.disk.persistent.percent system_disk_persistent_percent
system.cpu.user system_cpu_user

The new System Metrics Agent adds about 50 metric envelopes per VM each minute.

The existing BOSH system metrics forwarder emits 2 metric envelopes per VM each minute. Each envelope contains 13 metrics.

Disable the Smoke Test Errand If You Disable the Firehose

If you disable the V1 or V2 Firehose in PAS v2.8, you must also disable the smoke test errand.

If you do not disable the smoke test errand, the deploy fails with an error similar to the following:

[91m[1m[Fail] [0m[90mLoggregator: [0m[0mcf logs [0m[90mlinux [0m[91m[1m[It] can see app messages in the logs [0m  
          [37m/var/vcap/packages/smoke_tests/src/github.com/cloudfoundry/cf-smoke-tests/smoke/logging/loggregator_test.go:42[0m  
          [1m[91mRan 1 of 2 Specs in 56.171 seconds[0m  
          [1m[91mFAIL![0m -- [32m[1m0 Passed[0m | [91m[1m1 Failed[0m | [33m[1m0 Pending[0m | [36m[1m1 Skipped[0m  
          --- FAIL: TestSmokeTests (56.17s)  
          FAIL  
          Ginkgo ran 2 suites in 1m7.050120251s  
          Test Suite Failed  
Stderr     Error: failed to run job-process: exit status 1 (exit status 1)

To disable the smoke test errand:

  1. Navigate to the Errands pane in the PAS tile.

  2. For Smoke Test Errand, select Off.

For more information, see the Configure Errands section in the Configuring PAS topic.

NFS Volume Service File Append Errors

 A defect in the mapfs FUSE driver causes errors to occur in file append operations when PAS users access the ID mapping feature with NFS in PAS v2.8.0 through PAS v2.8.1. PAS users enable the ID mapping feature when they specify either the uid or username option in their service instance or service bind configuration.  When this issue occurs, file append operations within the mounted file system (e.g. echo hello >> test.txt) fail with the error File operation not supported.

This issue is resolved in PAS v2.8.2.