Pivotal Application Service v2.4 Release Notes
Pivotal Cloud Foundry is certified by the Cloud Foundry Foundation for 2019.
- [Feature Improvement] Improved logging and error handling for Diego Sync job
- [Feature Improvement] Add note to description for “cf deployment name” option in PAS Config to ensure Healthwatch is correctly configured
- [Bug Fix] Fix memory allocated to be multiplied by number of instances for each process on space page app tab
- [Bug Fix] Update MySQL driver so
allowNativePasswordsno longer defaults to
falseto fix incompatibility with Autoscaler and some hosted MySQL services
- [Bug Fix] VXLAN-policy-agent now opens ports in non-ephemeral port range
- [Bug Fix] Fix help text for TCP router ports because comma separated lists of ports are not supported
- [Bug Fix] Fix some docs links to correct product versions
- [Bug Fix] Ensure logs and metrics are forwarded by adding syslog_forwarder and loggregator_agent to VMs missing them
- Bump ubuntu-xenial stemcell to version
- Bump cf-autoscaling to version
- Bump cflinuxfs2 to version
- Bump cflinuxfs3 to version
- Bump push-apps-manager-release to version
- Bump silk to version
- Breaking Changes
[Breaking change] Increase default and minimum CPU core count from 1 to 2 for internal blobstore VMs to improve reliability
[Breaking Change] Improve GrootFS garbage collection
[Breaking Change] Bump log-cache API to version 2.x which moves
[Breaking Change] Internal MySQL always uses Percona XtraDB Cluster
[Beta Feature] Users can now discover organization level usage durations across a foundation
[Beta Feature] Networking administrators can configure egress networking policies dynamically to IP ranges instead of ASGs
[Feature] Add cloud controller deployment-updater job to enable zero-downtime pushes with
[Feature] Enable fetching logs and metrics via gRPC
[Feature] Add smb-volume-release to PAS
[Feature] Users in
usage_service.auditgroup can access the Usage Service API using their CF OAuth token
[Feature] Apps manager includes new home page and allows users to search for apps, spaces, orgs and service instances
[Feature] Add metric-registrar to enable App Developers to output custom application metrics that can be monitored by platform-provided tooling
[Feature] Remove configuration for internal MySQL Load Balancer
[Feature] Co-locate BPM on all VMs in PAS and IST
tile_installerUAA client intended for use by service tiles during installation to create their own long-term operational client credentials
[Feature] Support restore from different Azure storage account
[Feature Improvement] Operators can opt-in to enable TLS for clients of the internal PXC database
[Feature Improvement] Allow TLS between usage-service and mysql to be turned off
[Feature Improvement] Improve performance of the system_report/service_usages endpoint in the
usages-serviceto prevent potential 502 or 504 responses on larger deployments
[Feature Improvement] Update version number in link to docs page for Apps Manager
[Feature Improvement] Add new endpoint in CredHub to find permissions given actor and path
cf sshis now compatible with mutual-TLS-based verification of ingress traffic from gorouters to application instances
[Feature Improvement] Notifications component talks TLS to database
[Feature Improvement] NFS Broker connects to its database using TLS
[Feature Improvement] Credhub can be configured to use the same external database as other components configured in PAS tile
[Feature Improvement] UAA connects using TLS to internal database default & external databases when a CA cert is provided
[Feature Improvement] UAA can be configured to use the same external database as other components configured in PAS tile
[Feature Improvement] Credhub connects to MySQL using TLS
[Feature Improvement] The usage-service uses the shared CF CLI which can be updated independently
[Feature Improvement] Operators can configure the default root filesystem for new applications created the platform
[Feature Improvement] The notification release uses the shared CF CLI which can be updated independently
[Feature Improvement] The notification ui release uses the shared CF CLI which can be updated independently
[Feature Improvement] Loggregator can be updated without rolling every VM
[Feature Improvement] Change default HAProxy instance count to 0
[Feature Improvement] Routing-API job connects to MySQL using TLS
[Feature Improvement] All errands use the CF CLI that is distributed as a BOSH release
[Feature Improvement] Update name of the Small Footprint PAS tile shown on the tile in Ops Manager UI
[Feature Improvement] clock_global now defaults to 2 instances to be highly available
[Feature Improvement] Allow disabling connection pooling for autoscaler API & escape special characters in external database passwords
[Feature Improvement] Improve UI for route integrity options
[Feature Improvement] Credhub is GA and on by default
[Feature Improvement] Scale Consul Server down to 0
[Feature Improvement] Remove option to disable route integrity
[Feature Improvement] network policy server connects to MySQL using TLS
[Feature Improvement] Operators can distinguish among metrics being emitted from different PAS tile deployments
[Feature Improvement] credhub connects to MySQL using TLS
[Feature Improvement] Diego connects to MySQL using TLS
[Feature Improvement] CAPI connects to MySQL using TLS
[Feature Improvement] PAS can be configured with a CA certificate for TLS connections to an external MySQL database
[Feature Improvement] Metrics emitted from each deployment are tagged with additional metadata
Add new release loggregator-agent at version
Add new release metric-registrar at version
Add new release smb-volume at version
Removed cf-mysql release
Bump backup-and-restore-sdk to version
Bump binary-offline-buildpack to version
Bump bosh-dns-aliases to version
Bump bosh-system-metrics-forwarder to version 0.0.16
Bump bpm to version
Bump cf-cli to version
Bump cf-smoke-tests to version
Bump cflinuxfs2 to version
Bump cflinuxfs3 to version
Bump consul to version
Bump consul-drain to version
Bump capi to version
Bump credhub to version
Bump diego to version
Bump dotnet-core-offline-buildpack to version
Bump garden-runc to version
Bump go-offline-buildpack to version
Bump haproxy to version 9.3.0
Bump java-offline-buildpack to version
Bump log-cache to version
Bump loggregator to version
Bump mysql-monitoring to version
Bump nats to version 26
Bump nodejs-offline-buildpack to version
Bump php-offline-buildpack to version
Bump pxc to version 0.14.0
Bump python-offline-buildpack to version
Bump routing to version 0.182.0
Bump ruby-offline-buildpack to version
Bump silk to version
Bump staticfile-offline-buildpack to version
Bump syslog to version
Bump ubuntu-xenial stemcell to version
Bump cf-autoscaling to version
Bump cf-networking to version
Bump cf-syslog-drain to version
Bump mapfs to version
Bump nfs-volume to version
Bump notifications-ui to version
Bump notifications to version
Bump push-apps-manager-release to version
Bump push-usage-service-release to version
Bump statsd-injector to version
Bump uaa to version
The procedure for upgrading to Pivotal Application Service (PAS) v2.4 is documented in the Upgrading Pivotal Cloud Foundry topic.
When upgrading to PAS v2.4, be aware of the following upgrade considerations:
If you previously used an earlier version of PAS, you must first upgrade to PAS v2.3 to successfully upgrade to PAS v2.4.
You must enable route integity before upgrading to PAS v2.4. To enable route integrity, go to Application Containers in PAS and select Router uses TLS to verify application identity. This option does use approximately 32MB more memory per app.
If you are running internal MySQL databases on MariaDB, you must migrate them to Percona server before upgrading to PAS v2.4. For more information, see Migrating to Internal Percona MySQL.
Some partner service tiles may be incompatible with PCF v2.4. Pivotal is working with partners to ensure their tiles are updated to work with the latest versions of PCF.
For information about which partner service releases are currently compatible with PCF v2.4, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.
Runtime CredHub is enabled by default in the PAS tile. For more information about where your credentials are stored, see Runtime CredHub.
CredHub is generally available and includes bug fixes, security updates, and improvements to the
find command in the CredHub CLI.
You can scale the number of CredHub instances in the Resource Config pane. In PAS v2.4, the number of CredHub instances defaults to
For upgrades to v2.4, PAS populates the CredHub row in the Resource Config pane with the value set in your previous version of the tile.
In the PAS v2.4 tile, you can configure runtime CredHub to use the same external database as other PAS components. If you select External Databases - (e.g. AWS RDS) on the Databases pane of the PAS tile, select PAS database on the CredHub pane for CredHub to use the same external database.
To use a separate external database for runtime CredHub, select Other external database on the CredHub pane of the PAS tile. If you use GCP, you cannot use an external database for CredHub. For more information, see CredHub Database Cannot Be External on GCP below.
For more information, see the PAS topic that corresponds to your IaaS:
- Deploying PAS on AWS
- Deploying PAS on AWS Using Terraform
- Deploying PAS on Azure
- Deploying PAS on Azure Using Terraform
- Deploying PAS on GCP
- Deploying PAS on GCP Using Terraform
- Deploying PAS on OpenStack
- Deploying PAS on vSphere
PAS v2.4 uses the
cflinuxfs3 stack and related buildpacks by default on new installs. Buildpacks have the same name but use a different stack.
You can switch between
cflinuxfs3 using the cf CLI (Cloud Foundry Command Line Interface). For more information, see Changing Stacks.
You can also set your default stack in the Cloud Controller pane of the PAS tile. For more information, see Cloud Controller.
cflinuxfs3 stack was introduced in PAS v2.3. For more information, see cflinuxfs3 Stack and Compatible Buildpacks in the PAS v2.3 release notes.
Consul Server VMs are now removed from PAS, saving VM resources and reducing maintenance around managing a clustered component. Consul functionality in older PCF versions has been replaced by BOSH DNS.
Platform components such as CAPI, Diego, and CredHub use TLS to communicate with the database. This happens automatically when using the internal database. For external databases such as RDS, you can provide the CA certificate in the Databases pane of the PAS tile.
In PAS v2.2 and v2.3, internal MySQL databases could run on either MariaDB or Percona servers. PAS v2.4 no longer uses MariaDB, so you must migrate internal MariaDB databases to Percona and redeploy PAS v2.3 before upgrading to PAS v2.4.
To migrate your databases to Percona, see Migrating to Internal Percona MySQL.
nfs-experimental service is promoted from “experimental” to “production”. The original fuse-based
nfs service and existing
nfs service bindings are now called
To switch over to the new
nfs service, you must re-create and re-bind your existing service bindings to the
For all PAS v2.4 foundations, the Gorouter always uses TLS to verify app identity. Verifying app identity improves resiliency and consistency for app routes, as well as increases security by encrypting data in flight from the Gorouter.
In PAS v2.3 and earlier, you can have insecure routing without TLS. Before you upgrade to v2.4, you must have secure routing with TLS enabled. For more information, see Upgrade Preparation Checklist for PCF v2.4.
For more information about the verifying app identity feature, see Increased Resiliency, Consistency, and Security for HTTP Routing in the PAS v2.1 release notes.
As the value for
deployment, metrics use
cf-GUID, which corresponds to the BOSH deployment name of your PAS tile. With a GUID, you can uniquely identify your metrics by tile. In PAS v2.3 and earlier, metrics have a
deployment value of
This feature is enabled by default in new deployments of PAS v2.4, but disabled by default for PAS deployments upgrading to PAS v2.4.
Breaking Change: If you have scripts that rely on
cf as the value for
deployment, your scripts may break. For more information, see Changed Deployment Value for PAS Metrics.
PAS tags metrics with additional metadata to help operators better parse the metrics coming from their different deployments. These metadata tags also enable downstream monitoring products, such as PCF Healthwatch, to easily display human-readable names.
The tags are as follows:
product: The value of this tag is always
Pivotal Application Servicefor the PAS tile. The tags for other products are
PCF Isolation Segment,
PCF Small Footprint,
Pivotal Application Service for Windows 2012R2, and
Pivotal Application Service for Windows.
system_domain: The value of this tag corresponds to what you set in the System Domain field in the Domains tab of the PAS tile.
placement_tag: The value of this tag is always
nullfor PAS. However, for PAS for Windows and PCF Isolation Segment tiles, you can configure this value using the Segment Name field in the Application Containers pane.
Previously, you could not easily know the deployment that a metric from an Isolation Segment was emitted from. Now, an operator can display capacity and other relevant metrics using the
placement_tag name. This makes it easier to reason the importance of a given segment when issues arise.
These tags are properties of the metron agent running on each VM in a deployment.
Apps Manager has a global searchbar on every Apps Manager page. With the global searchbar, you can search for an org name, space name, service instance name, and app name without leaving the page.
Search results populate with each keypress and display below the searchbar. When the global searchbar is selected but empty, recently-accessed apps, services, spaces, and orgs display below the searchbar.
This feature does not have an equivalent in the cf CLI.
To improve navigation, Apps Manager v2.4 includes the following UI changes:
- New home page shows the following:
- Up to ten of the most-recently accessed apps with status, name, org and space, instances, and time since last push
- All orgs in your PCF deployment
- Simplified sidebar navigation with Home and Marketplace only
- Breadcrumbs help you navigate to previous screens
Apps Manager includes a restage button. The button is in addition to the stop and start buttons on the app management page. Restaging your app stops your app and restages it by compiling a new droplet and starting it.
Restage your app if you changed the environment in a way that affects your staging process. For example, you might set an environment variable that the buildpack consumes.
For more information about the restage button in the Apps Manager UI, see Start, Stop, or Restage an App.
When service instance sharing is enabled by an operator, space developers can share a service instance from its Overview tab in Apps Manager, making it available for use in multiple spaces that they have access to.
For more information, see Service Instance Sharing.
As a nozzle developer, you can access the Loggregator v2 API through a Reverse Log Proxy (RLP) gateway. The RLP gateway provides an HTTP API to access the RLP. With the RLP gateway, you do not need to manage mutual TLS to access the Loggregator v2 API.
By default, the RLP communicates with clients using gRPC over mutual TLS. To enable HTTP access instead, use the RLP Gateway. For more information about the RLP Gateway, see Reverse Log Proxy (RLP) Gateway in the Loggregator GitHub repository.
For more information about the Loggregator API, see loggregator-api in GitHub.
PAS v2.4 includes a new component: the Metric Registrar. The Metric Registrar allows app developers to export custom app metrics and events in a format that Loggregator can consume. App developers can then use the custom metrics to monitor apps with PCF Metrics and configure autoscaling rules with PCF Autoscaler.
For more information, see the following topics:
PAS v2.4 includes a beta feature that allows you to create dynamic egress policies so your apps can communicate with external services. These policies are similar to Application Security Groups (ASGs) but include the following advantages:
- You do not have to restart your apps when applying these policies, so there is no downtime.
- The policies include an additional level of granularity: you can apply them to specific apps.
For more information, see Administering Dynamic Egress Policies (Beta).
PAS v2.4 supports SMB volume services, allowing developers to bind existing SMB shares to their apps. The SMB protocol has native password authentication, which means you can control access to file shares without the overhead of configuring an LDAP server.
This feature is disabled by default. To enable this feature, see Enable SMB Volume Services.
For information about how to bind a volume service to an app, see Using an External File System (Volume Services).
Note: The SMB volume service is available for Linux cells only. This service is not available for Windows cells.
PAS v2.4 supports enabling TLS for clients of the internal system database. This feature is in beta and disabled by default. To enable this feature, see the Advanced Features section of the PAS deployment topic for your IaaS.
You can configure PAS v2.4 to use TLS for all components’ connections to an external MySQL database by providing a Certificate Authority (CA) certificate. For more information, see the External Database Configuration section of the PAS configuration topic for your IaaS.
PAS v2.4 includes support for native zero downtime app deployments using experimental cf CLI commands. For more information about using this feature, see the Deploying Apps with Zero Downtime (Beta) topic.
This feature is enabled by default. You can optionally disable it in the Advanced Features pane.
cf ssh into apps when mutual TLS (mTLS) is enabled for app identity verification.
Operators enable mTLS by selecting the Router and applications use mutual TLS to verify each other’s identity option in the Application Containers pane of PAS.
This removes the
cf ssh known issue described in the Limitations with Mutual TLS App Identity Verification section of the PAS v2.3 Release Notes.
PAS v2.4 does not deploy the optional HAProxy component by default. Previous versions of PAS deployed three instances of HAProxy by default.
Breaking Change: This is a breaking change if you are using HAProxy with the Automatic setting for your instance count. On upgrading to PAS v2.4, you will have zero HAProxy instances unless you specify an instance count other than the Automatic setting. For example, if you have the instance count set to Automatic: 3, modify the instance count to 3 in the Resource Config pane before you upgrade.
Note: PAS still deploys one HAProxy instance for vSphere environments by default.
Operators configure container filesystem garbage collection based on the disk usage of other jobs rather than a garbage collection threshold.
PAS v2.4 replaces the Clean up disk-space once threshold is reached option with Clean up disk-space once usage fills disk in the Application Containers pane.
For information about configuring garbage collection values, see Configuring Cell Disk Cleanup Scheduling.
Starting in v2.3, some PAS components used BOSH Process Manager (BPM). In v2.4, all components use BPM. For more information, see BOSH Process Manager in the Pivotal Application Service v2.3 Release Notes.
If your PAS deployment is on GCP and you want to use Runtime CredHub, you must select PAS database for your CredHub database and Internal for your system database. If you are using external system databases, you cannot use CredHub.
CredHub is not compatible with the external database option on GCP. GCP Cloud SQL presents its certificate in a way that CredHub cannot connect to.
This section describes an issue and workaround related to configuring multiple TCP Routing Ports in the PAS tile UI.
You cannot enter a comma-separated list of ports in the TCP Routing Ports field of the PAS tile. If you enter a comma-separated list, the Routing API does not start. The TCP Routing Ports field allows entries in the following formats:
- A single value, such as
- A range of values, such as
If you want to configure multiple ports, do the following:
Note: This procedure causes brief downtime for TCP apps listening on ports that you open after deploying PAS.
Configure PAS with Enable TCP Routing selected.
Enter one port you want to use in the TCP Routing Ports field.
Use the Routing API to add all desired TCP ports by following the instructions in the Modify your TCP ports section of the Enabling TCP Routing topic. When using the Routing API, you can include a comma separated list of ports.
Operators intermittently receive errors when pushing apps during a rolling upgrade.
Above approximately 40 Doppler instances and 20 Traffic Controller instances, horizontal scaling is no longer useful for improving Loggregator Firehose performance. To improve performance, increase CPU resources for the existing Doppler and Traffic Controller instances to add vertical scale.