Pivotal Application Service v2.4 Release Notes

Pivotal Cloud Foundry is certified by the Cloud Foundry Foundation for 2019.

Read more about the certified provider program and the requirements of providers.


Releases

2.4.1

  • [Feature Improvement] Improved logging and error handling for Diego Sync job
  • [Feature Improvement] Add note to description for “cf deployment name” option in PAS Config to ensure Healthwatch is correctly configured
  • [Bug Fix] Fix memory allocated to be multiplied by number of instances for each process on space page app tab
  • [Bug Fix] Update MySQL driver so allowNativePasswords no longer defaults to false to fix incompatibility with Autoscaler and some hosted MySQL services
  • [Bug Fix] VXLAN-policy-agent now opens ports in non-ephemeral port range
  • [Bug Fix] Fix help text for TCP router ports because comma separated lists of ports are not supported
  • [Bug Fix] Fix some docs links to correct product versions
  • [Bug Fix] Ensure logs and metrics are forwarded by adding syslog_forwarder and loggregator_agent to VMs missing them
  • Bump ubuntu-xenial stemcell to version 170.14
  • Bump cf-autoscaling to version 216
  • Bump cflinuxfs2 to version 1.255.0
  • Bump cflinuxfs3 to version 0.46.0
  • Bump push-apps-manager-release to version 667.0.3
  • Bump silk to version 2.18.1
Component Version
ubuntu-xenial stemcell170.14
backup-and-restore-sdk1.10.0
binary-offline-buildpack1.0.27
bosh-dns-aliases0.0.3
bosh-system-metrics-forwarder0.0.16
bpm0.13.0
capi1.71.4
cf-autoscaling216
cf-backup-and-restore0.0.11
cf-cli1.9.0
cf-networking2.18.2
cf-smoke-tests40.0.40
cf-syslog-drain8.0
cflinuxfs21.255.0
cflinuxfs30.46.0
consul-drain0.0.3
consul198
credhub2.1.2
diego2.22.0
dotnet-core-offline-buildpack2.2.0
garden-runc1.16.7
go-offline-buildpack1.8.29
haproxy9.3.0
java-offline-buildpack4.16.1
log-cache2.0.1
loggregator-agent2.2
loggregator103.3
mapfs1.1.2
metric-registrar1.0.4
mysql-monitoring9.1.0
nats26
nfs-volume1.7.4
nodejs-offline-buildpack1.6.34
notifications-ui36
notifications54
php-offline-buildpack4.3.64
push-apps-manager-release667.0.3
push-usage-service-release668.0.10
pxc0.14.0
python-offline-buildpack1.6.23
routing0.182.0
ruby-offline-buildpack1.7.27
silk2.18.1
smb-volume0.2.2
staticfile-offline-buildpack1.4.35
statsd-injector1.5.0
syslog11.4.0
uaa66.0

2.4.0

See also:

  • Breaking Changes
  • New Features

  • [Breaking change] Increase default and minimum CPU core count from 1 to 2 for internal blobstore VMs to improve reliability

  • [Breaking Change] Improve GrootFS garbage collection

  • [Breaking Change] Bump log-cache API to version 2.x which moves /v1 endpoints to /api/v1

  • [Breaking Change] Internal MySQL always uses Percona XtraDB Cluster

  • [Beta Feature] Users can now discover organization level usage durations across a foundation

  • [Beta Feature] Networking administrators can configure egress networking policies dynamically to IP ranges instead of ASGs

  • [Feature] Add cloud controller deployment-updater job to enable zero-downtime pushes with cf v3-zdt-push

  • [Feature] Enable fetching logs and metrics via gRPC

  • [Feature] Add smb-volume-release to PAS

  • [Feature] Users in usage_service.audit group can access the Usage Service API using their CF OAuth token

  • [Feature] Apps manager includes new home page and allows users to search for apps, spaces, orgs and service instances

  • [Feature] Add metric-registrar to enable App Developers to output custom application metrics that can be monitored by platform-provided tooling

  • [Feature] Remove configuration for internal MySQL Load Balancer

  • [Feature] Co-locate BPM on all VMs in PAS and IST

  • [Feature] Add tile_installer UAA client intended for use by service tiles during installation to create their own long-term operational client credentials

  • [Feature] Support restore from different Azure storage account

  • [Feature Improvement] Operators can opt-in to enable TLS for clients of the internal PXC database

  • [Feature Improvement] Allow TLS between usage-service and mysql to be turned off

  • [Feature Improvement] Improve performance of the system_report/service_usages endpoint in the usages-service to prevent potential 502 or 504 responses on larger deployments

  • [Feature Improvement] Update version number in link to docs page for Apps Manager

  • [Feature Improvement] Add new endpoint in CredHub to find permissions given actor and path

  • [Feature Improvement] cf ssh is now compatible with mutual-TLS-based verification of ingress traffic from gorouters to application instances

  • [Feature Improvement] Notifications component talks TLS to database

  • [Feature Improvement] NFS Broker connects to its database using TLS

  • [Feature Improvement] Credhub can be configured to use the same external database as other components configured in PAS tile

  • [Feature Improvement] UAA connects using TLS to internal database default & external databases when a CA cert is provided

  • [Feature Improvement] UAA can be configured to use the same external database as other components configured in PAS tile

  • [Feature Improvement] Credhub connects to MySQL using TLS

  • [Feature Improvement] The usage-service uses the shared CF CLI which can be updated independently

  • [Feature Improvement] Operators can configure the default root filesystem for new applications created the platform

  • [Feature Improvement] The notification release uses the shared CF CLI which can be updated independently

  • [Feature Improvement] The notification ui release uses the shared CF CLI which can be updated independently

  • [Feature Improvement] Loggregator can be updated without rolling every VM

  • [Feature Improvement] Change default HAProxy instance count to 0

  • [Feature Improvement] Routing-API job connects to MySQL using TLS

  • [Feature Improvement] All errands use the CF CLI that is distributed as a BOSH release

  • [Feature Improvement] Update name of the Small Footprint PAS tile shown on the tile in Ops Manager UI

  • [Feature Improvement] clock_global now defaults to 2 instances to be highly available

  • [Feature Improvement] Allow disabling connection pooling for autoscaler API & escape special characters in external database passwords

  • [Feature Improvement] Improve UI for route integrity options

  • [Feature Improvement] Credhub is GA and on by default

  • [Feature Improvement] Scale Consul Server down to 0

  • [Feature Improvement] Remove option to disable route integrity

  • [Feature Improvement] network policy server connects to MySQL using TLS

  • [Feature Improvement] Operators can distinguish among metrics being emitted from different PAS tile deployments

  • [Feature Improvement] credhub connects to MySQL using TLS

  • [Feature Improvement] Diego connects to MySQL using TLS

  • [Feature Improvement] CAPI connects to MySQL using TLS

  • [Feature Improvement] PAS can be configured with a CA certificate for TLS connections to an external MySQL database

  • [Feature Improvement] Metrics emitted from each deployment are tagged with additional metadata

  • Add new release loggregator-agent at version 2.2

  • Add new release metric-registrar at version 1.0.4

  • Add new release smb-volume at version 0.2.2

  • Removed cf-mysql release

  • Bump backup-and-restore-sdk to version 1.10.0

  • Bump binary-offline-buildpack to version 1.0.27

  • Bump bosh-dns-aliases to version 0.0.3

  • Bump bosh-system-metrics-forwarder to version 0.0.16

  • Bump bpm to version 0.13.0

  • Bump cf-cli to version 1.9.0

  • Bump cf-smoke-tests to version 40.0.40

  • Bump cflinuxfs2 to version 1.253.0

  • Bump cflinuxfs3 to version 0.44.0

  • Bump consul to version 198

  • Bump consul-drain to version 0.0.3

  • Bump capi to version 1.72.2

  • Bump credhub to version 2.1.2

  • Bump diego to version 2.22.0

  • Bump dotnet-core-offline-buildpack to version 2.2.0

  • Bump garden-runc to version 1.16.7

  • Bump go-offline-buildpack to version 1.8.29

  • Bump haproxy to version 9.3.0

  • Bump java-offline-buildpack to version 4.16.1

  • Bump log-cache to version 2.0.1

  • Bump loggregator to version 103.3

  • Bump mysql-monitoring to version 9.1.0

  • Bump nats to version 26

  • Bump nodejs-offline-buildpack to version 1.6.34

  • Bump php-offline-buildpack to version 4.3.64

  • Bump pxc to version 0.14.0

  • Bump python-offline-buildpack to version 1.6.23

  • Bump routing to version 0.182.0

  • Bump ruby-offline-buildpack to version 1.7.27

  • Bump silk to version 2.17.0

  • Bump staticfile-offline-buildpack to version 1.4.35

  • Bump syslog to version 11.4.0

  • Bump ubuntu-xenial stemcell to version 170.9

  • Bump cf-autoscaling to version 215

  • Bump cf-networking to version 2.18.2

  • Bump cf-syslog-drain to version 8.0

  • Bump mapfs to version 1.1.2

  • Bump nfs-volume to version 1.7.4

  • Bump notifications-ui to version 36

  • Bump notifications to version 54

  • Bump push-apps-manager-release to version 667.0.2

  • Bump push-usage-service-release to version 668.0.10

  • Bump statsd-injector to version 1.5.0

  • Bump uaa to version 66.0

Component Version
ubuntu-xenial stemcell170.9
backup-and-restore-sdk1.10.0
binary-offline-buildpack1.0.27
bosh-dns-aliases0.0.3
bosh-system-metrics-forwarder0.0.16
bpm0.13.0
capi1.72.2
cf-autoscaling215
cf-backup-and-restore0.0.11
cf-cli1.9.0
cf-networking2.18.2
cf-smoke-tests40.0.40
cf-syslog-drain8.0
cflinuxfs21.253.0
cflinuxfs30.44.0
consul-drain0.0.3
consul198
credhub2.1.2
diego2.22.0
dotnet-core-offline-buildpack2.2.0
garden-runc1.16.7
go-offline-buildpack1.8.29
haproxy9.3.0
java-offline-buildpack4.16.1
log-cache2.0.1
loggregator-agent2.2
loggregator103.3
mapfs1.1.2
metric-registrar1.0.4
mysql-monitoring9.1.0
nats26
nfs-volume1.7.4
nodejs-offline-buildpack1.6.34
notifications-ui36
notifications54
php-offline-buildpack4.3.64
push-apps-manager-release667.0.2
push-usage-service-release668.0.10
pxc0.14.0
python-offline-buildpack1.6.23
routing0.182.0
ruby-offline-buildpack1.7.27
silk2.17.0
smb-volume0.2.2
staticfile-offline-buildpack1.4.35
statsd-injector1.5.0
syslog11.4.0
uaa66.0

How to Upgrade

The procedure for upgrading to Pivotal Application Service (PAS) v2.4 is documented in the Upgrading Pivotal Cloud Foundry topic.

When upgrading to PAS v2.4, be aware of the following upgrade considerations:

  • If you previously used an earlier version of PAS, you must first upgrade to PAS v2.3 to successfully upgrade to PAS v2.4.

  • You must enable route integity before upgrading to PAS v2.4. To enable route integrity, go to Application Containers in PAS and select Router uses TLS to verify application identity. This option does use approximately 32MB more memory per app.

  • If you are running internal MySQL databases on MariaDB, you must migrate them to Percona server before upgrading to PAS v2.4. For more information, see Migrating to Internal Percona MySQL.

  • Some partner service tiles may be incompatible with PCF v2.4. Pivotal is working with partners to ensure their tiles are updated to work with the latest versions of PCF.

    For information about which partner service releases are currently compatible with PCF v2.4, review the appropriate partners services release documentation at https://docs.pivotal.io, or contact the partner organization that produces the tile.

New Features in PAS v2.4

Runtime CredHub is GA and Enabled by Default

Runtime CredHub is enabled by default in the PAS tile. For more information about where your credentials are stored, see Runtime CredHub.

CredHub is generally available and includes bug fixes, security updates, and improvements to the find command in the CredHub CLI.

You can scale the number of CredHub instances in the Resource Config pane. In PAS v2.4, the number of CredHub instances defaults to 2.

For upgrades to v2.4, PAS populates the CredHub row in the Resource Config pane with the value set in your previous version of the tile.

CredHub Can Use the Same External Database as Other PAS Components

In the PAS v2.4 tile, you can configure runtime CredHub to use the same external database as other PAS components. If you select External Databases - (e.g. AWS RDS) on the Databases pane of the PAS tile, select PAS database on the CredHub pane for CredHub to use the same external database.

To use a separate external database for runtime CredHub, select Other external database on the CredHub pane of the PAS tile. If you use GCP, you cannot use an external database for CredHub. For more information, see CredHub Database Cannot Be External on GCP below.

For more information, see the PAS topic that corresponds to your IaaS:

PAS Uses cflinuxfs3 by Default

PAS v2.4 uses the cflinuxfs3 stack and related buildpacks by default on new installs. Buildpacks have the same name but use a different stack.

You can switch between cflinuxfs2 and cflinuxfs3 using the cf CLI (Cloud Foundry Command Line Interface). For more information, see Changing Stacks.

You can also set your default stack in the Cloud Controller pane of the PAS tile. For more information, see Cloud Controller.

The cflinuxfs3 stack was introduced in PAS v2.3. For more information, see cflinuxfs3 Stack and Compatible Buildpacks in the PAS v2.3 release notes.

Consul Server VMs Are Removed from PAS

Consul Server VMs are now removed from PAS, saving VM resources and reducing maintenance around managing a clustered component. Consul functionality in older PCF versions has been replaced by BOSH DNS.

Components Communicate to Database Using TLS

Platform components such as CAPI, Diego, and CredHub use TLS to communicate with the database. This happens automatically when using the internal database. For external databases such as RDS, you can provide the CA certificate in the Databases pane of the PAS tile.

Internal MySQL Databases Run Only on Percona

In PAS v2.2 and v2.3, internal MySQL databases could run on either MariaDB or Percona servers. PAS v2.4 no longer uses MariaDB, so you must migrate internal MariaDB databases to Percona and redeploy PAS v2.3 before upgrading to PAS v2.4.

To migrate your databases to Percona, see Migrating to Internal Percona MySQL.

NFS-Experimental Service Graduation

The existing nfs-experimental service is promoted from “experimental” to “production”. The original fuse-based nfs service and existing nfs service bindings are now called nfs-legacy.

To switch over to the new nfs service, you must re-create and re-bind your existing service bindings to the nfs service.

Gorouter Always Verifies App Identity

For all PAS v2.4 foundations, the Gorouter always uses TLS to verify app identity. Verifying app identity improves resiliency and consistency for app routes, as well as increases security by encrypting data in flight from the Gorouter.

In PAS v2.3 and earlier, you can have insecure routing without TLS. Before you upgrade to v2.4, you must have secure routing with TLS enabled. For more information, see Upgrade Preparation Checklist for PCF v2.4.

For more information about the verifying app identity feature, see Increased Resiliency, Consistency, and Security for HTTP Routing in the PAS v2.1 release notes.

Uniquely Identify Metrics by Tile

As the value for deployment, metrics use cf-GUID, which corresponds to the BOSH deployment name of your PAS tile. With a GUID, you can uniquely identify your metrics by tile. In PAS v2.3 and earlier, metrics have a deployment value of cf.

This feature is enabled by default in new deployments of PAS v2.4, but disabled by default for PAS deployments upgrading to PAS v2.4.

Breaking Change: If you have scripts that rely on cf as the value for deployment, your scripts may break. For more information, see Changed Deployment Value for PAS Metrics.

See Human-Friendly Metadata for Metrics

PAS tags metrics with additional metadata to help operators better parse the metrics coming from their different deployments. These metadata tags also enable downstream monitoring products, such as PCF Healthwatch, to easily display human-readable names.

The tags are as follows:

  • product: The value of this tag is always Pivotal Application Service for the PAS tile. The tags for other products are PCF Isolation Segment, PCF Small Footprint, Pivotal Application Service for Windows 2012R2, and Pivotal Application Service for Windows.

  • system_domain: The value of this tag corresponds to what you set in the System Domain field in the Domains tab of the PAS tile.

  • placement_tag: The value of this tag is always null for PAS. However, for PAS for Windows and PCF Isolation Segment tiles, you can configure this value using the Segment Name field in the Application Containers pane.

Previously, you could not easily know the deployment that a metric from an Isolation Segment was emitted from. Now, an operator can display capacity and other relevant metrics using the placement_tag name. This makes it easier to reason the importance of a given segment when issues arise.

These tags are properties of the metron agent running on each VM in a deployment.

Global Search in Apps Manager

Apps Manager has a global searchbar on every Apps Manager page. With the global searchbar, you can search for an org name, space name, service instance name, and app name without leaving the page.

Search results populate with each keypress and display below the searchbar. When the global searchbar is selected but empty, recently-accessed apps, services, spaces, and orgs display below the searchbar.

This feature does not have an equivalent in the cf CLI.

Navigation Updates in Apps Manager

To improve navigation, Apps Manager v2.4 includes the following UI changes:

  • New home page shows the following:
    • Up to ten of the most-recently accessed apps with status, name, org and space, instances, and time since last push
    • All orgs in your PCF deployment
  • Simplified sidebar navigation with Home and Marketplace only
  • Breadcrumbs help you navigate to previous screens

Restage Apps in Apps Manager

Apps Manager includes a restage button. The button is in addition to the stop and start buttons on the app management page. Restaging your app stops your app and restages it by compiling a new droplet and starting it.

Restage your app if you changed the environment in a way that affects your staging process. For example, you might set an environment variable that the buildpack consumes.

For more information about the restage button in the Apps Manager UI, see Start, Stop, or Restage an App.

Share Service Instances in Apps Manager

When service instance sharing is enabled by an operator, space developers can share a service instance from its Overview tab in Apps Manager, making it available for use in multiple spaces that they have access to.

For more information, see Service Instance Sharing.

Loggregator v2 API Is Readable through RLP Gateway

As a nozzle developer, you can access the Loggregator v2 API through a Reverse Log Proxy (RLP) gateway. The RLP gateway provides an HTTP API to access the RLP. With the RLP gateway, you do not need to manage mutual TLS to access the Loggregator v2 API.

By default, the RLP communicates with clients using gRPC over mutual TLS. To enable HTTP access instead, use the RLP Gateway. For more information about the RLP Gateway, see Reverse Log Proxy (RLP) Gateway in the Loggregator GitHub repository.

For more information about the Loggregator API, see loggregator-api in GitHub.

Emit Custom App Metrics to the Metric Registrar

PAS v2.4 includes a new component: the Metric Registrar. The Metric Registrar allows app developers to export custom app metrics and events in a format that Loggregator can consume. App developers can then use the custom metrics to monitor apps with PCF Metrics and configure autoscaling rules with PCF Autoscaler.

For more information, see the following topics:

Create Dynamic Egress Policies (Beta)

PAS v2.4 includes a beta feature that allows you to create dynamic egress policies so your apps can communicate with external services. These policies are similar to Application Security Groups (ASGs) but include the following advantages:

  • You do not have to restart your apps when applying these policies, so there is no downtime.
  • The policies include an additional level of granularity: you can apply them to specific apps.

For more information, see Administering Dynamic Egress Policies (Beta).

SMB Volume Services

PAS v2.4 supports SMB volume services, allowing developers to bind existing SMB shares to their apps. The SMB protocol has native password authentication, which means you can control access to file shares without the overhead of configuring an LDAP server.

This feature is disabled by default. To enable this feature, see Enable SMB Volume Services.

For information about how to bind a volume service to an app, see Using an External File System (Volume Services).

Note: The SMB volume service is available for Linux cells only. This service is not available for Windows cells.

TLS for Internal System Database (Beta)

PAS v2.4 supports enabling TLS for clients of the internal system database. This feature is in beta and disabled by default. To enable this feature, see the Advanced Features section of the PAS deployment topic for your IaaS.

TLS for External Databases

You can configure PAS v2.4 to use TLS for all components’ connections to an external MySQL database by providing a Certificate Authority (CA) certificate. For more information, see the External Database Configuration section of the PAS configuration topic for your IaaS.

Zero Downtime App Deployments (Beta)

PAS v2.4 includes support for native zero downtime app deployments using experimental cf CLI commands. For more information about using this feature, see the Deploying Apps with Zero Downtime (Beta) topic.

This feature is enabled by default. You can optionally disable it in the Advanced Features pane.

Support for cf ssh with Mutual TLS App Identity Verification

Developers can cf ssh into apps when mutual TLS (mTLS) is enabled for app identity verification.

Operators enable mTLS by selecting the Router and applications use mutual TLS to verify each other’s identity option in the Application Containers pane of PAS.

This removes the cf ssh known issue described in the Limitations with Mutual TLS App Identity Verification section of the PAS v2.3 Release Notes.

HAProxy Defaults to Zero Instances

PAS v2.4 does not deploy the optional HAProxy component by default. Previous versions of PAS deployed three instances of HAProxy by default.

Breaking Change: This is a breaking change if you are using HAProxy with the Automatic setting for your instance count. On upgrading to PAS v2.4, you will have zero HAProxy instances unless you specify an instance count other than the Automatic setting. For example, if you have the instance count set to Automatic: 3, modify the instance count to 3 in the Resource Config pane before you upgrade.

Note: PAS still deploys one HAProxy instance for vSphere environments by default.

Improved Garbage Collection

Operators configure container filesystem garbage collection based on the disk usage of other jobs rather than a garbage collection threshold.

PAS v2.4 replaces the Clean up disk-space once threshold is reached option with Clean up disk-space once usage fills disk in the Application Containers pane.

For information about configuring garbage collection values, see Configuring Cell Disk Cleanup Scheduling.

All Components Use BOSH Process Manager

Starting in v2.3, some PAS components used BOSH Process Manager (BPM). In v2.4, all components use BPM. For more information, see BOSH Process Manager in the Pivotal Application Service v2.3 Release Notes.

Known Issues

CredHub Database Cannot Be External on GCP

If your PAS deployment is on GCP and you want to use Runtime CredHub, you must select PAS database for your CredHub database and Internal for your system database. If you are using external system databases, you cannot use CredHub.

CredHub is not compatible with the external database option on GCP. GCP Cloud SQL presents its certificate in a way that CredHub cannot connect to.

Configuring Multiple TCP Routing Ports

This section describes an issue and workaround related to configuring multiple TCP Routing Ports in the PAS tile UI.

Issue

You cannot enter a comma-separated list of ports in the TCP Routing Ports field of the PAS tile. If you enter a comma-separated list, the Routing API does not start. The TCP Routing Ports field allows entries in the following formats:

  • A single value, such as 1234
  • A range of values, such as 1234-5678

Workaround

If you want to configure multiple ports, do the following:

Note: This procedure causes brief downtime for TCP apps listening on ports that you open after deploying PAS.

  1. Configure PAS with Enable TCP Routing selected.

  2. Enter one port you want to use in the TCP Routing Ports field.

  3. Deploy PAS.

  4. Use the Routing API to add all desired TCP ports by following the instructions in the Modify your TCP ports section of the Enabling TCP Routing topic. When using the Routing API, you can include a comma separated list of ports.

API Downtime During Rolling Deploys

Operators intermittently receive errors when pushing apps during a rolling upgrade.

Loggregator Component Horizontal Scaling Thresholds

Above approximately 40 Doppler instances and 20 Traffic Controller instances, horizontal scaling is no longer useful for improving Loggregator Firehose performance. To improve performance, increase CPU resources for the existing Doppler and Traffic Controller instances to add vertical scale.

Create a pull request or raise an issue on the source for this page in GitHub