PCF Isolation Segment v1.12 Release Notes

Page last updated:

Warning: Pivotal Cloud Foundry (PCF) v1.12 is no longer supported because it has reached the End of General Support (EOGS) phase. To stay up to date with the latest software and security updates, upgrade to a supported version.

Releases

1.12.24

• [Bug fix] Prevent downtime when upgrading from 1.12 to 2.0 when deployment includes HAProxy

• Bump cflinuxfs2 to version 1.228.0

• Bump stemcell to version 3468.55

1.12.23

• [Bug Fix] Docker image based app resource reporting correctly includes image size in disk usage

• Bump cflinuxfs2 to version 1.227.0

• Bump diego to version 1.25.15

• Bump loggregator to version 96.5

• Bump stemcell to version 3468.54

1.12.22

• [Bug Fix] bump consul to v195

  • Includes golang 1.9.7, removes golang 1.8.*.
  • Deploying v193 could fail on some deployments due to a conflict with other tiles that compiled the release differently
  • Fixes intermittent consul DNS issues on Windows Cells

• Bump cflinuxfs2 to version 1.223.0

• Bump consul to version 195

• Bump stemcell to version 3468.51

1.12.21

• [Security Fix] Bump diego to version 1.25.14
  • CVE-2018-1265
• [Bug fix] bump nfs-volume-release to version 1.2.1
  • Fix incompatibility with new garden-runc release when using read-only NFS volume mounts
• [Bug Fix] Bump garden to version 1.13.3
  • Fix issue with deleted files in application containers created from docker images
• [Bug Fix] Fix issue upgrading IST 1.11 to IST 1.12 when user had previously selected the option to “Forward SSL to Isolation Segment Router with ERT Router certificates.”
• Bump cflinuxfs2 to version 1.213.0
• Bump consul to version 193 to use go 1.9
• Bump stemcell to version 3468.46

1.12.20

• [Security Fix] Bump cflinuxfs2 to version 1.210.0:
  • USN-3643-1
• Bump cf-networking to version 1.4.3
• Update grootfs checkbox to indicate the recreating VMs is recommended

1.12.19

• [Security Fix] Bump stemcell to v3468.42:
  • USN-3641
  • USN-3631-2
  • USN-3628-1
  • USN-3625-1
  • USN-3624-1
• [Security Fix] Bump cflinuxfs2-release to v1.201.0:
  • USN-3628-1
  • USN-3625-1
  • USN-3624-1
  • USN-3622-1
• [Feature Improvement] Bump routing-release to v0.163.14 to enable operator to disable logging of client IPs, in compliance with the EU General Data Protection Regulation (GDPR).
• [Bug Fix] Provide the Ops Manager root CA certificate and any other operator-provided trusted certificates to all containers in the /etc/cf-system-certificates directory.

1.12.18

• [Security Fix] Bumps garden-runc to version 1.13.1
  • CVE-2018-1277.
• [Bug Fix] Add “Shared and Segment” sharding mode for routers in Isolation Segments
  • Allows for zero downtime upgrades from IST 1.11 for applications that are only reachable through the isolation segment routers.
• [Feature Improvement] Bumps diego-release to v1.25.13 to add cell and instance identifiers in the container lifecycle logs.

1.12.17

• [Security Fix] Bumps cflinuxfs2 to v1.196.0:
  • USN-3611-1
  • USN-3610-1
• [Security Fix] Bumps stemcell to v3468.30:
  • USN-3619-2
  • USN-3611-1
  • USN-3610-1
  • USN-3598-1
  • USN-3586-1
  • USN-3584-1
• [Bug Fix] Bumps syslog-migration-release to v8.0.2:
  • Prevent logs from blackbox from being written to the default syslog log files to prevent logs from being written to the disk 3 additional times.
  • Fix rfc5424 compatibility by ensuring only 1 space occurs between the message and the structured data.
• [Feature Improvement] Adds field Custom rsyslog Configuration to specify custom logging rules in the System Logging tab. For more information, see custom syslog rules.

1.12.16

• [Feature Improvment] Bumps garden-runc-release to v1.12.1:
  • Includes fix for bug where users’ files could go missing in docker-based applications.
• [Bug fix] Bumps routing-release to 0.163.13:
  • Removes backends on any error to prevent 502 errors from being returned to clients.
  • Updates golang to v1.9.4.

1.12.15

Note: it is recommended that you re-create all VMs when upgrading to this release, due to the update to garden-runc-release. This will happen automatically if you are updating your stemcell. If not, you can check the “Recreate All VMs” checkbox on the Ops Manager Director > Director Config tab.

• [Security Fix] Bumps stemcell from version 3468.21 to version 3468.25 to address issues:
  • USN-3582-2
• [Security Fix] Bumps cflinuxfs2-release from v181.0 to v1.188.0 to address issues:
  • USN-3577-1
  • USN-3569-1
  • USN-3554-1
  • USN-3547-1
  • USN-3543-1
  • USN-3540-2
  • USN-3538-1
• [Feature Improvement] Bumps garden-runc-release to v1.11.1 which includes grootfs root filesystem by default.

1.12.14

• [Security Fix] Patches routing-release for CVE-2018-1221.
• [Bug Fix] Enables privileged containers to support upgrading from ERT 1.11 with apps that specify privileged containers.
• [Bug Fix] Fix to ensure that Diego rep will always exit during evacuation, even if Garden destroy hangs during evacuation.
• [Bug Fix] Patches syslog to prevent duplication from blackbox log forwarding.
• [Feature Improvements] New option in the Networking page to allow operators to enable Gorouter support for the PROXY protocol. This is disabled by default.
• [Feature Improvement] Enable Garden debug_listen_address to listen on a local interface.

1.12.13

• This release was intended to contain a fix to prevent duplication from blackbox log forwarding, but this did not take effect until the next release.

1.12.12

• [Security Fix] Bump stemcell to version 3468.21 to address issues:
  • USN-3534-1
  • USN-3540-2
• [Security Fix] Bump cflinuxfs2-release to v1.181.0 to address issues:
  • USN-3532-1
  • USN-3534-1
  • USN-3535-1
• [Feature Improvement] Bump syslog-migration-release to v8.0.1 and add a checkbox for log file forwarding through TCP to work around the Truncated Syslog Messages issue.
  • NOTE: Using TCP instead of the default UDP configuration may have a negative impact on performance.

1.12.11

• [Security Fix] Bumps stemcell version to 3445.22 for USN-3544-2 and USN-3544-4

1.12.10

• [Security Fix] Bumps cflinuxfs2-release to v1.176.0 for USN-3513-1.
• [Feature] Bumps garden-runc-release to v1.10.0:
  • It is now possible to specify a ProcessSpec.Image. Processes can now have their own filesystem view.
  • Limitation: It is only possible to use ProcessSpec.Image and ProcessSpec.OverrideContainerLimits with unprivileged containers.
    This will be fixed in future releases.
  • Limitation: APIs such as BulkMetrics and Process.Signal may not work immediately after container.Run(ProcessSpec) returns for processes with Image and/or OverrideContainerLimits specified. This will be fixed in future releases.
  • Reduced log volume in BulkMetrics for large environments.
  • Correctly declares that bundles it creates are OCI Runtime Spec version 1.0.0 compliant.
• The Garden property cleanup_process_dirs_on_wait is configured to true, to reduce the growth of directories in the Garden container.

1.12.9

• [Security Fix] Bumps stemcell version to 3445.19 for USN-3509-2.
• [Security Fix] Bumps cflinuxfs2-release to v1.171.0 to resolve several security vulnerabilities:
  • USN-3489-1: Berkeley DB vulnerability
  • USN-3496-1: Python vulnerability
  • USN-3496-3: Python vulnerability
  • USN-3498-1: curl vulnerabilities
  • USN-3501-1: libxcursor vulnerability
• [Bug Fix] Operators can now optionally disable Router Access logs. This will prevent the Router local disk from becoming filled when the Routers are experiencing increased incoming traffic.
• [Feature Improvement] Operators can now specify the mutual TLS certificate validation behavior for the Router. The Router will request certificates by default, and validate them if provided. Additionally, operators can configure the Router to ignore certificates, or to require them with every request.
• This release does not set the Garden property cleanup_process_dirs_on_wait to true, which can leave many directories in the depot for the Garden container. This will be set to true in the next release.

1.12.8

• [Security Fix] Bumps the stemcell to v3445.17 to resolve the following security issues:
  • USN-3457-1: curl vulnerability
  • USN-3458-1: ICU vulnerability
  • USN-3464-1: Wget vulnerabilities
  • USN-3469-2: Linux kernel (Xenial HWE) vulnerabilities
  • USN-3475-1: OpenSSL vulnerabilities
  • USN-3478-1: Perl vulnerabilities
  • USN-3485-2: Linux kernel (Xenial HWE) vulnerabilities

• [Security Fix] Bumps cflinuxfs2-release to v1.168.0 to resolve USN-3478-1: Perl vulnerabilities.

  Component	Version
  Stemcell	3445.17
  cf-networking	1.4.0*
  cflinuxfs2	1.168.0
  consul	187
  diego	1.25.3
  garden-runc	1.9.4
  grootfs	0.30.0
  haproxy	8.4.1
  loggregator	96*
  nfs-volume	1.0.9
  routing	0.163.0
  syslog-migration	8
  * Components marked with an asterisk have been patched to resolve security vulnerabilities or fix component behavior.

1.12.7

• [Security Fix] Bumps cflinuxfs2-release to v1.166.0 to resolve USN-3475-1. Release Notes
• [Security Fix] Bumps grootfs-release to v0.30.0 to resolve CVE-2017-14388. Release Notes.

1.12.6

• [Security Fix] Bumps the stemcell to v3445.16 to resolve several security vulnerabilities:
  • USN-3424-1
  • USN-3432-1
  • USN-3434-1
  • USN-3441-1
  • USN-3444-2
• [Security Fix] Bumps the cflinuxfs2-release to v1.165.0 to resolve several security vulnerabilities:
  • USN-3457-1
  • USN-3458-1
  • USN-3464-1
• [Bug Fix] Patches loggregator-release to remove the totalReceivedMessageCount metric from the v2 API.
• [Bug Fix] Garden is now configured to destroy containers on start. This setting will cause the garden process to remove any containers that are already running when it starts. That action will prevent issues where containers that should no longer be running are left up to run.
• The Router has now been configured to automatically validate and trust certificates issued by the Diego Instance Identity CA.

1.12.5

• [Security Fix] Bumps cflinuxfs2-release to v1.161.0 to resolve multiple security issues. Release Notes
• [Bug Fix] Bumps consul-release to v181 to ensure encrypt key rotation only occurs when the key changes.

1.12.4

• [Security Fix] Bumps cflinuxfs2-release to v1.158.0 to resolve multiple security issues. Release Notes

1.12.3

• [Security Improvement] Bumps garden-runc-release to v1.9.4. Release Notes.

1.12.2

• [Security Fix] Bumps cflinuxfs2-release to v1.156.0 to resolve multiple security issues. Release Notes
• [Security Fix] Resolves an issue with an incorrect Host header being set on incoming requests through the Router CVE Notice.
• [Bug Fix] Bumps haproxy-boshrelease to v8.4.1 to resolve an issue with certificate/key concatenation Release Notes.
• [Feature] Bumps diego-release to v1.25.3 to include support for Azure MySQL Release Notes.
• [Feature] Patches cf-networking-release to include support for Azure MySQL.
• [Stability Improvement] Changes the default Router Max Connections Per Backend from 0, or unlimited, to 500. This change prevents an unresponsive app from consuming all the router file descriptors. In some cases, this may impact the performance of existing apps and you may need to raise the setting. For guidance, see the documentation about the Max Connections Per Backend field in Installing PCF Isolation Segment.
  
• [Feature] Operators can now configure a “Frontend Idle Timeout” for the Router and HAProxy. The default timeout is 900 seconds.

1.12.1

• [Security Fix] Bumps stemcell to v3445.11 to address USN-3420-2.
• [Security Fix] Bumps cflinuxfs-release to v1.155.0 to address USN-3415-1.

1.12.0

About PCF Isolation Segment

The PCF Isolation Segment v1.12 tile is available for installation with PCF v1.12.

Isolation segments provide dedicated pools of resources where you can deploy apps and isolate workloads. Using isolation segments separates app resources as completely as if they were in different CF deployments but avoids redundant management and network complexity.

For more information about using isolation segments in your deployment, see the Managing Isolation Segments topic.

How to Install

The procedure for installing PCF Isolation Segment v1.12 is documented in the Installing PCF Isolation Segment topic.

To install a PCF Isolation Segment, you must first install PCF v1.12.

New Features in PCF Isolation Segment v1.12.0

This section describes new features of the release.

HAProxy

You can now use an HAProxy for the Isolation Segment tile that is independent from the Elastic Runtime HAProxy. The Isolation Segment tile includes its own HAProxy VM, which uses the haproxy-boshrelease.

For information on the configuration options for HAProxy, see the Networking section of Installing PCF Isolation Segment.

Router Sharding Mode

This release includes support for router sharding between the Elastic Runtime and Isolation Segment tiles. In addition to compute isolation, operators can now configure networking isolation for their isolation segments. Alternatively, operators can deploy the Isolation Segment tile for the purpose of using its routers for the Elastic Runtime tile.

You can configure this feature using the following fields:

• Elastic Runtime tile: Routers reject requests for Isolation Segments checkbox
• Isolation Segment tile: Router Sharding Mode selector

See the Isolation Segment installation instructions for more information.

Router Configuration Options

The Networking pane of the Isolation Segment tile now includes the following fields for the router. Additionally, the default instance count for the Router VM is now set to 3 in the Resource Config pane.

• Router Max Idle Keepalive Connections
• Disable SSL certificate verification for this environment
• HTTP Headers to Log
• Disable insecure cookies on the Router
• Enable Zipkin tracking headers on the router
• Router Timeout to Backends
• Load Balancer Unhealthy Threshold
• Load Balancer Healthy Threshold
• Max Connections Per Backend

For more information about each field, see the Isolation Segment installation instructions.

Simplified TLS Configuration

The point of entry options on the Isolation Segment tile Networking pane have been restructured to be more understandable and flexible. Operators no longer need to configure the Gorouter or HAProxy separately as both components are configured using the same options. This includes the following changes:

• The Gorouter and HAProxy always listen for TLS requests. You provide an SSL certificate for both the Gorouter and HAProxy using a single field.
• HAProxy forwards all requests to the Gorouter over TLS by default. You can optionally disable this feature.
• You can configure the minimum version of TLS for the Gorouter and HAProxy with a single field.
• You can provide a list of CAs to HAProxy for it to validate the Gorouter certificate.
• You can optionally disable the HTTP listener for both the Gorouter and HAProxy with a single checkbox.
• You can specify TLS cipher suites for the Gorouter and HAProxy independently.

Mutual TLS Headers on Inbound Application Traffic

The Gorouter can now forward the X-Forwarded-Client-Cert header to app instances when provided. Alternatively, operators can configure the Gorouter to forward the header only when the mutual TLS connection from the client can be validated. Additionally, operators may now configure the Gorouter to overwrite the XFCC header with the client certificate received in mTLS handshakes.

This configuration is available in the Networking pane under Configure the CF Router support for the X-Forwarded-Client-Cert header. For more information, see the Isolation Segment installation instructions.

Support for Logging All App Traffic

Operators can enable logging of all accepted and denied packets due to ASGs or container-to-container networking policies. This provides more visibility into app traffic, including denied traffic.

Operators configure this global logging in the Networking pane of the Isolation Segment tile under the Log traffic for all accepted/denied application packets field. See the App Traffic Logging section for more information.

Introducing GrootFS

GrootFS is the new container image management plugin for Garden-runC. It helps with the filesystem isolation of Garden-runC containers, image caching, and disk quota enforcement. GrootFS replaces the previous built-in functionality, which used an obsolete layer filesystem (AUFS) that lacks support from the Linux Kernel community. Additionally, GrootFS uses OCI standards for container images.

For more information about GrootFS in PCF, see the following topics:

• Understanding Container Security
• Component: Garden

Garden Disk-Cleanup

Operators using PCF Isolation Segment can now configure the Garden and GrootFS cleanup thresholds independently of the Elastic Runtime tile. The Application Containers pane of the Isolation Segment tile now includes the following selector: Docker Images Disk-Cleanup Scheduling on Cell VMs.

Diego Cell Max-in-Flight Default

This release lowers the default max-in-flight percentage on Diego Cells to 4%. Previously, this value was set to 10%, which can cause the following issues in larger environments:

• Many simultaneous VM creates/deletes and BOSH blob updates placing significant stress on the underlying infrastructure, especially on vSphere which has a greater probability of being under-provisioned.
• Cells that are draining are no longer available for allocation, resulting in a 10% decrease in total memory and disk capacity during deployment. This can cause deployments to no longer have sufficient total capacity to run all work, or to have insufficient headroom to place larger workloads successfully.

Operators can still use the Ops Manager API to configure this setting to fit their needs. For more information about this property, see Managing Diego Cell Limits During Upgrade.

Secure Communication Between Diego and Loggregator

Diego Cells now use the Metron API v2. This gRPC-based API supports mutual TLS authentication and secures the communication path between the Diego rep and Loggregator.

NFSV3 Volume Services with LDAP

Operators can now configure LDAP for NFSv3 volume services. Using LDAP secures the NFSV3 volume service by preventing a developer from binding to an NFS share using an arbitrary UID and potentially gaining access to sensitive data stored by another user or app. If you enable LDAP support, developers must provide credentials for any user they wish to bind as. For more information, see Enabling NFS Volume Services.

Application Instance Identity Credentials

The instance identity system in Diego provides each app container with a PEM-encoded X.509 certificate and PKCS #1 RSA private key. The values of the environment variables CF_INSTANCE_CERT and CF_INSTANCE_KEY contain the absolute paths to the certificate and private key files. The validity period is 3 years for the Instance Identity root and 2 years for the intermediate CA certificates.

For more information, see the App Instance Container Identity Credentials.

Container-to-Container Networking Updates

Container-to-container networking is now always enabled. The commands are integrated with the cf CLI and now include the option to specify a port range when adding and removing policies. See Create Policies for Container-to-Container Networking.

About Advanced Features

The Advanced Features section of the PCF Isolation Segment tile includes new functionality that may have certain constraints. Although these features are fully supported, Pivotal recommends caution when using them in production.

Known Issues

Apps Deployed to PCF Isolation Segment Unreachable

Note: This has been fixed in PCF Isolation Segment v1.12.18. For more information, see Sharding Routers for Isolation Segments in Routing for Isolation Segments and Step 7: Upgrade PCF Isolation Segment in Upgrading Pivotal Cloud Foundry.

If you upgrade from PCF v1.11 to v1.12 and the PCF Isolation Segment tile is installed on your foundation, any apps deployed to a space associated with the tile may become unreachable until you manually restart them or map an arbitrary route to each of them.

This happens because the isolation segment metadata for your existing apps is not automatically sent to the Gorouter on upgrade. After you restart a given app or map a route to it, the Gorouter receives the updated route information, including the isolation segment metadata.

This issue is addressed in PCF v1.12.18 and later. If you are upgrading from PCF v1.11 to v1.12, please upgrade to at least PCF v1.12.18.