PCF Testing, Release, and Security Lifecycle
This topic explains how Pivotal’s development practices, automated build tools, and organizational structures work together to create and support stable releases of Pivotal Cloud Foundry (PCF).
- PCF teams building system components receive frequent feedback, which helps to secure code from exposure to vulnerability.
- Every PCF release follows a strict workflow and passes through numerous quality and compliance checks before distribution.
- Teams build tests into the product consistently and run them automatically with any code change.
Pivotal releases, patches, and supports multiple versions of PCF simultaneously. This section explains the versioning and support conventions Pivotal follows.
Pivotal numbers PCF releases following a semantic versioning style format, X.Y.Z, where X and Y indicate major and minor releases, and Z designates patch releases. Major and minor releases change PCF functionality; patch releases are backward-compatible security patches or bug fixes.
As of PCF 1.8, Pivotal supports each major and minor PCF release according to the following criteria:
- Pivotal supports the release for at least 9 months following its first publication date.
- Pivotal supports the last three major or minor releases, even if this extends coverage beyond 9 months.
Support includes maintenance updates and upgrades, bug and security fixes, and technical assistance. The Pivotal Support Policy describes support standards, technical guidance, and publication phases in more detail. The Pivotal Support Services Terms and Conditions defines Pivotal support in legal terms.
Patch releases are more frequent and less predictable than major/minor releases. The v1.6.x line provides a good example of their frequency. PCF 1.6.1 was released on October 26, 2015. Through August 2016, 36 additional patches of Elastic Runtime 1.6.x and 18 patches of Ops Manager 1.6.x provided security and bug fixes to customers.
Pivotal identifies security issues using standard nomenclature from Common Vulnerabilities and Exposures (CVE), Ubuntu Security Notices (USN), and other third party sources. Read about security fixes in core Cloud Foundry code or packaged dependencies in the release notes for Ops Manager and Pivotal Application Service (PAS).
Pivotal.io/security maintains a running list of security fixes in PCF and PCF dependencies. Consult that page to see the most recent findings from Pivotal’s security team.
All PCF releases pass through extensive test suites that include automated unit, integration, and acceptance tests on multiple IaaSes. Regardless of this extensive testing, Pivotal recommends that you test major and minor releases in a non-production environment before implementing them across your deployment. Upgrade your production environment as soon as possible after you validate the new release on your test environment.
Release Testing, Integration, and Validation
This section describes Pivotal’s software development processes and explains compliance and regulatory standards to which Pivotal software adheres.
Pivotal’s development process relies on a strict workflow with continuous automated testing. Pivotal R&D does not separate engineering and testing teams. Rather, every Pivot on each engineering team is responsible for ensuring the quality of their code. They write tests for all of the software components that they develop, often before writing the software itself.
With every software change, automated build pipelines trigger these tests for the new software component and for everything it touches. If a new code check-in does not pass its tests or causes a failure elsewhere, it pauses the build pipeline for the entire team, or sometimes all of Pivotal R&D. The transparency of this process encourages developers to work together to address code issues quickly.
Pivotal applies the following automated testing approaches, scenarios, and frameworks to PCF components and to the release as a whole:
Unit tests: Development teams write unit tests to express and validate desired functional behavior of product components. Typical frameworks used are RSpec and Ginkgo. These tests run continuously throughout the development cycle.
OSS integration tests: The Release Integration team exercises a full deployment of open-source Cloud Foundry to validate all end-user features. They maintain the Cloud Foundry Acceptance Test (CATs) suite alongside the OSS cf-release. Cloud Foundry component teams also contribute acceptance test suites at the OSS Integration Test level. These tests exercise and validate their components’ functional, performance, and integration health.
PCF integration tests: The PCF Release Engineering (RelEng) team validates the quality and cross-product integration health of the commercial PCF release. RelEng runs OSS Acceptance Tests against all supported releases. These tests run on full PCF instances configured to represent diverse real-world customer scenarios on various IaaSes and using both internal and external load balancer, database, blobstore, and user store solutions.
Additional Pre-Release Gates: Internal, PWS, and Compliance
In addition to its automated unit and integration testing, Pivotal deploys all upgrades slated for upcoming PCF releases on at-scale test environments. Prior to each Major or Minor commercial release, Pivotal runs the entire Pivotal Cloud Foundry Suite of services on several internally-managed large integration environments that run customer-like data and workloads.
Pivotal also pushes upcoming PCF feature upgrades and patches to its Pivotal Web Services platform, where customers continually deploy and host hundreds of mission-critical applications at scale, 24/7. The PWS environment gives Pivotal a continuous source of real-world usage and performance metrics that inform product development teams.
All PCF product teams participate in go-to-market steps for each release, as is often required for shipping a legally compliant product. Examples include Open Source License File attribution and an Export Compliance classification.
Patch Releases: Security and Bug Fixes
Pivotal uses established processes to track, disclose, and remediate vulnerabilities in PCF and related dependent components. This section explains how Pivotal identifies vulnerabilities and implements fixes for them.
Identifying Security Vulnerabilities
Pivotal has an established process to track and patch vulnerabilities in software dependencies and PCF software. Additionally, pivotal.io/security describes a responsible disclosure process for reporting vulnerabilities identified in Pivotal software by 3rd parties.
Pivotal uses multiple methods to identify security vulnerabilities in Pivotal software and dependencies internally, including:
- Security notifications from Canonical for their Ubuntu operating system, provided through Pivotal’s commercial relationship with Canonical
- Software component scans several times per day, using 3rd party security software which updates continuously from external security vulnerability sources
- Dependency analysis software that identifies and catalogs software dependencies
- Security vulnerability notifications from known software dependencies
Pivotal also monitors externally-reported vulnerabilities from many sources, including:
- 3rd party security analysis requested by Pivotal
- Cloud Foundry Foundation security notifications from member companies
- Customer, prospect and other 3rd party security reports
When Pivotal discovers a potential security vulnerability in PCF, the security team opens an issue to assess it. If it confirms the vulnerability exists, Pivotal identifies and updates affected components with plans to backport the fix to stable releases. Fixes are implemented on a target timeline based on the severity level of the vulnerability.
Fix, Test, and Release Lifecycle
This flowchart details the steps that Pivotal performs on a typical high-priority CVE, to publish a patch release fix on https://network.pivotal.io: