Testing, Release, and Security Lifecycle
Page last updated:
This topic explains how Pivotal’s development practices, automated build tools, and organizational structures work together to create and support stable releases of Pivotal Platform.
Pivotal Platform teams building system components receive frequent feedback, which helps to secure code from exposure to vulnerability.
Every Pivotal Platform release follows a strict workflow and passes through numerous quality and compliance checks before distribution.
Teams build tests into the product consistently and run them automatically with any code change.
Pivotal releases, patches, and supports multiple versions of Pivotal Platform simultaneously. This section explains the versioning and support conventions Pivotal follows.
Pivotal numbers Pivotal Platform releases following a semantic versioning format, X.Y.Z, where X and Y indicate major and minor releases, and Z designates patch releases. Major and minor releases change Pivotal Platform functionality. Patch releases are backward-compatible security patches or bug fixes. For more information about semantic versioning, see the Semantic Versioning 2.0.0 website.
As of v1.8, Pivotal supports each major and minor Pivotal Platform release according to the following criteria:
- Pivotal supports the release for at least 9 months following its first publication date.
- Pivotal supports the last three major or minor releases, even if this extends coverage beyond 9 months.
Support includes maintenance updates and upgrades, bug and security fixes, and technical help. The Pivotal Support Policy describes support standards, technical guidance, and publication phases in more detail. The Pivotal Support Services Terms and Conditions defines Pivotal support in legal terms.
Patch releases are more frequent and less predictable than major/minor releases. The v1.6.x line provides a good example of their frequency. v1.6.1 was released on October 26, 2015. Through August 2016, 36 additional patches of Elastic Runtime v1.6.x and 18 patches of Ops Manager v1.6.x provided security and bug fixes to customers.
Pivotal identifies security issues using standard nomenclature from Common Vulnerabilities and Exposures (CVE), Ubuntu Security Notices (USN), and other third party sources. Read about security fixes in core Pivotal Platform code or packaged dependencies in the release notes for Ops Manager and Pivotal Application Service.
Pivotal.io/security maintains a running list of security fixes in Pivotal Platform and Pivotal Platform dependencies. Consult that page to see the most recent findings from Pivotal’s security team.
All Pivotal Platform releases pass through extensive test suites that include automated unit, integration, and acceptance tests on multiple IaaSes. Regardless of this extensive testing, Pivotal recommends that you test major and minor releases in a non-production environment before implementing them across your deployment. Upgrade your production environment as soon as possible after you validate the new release on your test environment.
This section describes Pivotal’s software development processes and explains compliance and regulatory standards to which Pivotal software adheres.
Pivotal’s development process relies on a strict workflow with continuous automated testing. Pivotal R&D does not separate engineering and testing teams. Rather, every Pivot on each engineering team is responsible for ensuring the quality of their code. They write tests for all of the software components that they develop, often before writing the software itself.
With every software change, automated build pipelines trigger these tests for the new software component and for everything it touches. If a new code check-in does not pass its tests or causes a failure elsewhere, it pauses the build pipeline for the entire team, or sometimes all of Pivotal R&D. The transparency of this process encourages developers to work together to address code issues quickly.
Pivotal applies the following automated testing approaches, scenarios, and frameworks to Pivotal Platform components and to the release as a whole:
Unit tests: Development teams write unit tests to express and validate desired functional behavior of product components. Typical frameworks used are RSpec and Ginkgo. These tests run continuously throughout the development cycle.
OSS integration tests: The Release Integration team exercises a full deployment of open-source Cloud Foundry to validate all end-user features. They maintain the Cloud Foundry Acceptance Test (CATs) suite alongside the OSS cf-release. Cloud Foundry component teams also contribute acceptance test suites at the OSS Integration Test level. These tests exercise and validate their components’ functional, performance, and integration health.
Pivotal Platform integration tests: The Pivotal Platform Release Engineering (RelEng) team validates the quality and cross-product integration health of the commercial Pivotal Platform release. RelEng runs OSS Acceptance Tests against all supported releases. These tests run on full Pivotal Platform instances configured to represent diverse real-world customer scenarios on various IaaSes and using both internal and external load balancer, database, blobstore, and user store solutions.
In addition to its automated unit and integration testing, Pivotal deploys all upgrades slated for upcoming Pivotal Platform releases on at-scale test environments. Prior to each Major or Minor commercial release, Pivotal runs the entire Pivotal Platform Suite of services on several internally-managed large integration environments that run customer-like data and workloads.
Pivotal also pushes upcoming Pivotal Platform feature upgrades and patches to its Pivotal Web Services (PWS) platform, where customers continually deploy and host hundreds of mission-critical apps at scale, 24/7. The PWS environment gives Pivotal a continuous source of real-world usage and performance metrics that inform product development teams.
All Pivotal Platform product teams participate in go-to-market steps for each release, as is often required for shipping a legally compliant product. Examples include Open Source License File attribution and an Export Compliance classification.
Pivotal uses established processes to track, disclose, and remediate vulnerabilities in Pivotal Platform and related dependent components. This section explains how Pivotal identifies vulnerabilities and implements fixes for them.
Pivotal has an established process to track and patch vulnerabilities in software dependencies and Pivotal Platform software. Additionally, pivotal.io/security describes a responsible disclosure process for reporting vulnerabilities identified in Pivotal software by third parties.
Pivotal uses multiple methods to identify security vulnerabilities in Pivotal software and dependencies internally, including:
- Security notifications from Canonical for their Ubuntu operating system, provided through Pivotal’s commercial relationship with Canonical
- Software component scans that use third-party security software to update continuously from external security vulnerability sources
- Dependency analysis software that identifies and catalogs software dependencies
- Security vulnerability notifications from known software dependencies
Pivotal also monitors externally-reported vulnerabilities from many sources, including:
- Third-party security analysis requested by Pivotal
- Cloud Foundry Foundation security notifications from member companies
- Customer, prospect and other third-party security reports
When Pivotal discovers a potential security vulnerability in Pivotal Platform, the security team opens an issue to assess it. If it confirms the vulnerability exists, Pivotal identifies and updates affected components with plans to backport the fix to stable releases. Fixes are implemented on a target timeline based on the severity level of the vulnerability.
The following flowchart details the steps that Pivotal performs on a typical high-priority CVE, to publish a patch release fix on Pivotal Network: