Testing, Release, and Security Lifecycle
Page last updated:
This topic explains how Pivotal’s development practices, automated build tools, and organizational structures work together to create and support stable releases of Ops Manager.
Ops Manager teams building system components receive frequent feedback, which helps to secure code from exposure to vulnerability.
Every Ops Manager release follows a strict workflow and passes through numerous quality and compliance checks before distribution.
Teams build tests into the product consistently and run them automatically with any code change.
Pivotal releases, patches, and supports multiple versions of Ops Manager and runtime products simultaneously. This section explains the versioning and support conventions Pivotal follows.
Pivotal numbers releases following a semantic versioning format, X.Y.Z, where X and Y indicate major and minor releases, and Z designates patch releases. Major and minor releases change functionality. Patch releases are backward-compatible security patches or bug fixes. For more information about semantic versioning, see the Semantic Versioning 2.0.0 website.
As of Pivotal Cloud Foundry (PCF) v1.8, Pivotal supports each major and minor platform release according to the following criteria:
- Pivotal supports the release for at least nine months following its first publication date.
- Pivotal supports the last three major or minor releases, even if this extends coverage beyond nine months.
Support includes maintenance updates and upgrades, bug and security fixes, and technical help. For more details about support standards, technical guidance, and publication phases, see Support Lifecycle Policy on the Pivotal website. For a definition of Pivotal support in legal terms, see Pivotal Support Services Terms & Conditions on the Pivotal website.
Patch releases are more frequent and less predictable than major and minor releases. The PCF v1.6.x line provides a good example of their frequency. PCF v1.6.1 was released on October 26, 2015. Through August 2016, 36 additional patches of Elastic Runtime v1.6.x and 18 patches of Ops Manager v1.6.x provided security and bug fixes to customers.
VMware identifies security issues using standard nomenclature from Common Vulnerabilities and Exposures (CVE), Ubuntu Security Notices (USN), and other third-party sources. For more information about security fixes in core Ops Manager code or packaged dependencies, see Ops Manager v2.7 Release Notes and Pivotal Application Service v2.7 Release Notes.
The Pivotal Application Security Team maintains a running list of security fixes in Ops Manager and Ops Manager dependencies. To see their most recent findings, see Pivotal Application Security Team on the VMware Tanzu website.
All Ops Manager releases pass through extensive test suites that include automated unit, integration, and acceptance tests on multiple IaaSes. Regardless of this extensive testing, VMware recommends that you test major and minor releases in a non-production environment before implementing them across your deployment. Upgrade your production environment as soon as possible after you validate the new release on your test environment.
This section describes Pivotal’s software development processes and explains compliance and regulatory standards to which Pivotal software adheres.
Pivotal’s development process relies on a strict workflow with continuous automated testing. Pivotal R&D does not separate engineering and testing teams. Rather, every Pivot on each engineering team is responsible for ensuring the quality of their code. They write tests for all of the software components that they develop, often before writing the software itself.
With every software change, automated build pipelines trigger these tests for the new software component and for everything it touches. If a new code check-in does not pass its tests or causes a failure elsewhere, it pauses the build pipeline for the entire team, or sometimes all of Pivotal R&D. The transparency of this process encourages developers to work together to address code issues quickly.
VMware applies the following automated testing approaches, scenarios, and frameworks to Ops Manager components and to the release as a whole:
Unit tests: Development teams write unit tests to express and validate desired functional behavior of product components. Typical frameworks used are RSpec and Ginkgo. These tests run continuously throughout the development cycle.
OSS integration tests: The Release Integration team exercises a full deployment of open-source Cloud Foundry to validate all end-user features. They maintain the Cloud Foundry Acceptance Test (CATs) suite alongside the OSS cf-release. Cloud Foundry component teams also contribute acceptance test suites at the OSS Integration Test level. These tests exercise and validate their components’ functional, performance, and integration health.
Ops Manager integration tests: The Ops Manager Release Engineering (RelEng) team validates the quality and cross-product integration health of the commercial Ops Manager release. RelEng runs OSS Acceptance Tests against all supported releases. These tests run on full Ops Manager instances configured to represent diverse real-world customer scenarios on various IaaSes and using both internal and external load balancer, database, blobstore, and user store solutions.
In addition to its automated unit and integration testing, VMware deploys all upgrades slated for upcoming Ops Manager releases on at-scale test environments. Prior to each major or minor commercial release, VMware runs the entire Ops Manager suite of services on several internally-managed large integration environments that run customer-like data and workloads.
VMware also pushes upcoming Ops Manager feature upgrades and patches to its Pivotal Web Services (PWS) platform, where customers continually deploy and host hundreds of mission-critical apps at scale, 24/7. The PWS environment gives VMware a continuous source of real-world usage and performance metrics that inform product development teams.
All Ops Manager product teams participate in go-to-market steps for each release, as is often required for shipping a legally compliant product. Examples include Open Source License File attribution and an Export Compliance classification.
VMware uses established processes to track, disclose, and remediate vulnerabilities in Ops Manager and related dependent components. This section explains how VMware identifies vulnerabilities and implements fixes for them.
VMware has an established process to track and patch vulnerabilities in software dependencies and Ops Manager software. Additionally, the Pivotal Application Security Team page describes a responsible disclosure process for reporting vulnerabilities identified in VMware software by third parties.
Pivotal uses multiple methods to identify security vulnerabilities in Pivotal software and dependencies internally, including:
- Security notifications from Canonical for their Ubuntu operating system, provided through Pivotal’s commercial relationship with Canonical
- Software component scans that use third-party security software to update continuously from external security vulnerability sources
- Dependency analysis software that identifies and catalogs software dependencies
- Security vulnerability notifications from known software dependencies
Pivotal also monitors externally-reported vulnerabilities from many sources, including:
- Third-party security analysis requested by Pivotal
- Cloud Foundry Foundation security notifications from member companies
- Customer, prospect and other third-party security reports
When VMware discovers a potential security vulnerability in Ops Manager, the security team opens an issue to assess it. If it confirms the vulnerability exists, VMware identifies and updates affected components with plans to backport the fix to stable releases. Fixes are implemented on a target timeline based on the severity level of the vulnerability. For more information, see Ops Manager Security Overview and Policy.
The following flowchart details the steps that Pivotal performs on a typical high-priority CVE, to publish a patch release fix on Pivotal Network:
The triage flowchart shows the following steps:
Vulnerability reported in a Cloud Foundry Dependency
OSS Cloud Foundry team addresses vulnerability
- Story detailing the vulnerability and acceptance criteria for fiex priorized on OSS backlog
- Unit Tests written before coding the fix
- Fix written; Unit Tests pass
- Code committed; OSS Integration Tests pass
- Check: Does Product Owner verify that fix meets acceptance criteria?
- If yes, fix is ready for integration.
VMware Tanzu team integrates fix
- Story detailing the issue and acceptance criteria for fix prioritized on Commercial backlog
- Fix patched in all release lines under General Support
- Check: Have all tests passed each stage?
- If yes, patched builds kick off integration pipeline.
Release Engineering Integration Pipelines validate quality
- Tests pass by scenario:
- New installation scenario: integration tests pass
- Upgrade scenario: Integration Tests pass
- Additional customer scenarios: Integration Tests pass
- Check: Have all pipelines passed on every supported IaaS?
- If yes, ready for Publishing
- Tests pass by scenario:
Release is published for distribution
- Post to VMware Network; with High CVE vulnerab ility release the fix with as fast a turnaround as possible
- Customer receives announcements and downloads release for their IaaS