LATEST VERSION: 1.10 - CHANGELOG
Pivotal Cloud Foundry v1.10

Upgrading Dependency Versions

Page last updated:

This topic describes how to upgrade a dependency version in a custom buildpack. These procedures enable Cloud Foundry (CF) operators to maintain custom buildpacks that contain dependencies outside of the dependencies in the CF system buildpacks.

Cloud Foundry Buildpacks Team Process

Note: The procedures in this topic refer to the tools used by the CF buildpacks team, but they do not require the specific tools listed below. You can use any continuous integration (CI) system and workflow management tool to update dependencies in custom buildpacks.

The CF buildpacks team uses the following tools to update dependencies:

When the New Releases job in the notifications pipeline detects a new version of a tracked dependency in a buildpack, it creates a Tracker story about building and including the new version of the dependency in the buildpack manifests. It also posts a message as the dependency-notifier to the #buildpacks channel in the Cloud Foundry Slack channel.

Build the Binaries

For all dependencies, you must build the binary from source or acquire the binary as a tarball from a trusted source. For most dependencies, the CF buildpacks team builds the binaries from source.

Note: The steps below assume you are using a Concourse deployment of the buildpacks-ci pipelines and Pivotal Tracker.

To build the binary for a dependency, perform the following steps:

  1. Navigate to the public-buildpacks-ci-robots directory and verify no uncommitted changes exist.
    $ cd ~/workspace/public-buildpacks-ci-robots
    $ git status
    
  2. Run the git pull command in the directory to ensure it contains most recent version of the contents.
    $ git pull -r
    
  3. Navigate to the binary-builds directory.
    $ cd binary-builds
    
  4. Locate the YAML file for the buildpack you want to build a binary for. The directory contains YAML files for all the packages and dependencies tracked by the CF buildpacks team. Each YAML file correlates to the build queue for one dependency or package, and uses the naming format DEPENDENCY-NAME.yml. For example, the YAML file tracking the build queue for Ruby is named ruby-builds.yml and contains the following contents:

    ---
    ruby: []
    
  5. Different buildpacks use different signatures for verification. Determine which signature your buildpack requires by consulting the list in the buildpacks section of this topic and run follow the instructions to locate the SHA256, MD5, or GPG signature for the binary:

    • For the SHA256 of a file, run shasum -a 256 FILE-NAME.
    • For the MD5 of a file, run md5 FILE-NAME.
    • For the GPG signature (for Nginx), see the Nginx Downloads page.
  6. Add the version and verification for the new binary to the YAML file as attributes of an element under the dependency name. For example, to build the Ruby 2.3.0 binary verified with SHA256, add the following to the YAML file:

    ---
    ruby:
    - version: 2.3.0
      sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
    

    Note: Do not preface the version number with the name of the binary or language. For example, specify 2.3.0 for version instead of ruby-2.3.0.

    You can enqueue builds for multiple versions at the same time. For example, to build both the Ruby 2.3.0 binary and the Ruby 2.3.1 binary, add the following to the YAML file:

    ---
    ruby:
    - version: 2.3.0
      sha256: ba5ba60e5f1aa21b4ef8e9bf35b9ddb57286cb546aac4b5a28c71f459467e507
    - version: 2.3.1
      sha256: b87c738cb2032bf4920fef8e3864dc5cf8eae9d89d8d523ce0236945c5797dcd
    
  7. Use the git add command to stage your changes:

    $ git add .
  8. Use the git commit -m "YOUR-COMMIT-MESSAGE [#STORY-NUMBER]" command to commit your changes using the Tracker story number. Replace YOUR-COMMIT-MESSAGE with your commit message, and STORY-NUMBER with the number of your Tracker story.

    $ git commit -m "make that change [#1234567890]"
  9. Run git push to push your changes to the remote origin.

    $ git push
  10. Pushing your changes triggers the binary building process, which you can monitor at the binary-builder pipeline of your own buildpacks-ci Concourse deployment. When the build completes, it adds a link to the Concourse build run to the Tracker story specified in the commit message for the new release.

    Note: Binary builds are executed by the Cloud Foundry Binary Builder and the binary-builder pipeline.

Update Buildpack Manifests

After you build the binary for a dependency that you can access and download from a URL, follow the instructions below to add the dependency version to the buildpack manifest.

Note: The steps below assume you are using a Concourse deployment of the buildpacks-ci pipelines and Pivotal Tracker.

  1. Navigate to the directory of the buildpack for which you want to update dependencies and run git checkout develop to check out the develop branch.

    $ cd ~/workspace/ruby-buildpack
    $ git checkout develop
    

  2. Edit the manifest.yml file for the buildpack to add or remove dependencies.

    dependencies:
      - name: ruby
        version: 2.3.0
        md5: 535342030a11abeb11497824bf642bf2
        uri: https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/ruby/ruby-2.3.0-linux-x64.tgz
        cf_stacks:
          - cflinuxfs2
    
    • Follow the current structure of the manifest. For example, if the manifest includes the two most recent patch versions for each minor version of the language, you should also include the two most recent patch versions for each minor version of the language, such as both ruby-2.1.9 and ruby-2.1.8.
    • Copy the uri and the md5 from the build-BINARY-NAME job that ran in the Concourse binary-builder pipeline and add them to the manifest.

      Note: In the PHP buildpack, you may see a modules line for each PHP dependency in the manifest. Do not include this modules line in your new PHP dependency entry. The modules line will be added to the manifest by the ensure-manifest-has-modules Concourse job in the php-buildpack when you commit and push your changes. You can see this in the output logs of the build-out task.

  3. Replace any other mentions of the old version number in the buildpack repository with the new version number. The CF buildpack team uses Ag for text searching.

    $ ag OLD-VERSION
  4. Run the following command to package and upload the buildpack, set up the org and space for tests in the specified CF deployment, and run the CF buildpack tests.

    $ BUNDLE_GEMFILE=cf.Gemfile buildpack-build

    If the command fails, you may need to fix or change the tests, fixtures, or other parts of the buildpack.

  5. Once the test suite completely passes, use git commands to stage, commit, and push your changes:

    $ git add .
    $ git commit -m "YOUR-MESSAGE[#TRACKER-STORY-ID]"
    $ git push
    
  6. Monitor the LANGUAGE-buildpack pipeline in Concourse. Once the test suite builds, the specs-lts-develop job and specs-edge-develop job, pass for the buildpack, you can deliver the Tracker story for the new Dependency release. Copy and paste links for the successful test suite builds into the Tracker story.

Buildpacks

The following list contains information about the buildpacks maintained by the CF buildpacks team.

Go Buildpack

Go:

Godep:

Node.js Buildpack

Node:

  • Verified with: The SHA256 of the node-vVERSION.tar.gz file listed on https://nodejs.org/dist/vVERSION/SHASUMS256.txt For example, for Node version 4.4.6, the CF buildpacks team verifies with the SHA256 for node-v4.4.6.tar.gz on its SHASUMS256 page.
  • Example: Enqueuing binary builds for Node 4.4.5 and 6.2.0

Python Buildpack

Python:

  • Verified with: The MD5 of the Gzipped source tarball, listed on https://www.python.org/downloads/release/python-VERSION/, where VERSION has no periods. For example, for Python version 2.7.12, use the MD5 for the Gzipped source tarball on its downloads page.
  • Example: Enqueuing binary build for Python 2.7.12

Java Buildpack

OpenJDK:

  • Built from: The tarred OpenJDK files managed by the CF Java Buildpack team
  • Verified with: The MD5 of the tarred OpenJDK files

Ruby Buildpack

JRuby:

Ruby:

Bundler:

PHP Buildpack

PHP:

  • Verified with: The SHA256 of the .tar.gz file from the PHP Downloads page.
  • For PHP5 versions, the CF buildpacks team enqueues builds in the php-builds.yml file in the binary-builds branch. For PHP7 versions, the CF buildpacks team enqueues builds in the php7-builds.yml file in the binary-builds branch.
  • Example: Enqueuing binary builds for PHP 5.5.37, 5.6.23, and 7.0.8

Nginx:

  • Verified with: The gpg-rsa-key-id and gpg-signature of the version. The gpg-rsa-key-id is the same for each version/build, but the gpg-signature will be different. This information is located on the Nginx Downloads page.
  • Example: Enqueuing binary build for Nginx 1.11.0

HTTPD:

Composer:

  • Verified with: The SHA256 of the composer.phar file from the Composer Downloads page
  • For Composer, there is no build process as the composer.phar file is the binary. In the manual process, connect to the pivotal-buildpacks S3 bucket using the correct AWS credentials. Create a new directory with the name of the composer version, for example 1.0.2, and put the appropriate composer.phar file into that directory. For Composer v1.0.2, connect and create the php/binaries/trusty/composer/1.0.2 directory. Then place the composer.phar file into that directory so the binary is available at php/binaries/trusty/composer/1.0.2/composer.phar.

    Note: The buildpacks-ci binary-builder pipeline automates the process of detecting, uploading, and updating Composer in the manifest.

  • Example: Automated enqueuing of binary build for Composer 1.1.2

Staticfile Buildpack

Nginx:

  • Verified with: The gpg-rsa-key-id and gpg-signature of the version. The gpg-rsa-key-id is the same for each version/build, but the gpg-signature will be different. This information is located on the Nginx Downloads page.
  • Example: Enqueuing binary build for Nginx 1.11.0

Binary Buildpack

The Binary buildpack has no dependencies.

Create a pull request or raise an issue on the source for this page in GitHub