Skip to content
Support Downloads Contact Us Sign In

Retrieving External Dependencies

Below you will find a reference pipeline that illustrates the tasks and provides an example of a basic pipeline design. You know your environment and constraints and we don't - we recommend you look at the tasks that make up the pipeline, and see how they can be arranged for your specific automation needs. For a deeper dive into each task see the Task Reference.

These Concourse pipelines are examples on how to use the tasks. If you use a different CI/CD platform, you can use these Concourse files as examples of the inputs, outputs, and arguments used in each step in the workflow.

Prerequisites

  • Deployed Concourse

Info

Platform Automation Toolkit is based on Concourse CI. We recommend that you have some familiarity with Concourse before getting started. If you are new to Concourse, Concourse CI Tutorials would be a good place to start.

  • Persisted datastore that can be accessed by Concourse resource (e.g. s3, gcs, minio)
  • A set of valid download-product-config files: Each product has a configuration YAML of what version to download from Tanzu Network.
  • Tanzu Network access to Platform Automation Toolkit

Retrieval from Tanzu Network

Ops Manager 2.5

The filename for the artifact downloaded from Ops Manager is changed! If your resources or pipelines have a regex for the Ops Manager filename, you may be affected. (Please see Ops Manager's official notice for more information)

The pipeline downloads dependencies consumed by the tasks and places them into a trusted s3-like storage provider. This helps other concourse deployments without internet access retrieve task dependencies.

Blobstore filename prefixing

Note the unique regex format for blob names, for example: \[p-healthwatch,(.*)\]p-healthwatch-.*.pivotal. Tanzu Network filenames will not always contain the necessary metadata to accurately download files from a blobstore (i.e. s3, gcs, azure). So, the product slug and version are prepended when using download-product. For more information on how this works, and what to expect when using download-product, refer to the download-product task reference.

The pipeline requires configuration for the download-product task. Below are examples that can be used.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: p-healthwatch

file-glob: "*.pivotal"
product-version-regex: ^1\.8\..*$
stemcell-iaas: google

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: healthwatch-stemcell

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: ops-manager

file-glob: "ops-manager-gcp*.yml"
product-version-regex: 2.9.9 #^2\.\d+\.\d+$|^2\.\d+\.\d+-rc.*$|^2\.\d+\.\d+-alpha.*$

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: pivotal-container-service

file-glob: "pivotal-container-service*.pivotal"
product-version-regex: ^1\.7\..*$
stemcell-iaas: google

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: pks-stemcell

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
---
pivnet-api-token: ((pivnet_token))
pivnet-product-slug: elastic-runtime

file-glob: "srt*.pivotal"
product-version-regex: ^2\.9\.8*$
stemcell-iaas: google

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: tas-stemcell

Full Pipeline and Reference Configurations

There is a git repository containing containing the full pipeline file, along with other pipeline and configuration examples.

This can be useful when you want to take a fully assembled pipeline as a starting point; the rest of this document covers the sections of the full pipeline in more detail.

Pipeline Components

Resource Types

This custom resource type uses the pivnet-resource to pull down and separate both pieces of the Platform Automation Toolkit product (tasks and image) so they can be stored separately in S3.

1
2
3
4
5
6
resource_types:
- name: pivnet
  type: docker-image
  source:
    repository: pivotalcf/pivnet-resource
    tag: latest-final

Product Resources

S3 resources where Platform Automation Toolkit download-product outputs will be stored. Each product/stemcell needs a separate resource defined. Platform Automation Toolkit will not create these resources for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
resources:
- name: opsman-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[ops-manager,(.*)\]ops-manager-gcp.*.yml

- name: pks-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[pivotal-container-service,(.*)\]pivotal-container-service-.*.pivotal

- name: pks-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: pks-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz

- name: tas-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[elastic-runtime,(.*)\]srt-.*.pivotal

- name: tas-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: tas-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz

- name: healthwatch-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[p-healthwatch,(.*)\].*.pivotal

- name: healthwatch-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: healthwatch-stemcell/\[stemcells-ubuntu-xenial,(.*)\]light-bosh-stemcell-.*-google.*\.tgz

Platform Automation Toolkit Resources

platform-automation-pivnet is downloaded directly from Tanzu Network and will be used to download all other products from Tanzu Network.

platform-automation-tasks and platform-automation-image are S3 resources that will be stored for internet-restricted, or faster, access. Platform Automation Toolkit will not create this resource for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
- name: platform-automation-pivnet
  type: pivnet
  source:
    api_token: ((pivnet_token))
    product_slug: platform-automation
    product_version: 2\.(.*)
    sort_by: semver

- name: platform-automation-tasks
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: platform-automation-tasks-(.*).zip

- name: platform-automation-image
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: platform-automation-image-(.*).tgz

Configured Resources

You will need to add your download-product configuration configuration files to your configurations repo. Platform Automation Toolkit will not create these resources for you. For more details, see the Inputs and Outputs section.

1
2
3
4
5
6
7
8
- name: configuration
  type: git
  source:
    private_key: ((docs-ref-pipeline-repo-key.private_key))
    uri: ((docs-ref-pipeline-repo-uri))
    branch: develop
    submodules: all
    depth: 1

Trigger Resources

1
2
3
4
- name: daily
  type: time
  source:
    interval: 24h

Secrets Handling

This helps load secrets stored in an external credential manager -- such as Credhub. Concourse supports several credential managers natively.

The configuration below uses the prepare-tasks-with-secrets task to load secrets from your external configuration files.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# This task is used in multiple jobs
# The yaml anchor "*prepare-tasks-with-secrets" is used in its place
prepare-tasks-with-secrets: &prepare-tasks-with-secrets
  image: platform-automation-image
  file: platform-automation-tasks/tasks/prepare-tasks-with-secrets.yml
  params:
    CONFIG_PATHS: config/download-product-pivnet
  input_mapping:
    tasks: platform-automation-tasks
    config: configuration
  output_mapping:
    tasks: platform-automation-tasks

Jobs

Each job corresponds to a "box" on the visual representation of your Concourse pipeline. These jobs consume resources defined above.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
jobs:
- name: fetch-opsman
  plan:
  - in_parallel:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: configuration
  - task: prepare-tasks-with-secrets
    <<: *prepare-tasks-with-secrets
  - task: download-opsman-image
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product-pivnet/download-opsman.yml
    input_mapping:
      config: configuration
  - in_parallel:
    - put: opsman-product
      params:
        file: downloaded-product/*
- name: fetch-pks
  plan:
  - in_parallel:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: configuration
  - task: prepare-tasks-with-secrets
    <<: *prepare-tasks-with-secrets
  - task: download-pks-product-and-stemcell
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product-pivnet/download-pks.yml
    input_mapping:
      config: configuration
    output_mapping: {downloaded-stemcell: pks-stemcell}
  - in_parallel:
      - put: pks-product
        params:
          file: downloaded-product/*.pivotal
      - put: pks-stemcell
        params:
          file: pks-stemcell/*.tgz

- name: fetch-tas
  plan:
    - in_parallel:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-tas-product-and-stemcell
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-tas.yml
      input_mapping:
        config: configuration
      output_mapping: {downloaded-stemcell: tas-stemcell}
    - in_parallel:
        - put: tas-product
          params:
            file: downloaded-product/*.pivotal
        - put: tas-stemcell
          params:
            file: tas-stemcell/*.tgz

- name: fetch-healthwatch
  plan:
    - in_parallel:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: configuration
    - task: prepare-tasks-with-secrets
      <<: *prepare-tasks-with-secrets
    - task: download-healthwatch-product-and-stemcell
      image: platform-automation-image
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-product-pivnet/download-healthwatch.yml
      input_mapping:
        config: configuration
      output_mapping: {downloaded-stemcell: healthwatch-stemcell}
    - in_parallel:
        - put: healthwatch-product
          params:
            file: downloaded-product/*.pivotal
        - put: healthwatch-stemcell
          params:
            file: healthwatch-stemcell/*.tgz

- name: fetch-platform-automation
  # We use the pivnet resource to bootstrap the pipeline,
  # and because this product is part of the pipeline, not the foundation
  plan:
  - get: platform-automation-pivnet
    trigger: true
  - in_parallel:
    - put: platform-automation-tasks
      params:
        file: platform-automation-pivnet/*tasks*.zip
    - put: platform-automation-image
      params:
        file: platform-automation-pivnet/*image*.tgz