Skip to content
Support Downloads Contact Us Sign In

Retrieving External Dependencies

Below you will find a reference pipeline that illustrates the tasks and provides an example of a basic pipeline design. You know your environment and constraints and we don't - we recommend you look at the tasks that make up the pipeline, and see how they can be arranged for your specific automation needs. For a deeper dive into each task see the Task Reference.

These Concourse pipelines are examples on how to use the tasks. If you use a different CI/CD platform, you can use these Concourse files as examples of the inputs, outputs, and arguments used in each step in the workflow.

Prerequisites

  • Deployed Concourse

Info

Pivotal Platform Automation is based on Concourse CI. We recommend that you have some familiarity with Concourse before getting started. If you are new to Concourse, Concourse CI Tutorials would be a good place to start.

  • Persisted datastore that can be accessed by Concourse resource (e.g. s3, gcs, minio)
  • Pivnet access to Platform Automation
  • A set of valid download-product-config files: Each product has a configuration YAML of what version to download from Pivotal Network.

Retrieval from Pivotal Network

Ops Manager 2.5

The filename for the artifact downloaded from Ops Manager is changed! If your resources or pipelines have a regex for the Ops Manager filename, you may be affected. (Please see Ops Manager's official notice for more information)

The pipeline downloads dependencies consumed by the tasks and places them into a trusted s3-like storage provider. This helps other concourse deployments without internet access retrieve task dependencies.

S3 filename prefixing

Note the unique regex format for blob names, for example: \[p-healthwatch,(.*)\]p-healthwatch-.*.pivotal. Pivnet filenames will not always contain the necessary metadata to accurately download files from S3. So, the product slug and version are prepended when using download-product. For more information on how this works, and what to expect when using download-product and download-product-s3, refer to the download-product task reference.

The pipeline requires configuration for the download-product task. Below are examples that can be used.

``` yaml tab="Healthwatch"

pivnet-api-token: ((pivnet-token)) pivnet-file-glob: "*.pivotal" pivnet-product-slug: p-healthwatch product-version: 1.4.4 stemcell-iaas: vsphere

s3-access-key-id: ((s3_access_key_id)) s3-secret-access-key: ((s3_secret_access_key)) s3-bucket: ((s3_pivnet_products_bucket)) s3-region-name: ((s3_region_name)) s3-stemcell-path: healthwatch-stemcell 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
``` yaml tab="PAS"
---
pivnet-api-token: ((pivnet-token))
pivnet-file-glob: "*cf*.pivotal"
pivnet-product-slug: elastic-runtime
product-version-regex: ^2\.[56]\.0.*$
stemcell-iaas: vsphere

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))
s3-stemcell-path: pas-stemcell

``` yaml tab="PAS Windows"

pivnet-api-token: ((pivnet-token)) pivnet-file-glob: ".pivotal" pivnet-product-slug: pas-windows product-version-regex: ^2.[56].0.$

s3-access-key-id: ((s3_access_key_id)) s3-secret-access-key: ((s3_secret_access_key)) s3-bucket: ((s3_pivnet_products_bucket)) s3-region-name: ((s3_region_name)) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
``` yaml tab="OpsMan"
---
pivnet-api-token: ((pivnet-token))
pivnet-file-glob: "ops-manager*.ova"
pivnet-product-slug: ops-manager
product-version-regex: ^2\.[56]\.\d+.*$

s3-access-key-id: ((s3_access_key_id))
s3-secret-access-key: ((s3_secret_access_key))
s3-bucket: ((s3_pivnet_products_bucket))
s3-region-name: ((s3_region_name))

``` yaml tab="PKS"

pivnet-api-token: ((pivnet-token)) pivnet-file-glob: "pivotal-container-service.pivotal" pivnet-product-slug: pivotal-container-service product-version-regex: ^1.3..$ stemcell-iaas: vsphere

s3-access-key-id: ((s3_access_key_id)) s3-secret-access-key: ((s3_secret_access_key)) s3-bucket: ((s3_pivnet_products_bucket)) s3-region-name: ((s3_region_name)) 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
## Pipeline Components

### Resource Types

This custom resource type uses the pivnet resource
to pull down and separate both pieces of the Platform Automation product (tasks and image)
so they can be stored separately in S3.



```yaml
resource_types:
- name: pivnet
  type: docker-image
  source:
    repository: pivotalcf/pivnet-resource
    tag: latest-final

Product Resources

S3 resources where Platform Automation download-product outputs will be stored. Each product/stemcell needs a separate resource defined. Platform Automation will not create these resources for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
resources:
- name: healthwatch-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[p-healthwatch,(.*)\]p-healthwatch-.*.pivotal

- name: healthwatch-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: healthwatch-stemcell/\[stemcells-ubuntu-xenial,(.*)\]bosh-stemcell-.*-vsphere.*\.tgz

- name: opsman-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[ops-manager,(.*)\].*.ova

- name: pas-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[elastic-runtime,(.*)\]cf-.*.pivotal

- name: pas-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: pas-stemcell/\[stemcells-ubuntu-xenial,(.*)\]bosh-stemcell-.*-vsphere.*\.tgz

- name: pks-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[pivotal-container-service,(.*)\]pivotal-container-service-.*.pivotal

- name: pks-stemcell
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: pks-stemcell/\[stemcells-ubuntu-xenial,(.*)\]bosh-stemcell-.*-vsphere.*\.tgz

- name: pas-windows-product
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: \[pas-windows,(.*)\]pas-windows-.*.pivotal

Platform Automation Resources

platform-automation-pivnet is downloaded directly from Pivnet and will be used to download all other products from Pivnet.

platform-automation-tasks and platform-automation-image are S3 resources that will be stored for internet-restricted, or faster, access. Platform Automation will not create this resource for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
- name: platform-automation-pivnet
  type: pivnet
  source:
    api_token: ((pivnet_token))
    product_slug: platform-automation
    product_version: 4.(.*)
    sort_by: semver

- name: platform-automation-tasks
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: platform-automation-tasks-(.*).zip

- name: platform-automation-image
  type: s3
  source:
    access_key_id: ((s3_access_key_id))
    bucket: ((s3_pivnet_products_bucket))
    region_name: ((s3_region_name))
    secret_access_key: ((s3_secret_access_key))
    regexp: platform-automation-image-(.*).tgz

Configured Resources

You will need to add your download-product configuration configuration files to your configurations repo. Platform Automation will not create these resources for you. For more details, see the Inputs and Outputs section.

1
2
3
4
5
6
7
8
- name: config
  type: git
  source:
    private_key: ((docs-ref-pipeline-repo-key.private_key))
    uri: ((docs-ref-pipeline-repo-uri))
    branch: master
    submodules: all
    depth: 1

Trigger Resources

1
2
3
4
- name: daily
  type: time
  source:
    interval: 24h

Credhub Interpolate Job

((foundation)) is a value intended to be replaced by the filepath of your foundation directory structure in github (if you are not using multi-foundation, this value can be removed).

((credhub-*)) are values for accessing your Concourse Credhub. These are set when fly-ing your pipeline. For more information on how to fly your pipeline and use ((foundation)), please reference our How To Guides for your specific workflow. Platform Automation will not create your Credhub or store values into your Credhub for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# This task is used in multiple jobs
# The yaml anchor "*credhub-interpolate" is used in its place
credhub-interpolate: &credhub-interpolate
  image: platform-automation-image
  file: platform-automation-tasks/tasks/credhub-interpolate.yml
  params:
    CREDHUB_CLIENT: ((credhub-client))
    CREDHUB_SECRET: ((credhub-secret))
    CREDHUB_SERVER: ((credhub-server))
    CREDHUB_CA_CERT: ((credhub-ca-cert))
    PREFIX: '/pipeline/vsphere'
    INTERPOLATION_PATHS: "download-product"
  input_mapping:
    files: config
  output_mapping:
    interpolated-files: config

Jobs

Each job corresponds to a "box" on the visual representation of your Concourse pipeline. These jobs consume resources defined above.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
jobs:
- name: fetch-healthwatch
  plan:
  - aggregate:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: config
  - task: credhub-interpolate
    <<: *credhub-interpolate
  - task: download-healthwatch-product-and-stemcell
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product/healthwatch.yml
    output_mapping: {downloaded-stemcell: healthwatch-stemcell}
  - aggregate:
    - put: healthwatch-product
      params:
        file: downloaded-product/*.pivotal
    - put: healthwatch-stemcell
      params:
        file: healthwatch-stemcell/*.tgz

- name: fetch-opsman
  plan:
  - aggregate:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: config
  - task: credhub-interpolate
    <<: *credhub-interpolate
  - task: download-opsman-image
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product/opsman.yml
  - aggregate:
    - put: opsman-product
      params:
        file: downloaded-product/*

- name: fetch-pas
  plan:
  - aggregate:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: config
  - task: credhub-interpolate
    <<: *credhub-interpolate
  - task: download-pas-product-and-stemcell
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product/pas.yml
    output_mapping: {downloaded-stemcell: pas-stemcell}
  - aggregate:
    - put: pas-product
      params:
        file: downloaded-product/*.pivotal
    - put: pas-stemcell
      params:
        file: pas-stemcell/*.tgz

- name: fetch-pas-windows
  plan:
  - aggregate:
    - get: daily
      trigger: true
    - get: platform-automation-image
      params:
        unpack: true
    - get: platform-automation-tasks
      params:
        unpack: true
    - get: config
  - task: credhub-interpolate
    <<: *credhub-interpolate
  - task: download-pas-windows-product
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product/pas-windows.yml
  - aggregate:
    - put: pas-windows-product
      params:
        file: downloaded-product/*.pivotal

- name: fetch-pks
  plan:
  - aggregate:
      - get: daily
        trigger: true
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: config
  - task: credhub-interpolate
    <<: *credhub-interpolate
  - task: download-pks-product-and-stemcell
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product/pks.yml
    output_mapping: {downloaded-stemcell: pks-stemcell}
  - aggregate:
      - put: pks-product
        params:
          file: downloaded-product/*.pivotal
      - put: pks-stemcell
        params:
          file: pks-stemcell/*.tgz

- name: fetch-platform-automation
  # We use the pivnet resource to bootstrap the pipeline,
  # and because this product is part of the pipeline, not the foundation
  plan:
  - get: platform-automation-pivnet
    trigger: true
  - aggregate:
    - put: platform-automation-tasks
      params:
        file: platform-automation-pivnet/*tasks*.zip
    - put: platform-automation-image
      params:
        file: platform-automation-pivnet/*image*.tgz