Skip to content
Support Downloads Contact Us Sign In

Ops Manager + a product

Below you will find a reference pipeline that illustrates the tasks and provides an example of a basic pipeline design. You know your environment and constraints and we don't - we recommend you look at the tasks that make up the pipeline, and see how they can be arranged for your specific automation needs. For a deeper dive into each task see the Task Reference.

These Concourse pipelines are examples on how to use the tasks. If you use a different CI/CD platform, you can use these Concourse files as examples of the inputs, outputs, and arguments used in each step in the workflow.

Prerequisites

  • Deployed Concourse

Info

Pivotal Platform Automation is based on Concourse CI. We recommend that you have some familiarity with Concourse before getting started. If you are new to Concourse, Concourse CI Tutorials would be a good place to start.

  • Persisted datastore that can be accessed by Concourse resource (e.g. s3, gcs, minio)
  • A valid generating-env-file: this file will contain credentials necessary to login to Ops Manager using the om CLI. It is used by every task within Pivotal Platform Automation
  • A valid auth-file: this file will contain the credentials necessary to create the Ops Manager login the first time the VM is created. The choices for this file are simple or saml authentication.

Info

There will be some crossover between the auth file and the env file due to how om is setup and how the system works. It is highly recommended to parameterize these values, and let a credential management system (such as Credhub) fill in these values for you in order to maintain consistency across files.

  • An opsman-configuration file: This file is required to connect to an IAAS, and control the lifecycle management of the Ops Manager VM
  • A director-configuration file: Each Ops Manager needs its own configuration, but it is retrieved differently from a product configuration. This config is used to deploy a new Ops Manager director, or update an existing one.
  • A set of valid product-configuration files: Each product configuration is a yaml file that contains the properties necessary to configure an Ops Manager product using the om tool. This can be used during install or update.
  • (Optional) A working credhub setup with its own UAA client and secret.

Retrieving products from Pivnet

Please ensure products have been procured from Pivotal Network using the reference-resources.

Installing Ops Manager and a single product

The pipeline shows how to compose the tasks to install Ops Manager and the Pivotal Container Service product. Its dependencies are coming from a trusted git repository and S3, which can be retrieved using this pipeline.

Pipeline Components

S3 Resources

These can either be uploaded manually or from the reference resources pipeline.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
resources:
  - name: platform-automation-tasks
    type: s3
    source:
      access_key_id: ((s3.access_key_id))
      secret_access_key: ((s3.secret_access_key))
      region_name: ((s3.region_name))
      bucket: ((s3.buckets.pivnet_products))
      regexp: .*tasks-(.*).zip

  - name: platform-automation-image
    type: s3
    source:
      access_key_id: ((s3.access_key_id))
      secret_access_key: ((s3.secret_access_key))
      region_name: ((s3.region_name))
      bucket: ((s3.buckets.pivnet_products))
      regexp: .*image-(.*).tgz

  - name: opsman-image
    type: s3
    source:
      access_key_id: ((s3.access_key_id))
      bucket: ((s3.buckets.pivnet_products))
      region_name: ((s3.region_name))
      secret_access_key: ((s3.secret_access_key))
      regexp: \[ops-manager,(.*)\].*.ova # vsphere ex: ops-manager-(.*).ova

  - name: pks-stemcell
    type: s3
    source:
      access_key_id: ((s3.access_key_id))
      secret_access_key: ((s3.secret_access_key))
      region_name: ((s3.region_name))
      bucket: ((s3.buckets.pivnet_products))
      regexp: pks-stemcell/\[stemcells-ubuntu-xenial,(.*)\]bosh-stemcell-.*-vsphere.*\.tgz

  - name: pks-product
    type: s3
    source:
      access_key_id: ((s3.access_key_id))
      secret_access_key: ((s3.secret_access_key))
      region_name: ((s3.region_name))
      bucket: ((s3.buckets.pivnet_products))
      regexp: \[pivotal-container-service,(.*)\]pivotal-container-service-.*.pivotal

Exported Installation Resource

Warning

It is recommended to persist the zip file exported from export-installation to an external file store (eg S3) on a regular basis. The exported installation can restore the Ops Manager to a working state if it is non-functional.

1
2
3
4
5
6
7
8
- name: installation
  type: s3
  source:
    access_key_id: ((s3.access_key_id))
    secret_access_key: ((s3.secret_access_key))
    region_name: ((s3.region_name))
    bucket: ((s3.buckets.installation))
    regexp: installation-(.*).zip

Configured Resources

These contain values for opsman vm creation, director, product, foundation-specific vars, auth, and env files. For more details, see the Inputs and Outputs section. Platform Automation will not create these resources for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# configurations
- name: configuration
  type: git
  source:
    private_key: ((configuration.private_key))
    uri: ((configuration.uri))
    branch: master

- name: variable
  type: git
  source:
    private_key: ((variable.private_key))
    uri: ((variable.uri))
    branch: master

Trigger Resources

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# triggers used to have jobs do something in a timely manner
- name: one-time-trigger
  type: time
  source:
    interval: 999999h

- name: daily-trigger
  type: time
  source:
    interval: 24h

Credhub Interpolate Job

((foundation)) is a value intended to be replaced by the filepath of your foundation directory structure in github (if you are not using multi-foundation, this value can be removed).

((credhub-*)) are values for accessing your Concourse Credhub. These are set when fly-ing your pipeline. For more information on how to fly your pipeline and use ((foundation)), please reference our How To Guides for your specific workflow. Platform Automation will not create your Credhub or store values into your Credhub for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# This task is used in multiple jobs
# The yaml anchor "*interpolate-creds" is used in its place
interpolate-creds: &interpolate-creds
  image: platform-automation-image
  file: platform-automation-tasks/tasks/credhub-interpolate.yml
  params:
    CREDHUB_CLIENT: ((credhub-client))
    CREDHUB_SECRET: ((credhub-secret))
    CREDHUB_SERVER: ((credhub-server))
    PREFIX: '/pipeline/vsphere-pks'
    INTERPOLATION_PATHS: ((foundation))/config ((foundation))/env
    SKIP_MISSING: true
  input_mapping:
    files: configuration
  output_mapping:
    interpolated-files: interpolated-creds

Jobs

Each job corresponds to a "box" on the visual representation of your Concourse pipeline. These jobs consume resources defined above.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
jobs:
  - name: install-opsman
    serial: true
    serial_groups: [ install ]
    plan:
      - aggregate:
        - get: platform-automation-image
          params:
            unpack: true
        - get: one-time-trigger
          trigger: true
        - get: platform-automation-tasks
          params:
            unpack: true
        - get: configuration
      - task: interpolate-download-opsman-image-creds
        image: platform-automation-image
        file: platform-automation-tasks/tasks/credhub-interpolate.yml
        params:
          CREDHUB_CLIENT: ((credhub-client))
          CREDHUB_SECRET: ((credhub-secret))
          CREDHUB_SERVER: ((credhub-server))
          PREFIX: '/pipeline/vsphere'
          INTERPOLATION_PATHS: "download-product-configs"
          SKIP_MISSING: true
        input_mapping:
          files: configuration
        output_mapping:
          interpolated-files: product-configs
      - task: download-opsman-image
        image: platform-automation-image
        file: platform-automation-tasks/tasks/download-product-s3.yml
        params:
          CONFIG_FILE: download-product-configs/opsman.yml
        input_mapping:
          config: product-configs
        output_mapping:
          downloaded-product: opsman-image
      - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: create-vm
        image: platform-automation-image
        file: platform-automation-tasks/tasks/create-vm.yml
        input_mapping:
          image: opsman-image
          state: configuration
          config: interpolated-creds
          vars: variable
        params:
          VARS_FILES: vars/((foundation))/vars/opsman-vars.yml
          STATE_FILE: ((foundation))/state/state.yml
          OPSMAN_CONFIG_FILE: ((foundation))/config/opsman.yml
        ensure: &make-state-commit
          do:
            - task: make-commit
              image: platform-automation-image
              file: platform-automation-tasks/tasks/make-git-commit.yml
              input_mapping:
                repository: configuration
                file-source: generated-state
              output_mapping:
                repository-commit: configuration-commit
              params:
                FILE_SOURCE_PATH: state.yml
                FILE_DESTINATION_PATH: ((foundation))/state/state.yml
                GIT_AUTHOR_EMAIL: "pcf-pipeline-bot@example.com"
                GIT_AUTHOR_NAME: "Platform Automation Bot"
                COMMIT_MESSAGE: 'Update state file'
            - put: configuration
              params:
                repository: configuration-commit
                merge: true
      - task: configure-authentication
        image: platform-automation-image
        file: platform-automation-tasks/tasks/configure-authentication.yml
        attempts: 10
        input_mapping:
          env: configuration
          config: interpolated-creds
        params:
          ENV_FILE: ((foundation))/env/env.yml
          AUTH_CONFIG_FILE: ((foundation))/config/auth.yml
      - task: configure-director
        image: platform-automation-image
        file: platform-automation-tasks/tasks/configure-director.yml
        input_mapping:
          config: interpolated-creds
          env: interpolated-creds
          vars: variable
        params:
          VARS_FILES: vars/((foundation))/vars/director-vars.yml
          ENV_FILE: ((foundation))/env/env.yml
          DIRECTOR_CONFIG_FILE: ((foundation))/config/director.yml
      - task: apply-director-changes
        image: platform-automation-image
        file: platform-automation-tasks/tasks/apply-director-changes.yml
        input_mapping:
          env: interpolated-creds
        params:
          ENV_FILE: ((foundation))/env/env.yml

  - name: export-installation
    serial: true
    plan:
      - aggregate:
          - get: daily-trigger
            trigger: true
          - get: platform-automation-image
            params:
              unpack: true
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: export-installation
        image: platform-automation-image
        file: platform-automation-tasks/tasks/export-installation.yml
        input_mapping:
          env: interpolated-creds
        params:
          ENV_FILE: ((foundation))/env/env.yml
          INSTALLATION_FILE: installation-$timestamp.zip
      - put: installation
        params:
          file: installation/installation*.zip

  - name: upgrade-opsman
    serial: true
    serial_groups: [ install ]
    plan:
      - aggregate:
          - get: one-time-trigger
            passed: [ install-opsman ]
          - get: platform-automation-image
            params:
              unpack: true
            trigger: true
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: opsman-image
          - get: installation
            passed: [ export-installation ]
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: upgrade-opsman
        image: platform-automation-image
        file: platform-automation-tasks/tasks/upgrade-opsman.yml
        input_mapping:
          image: opsman-image
          state: configuration
          config: interpolated-creds
          env: interpolated-creds
          vars: variable
        params:
          VARS_FILES: vars/((foundation))/vars/opsman-vars.yml
          ENV_FILE: ((foundation))/env/env.yml
          OPSMAN_CONFIG_FILE: ((foundation))/config/opsman.yml
          STATE_FILE: ((foundation))/state/state.yml
        ensure: *make-state-commit
      - task: apply-director-changes
        image: platform-automation-image
        file: platform-automation-tasks/tasks/apply-director-changes.yml
        input_mapping:
          env: interpolated-creds
        params:
          ENV_FILE: ((foundation))/env/env.yml

  - name: upload-stemcells
    serial: true
    plan:
      - aggregate:
          - get: platform-automation-image
            params:
              unpack: true
            passed:
              - upgrade-opsman
            trigger: true
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: pks-stemcell
            trigger: true
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: upload-pks-stemcell
        image: platform-automation-image
        file: platform-automation-tasks/tasks/upload-stemcell.yml
        input_mapping:
          env: interpolated-creds
          stemcell: pks-stemcell
        params:
          ENV_FILE: ((foundation))/env/env.yml

  - name: upload-and-stage-pks
    serial: true
    plan:
      - aggregate:
          - get: platform-automation-image
            params:
              unpack: true
            trigger: true
            passed: [ "upgrade-opsman" ]
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: pks-product
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: upload-and-stage-product
        image: platform-automation-image
        file: platform-automation-tasks/tasks/upload-and-stage-product.yml
        input_mapping:
          product: pks-product
          env: interpolated-creds
        params:
          ENV_FILE: ((foundation))/env/env.yml

  - name: configure-pks
    serial: true
    plan:
      - aggregate:
          - get: platform-automation-image
            params:
              unpack: true
            passed:
              - upload-and-stage-pks
            trigger: true
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: configure-pks
        image: platform-automation-image
        file: platform-automation-tasks/tasks/configure-product.yml
        input_mapping:
          config: interpolated-creds
          env: interpolated-creds
          vars: variable
        params:
          CONFIG_FILE: ((foundation))/config/pks.yml
          ENV_FILE: ((foundation))/env/env.yml

  - name: apply-product-changes
    serial: true
    plan:
      - aggregate:
          - get: platform-automation-image
            params:
              unpack: true
            passed:
              - configure-pks
            trigger: true
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: apply-product-changes
        image: platform-automation-image
        file: platform-automation-tasks/tasks/apply-changes.yml
        input_mapping:
          env: interpolated-creds
        params:
          ENV_FILE: ((foundation))/env/env.yml

  - name: staged-pks-config
    plan:
      - aggregate:
          - get: platform-automation-image
            params:
              unpack: true
          - get: platform-automation-tasks
            params:
              unpack: true
          - get: configuration
          - get: variable
      - task: interpolate-creds
        <<: *interpolate-creds
      - task: staged-config
        image: platform-automation-image
        file: platform-automation-tasks/tasks/staged-config.yml
        input_mapping:
          env: interpolated-creds
        params:
          PRODUCT_NAME: pivotal-container-service
          ENV_FILE: ((foundation))/env/env.yml