Ops Manager + multiple products
Below you will find a reference pipeline that illustrates the tasks and provides an example of a basic pipeline design. You know your environment and constraints and we don't - we recommend you look at the tasks that make up the pipeline, and see how they can be arranged for your specific automation needs. For a deeper dive into each task see the Task Reference.
These Concourse pipelines are examples on how to use the tasks . If you use a different CI/CD platform, you can use these Concourse files as examples of the inputs, outputs, and arguments used in each step in the workflow.
Prerequisites
Info
Pivotal Platform Automation is based on Concourse CI.
We recommend that you have some familiarity with Concourse before getting started.
If you are new to Concourse, Concourse CI Tutorials would be a good place to start.
Persisted datastore that can be accessed by Concourse resource (e.g. s3, gcs, minio)
A valid generating-env-file : this file will contain credentials necessary to login to Ops Manager using the om
CLI.
It is used by every task within Pivotal Platform Automation
A valid auth-file : this file will contain the credentials necessary to create the Ops Manager login the first time
the VM is created. The choices for this file are simple or saml authentication.
Info
There will be some crossover between the auth file and the env file due to how om is setup and how the system works. It is highly recommended to parameterize these values, and let a credential management system (such as Credhub) fill in these values for you in order to maintain consistency across files.
An opsman-configuration file: This file is required to connect to an IAAS, and control the lifecycle management
of the Ops Manager VM
A director-configuration file: Each Ops Manager needs its own configuration, but it is retrieved differently from
a product configuration. This config is used to deploy a new Ops Manager director, or update an existing one.
A set of valid product-configuration files: Each product configuration is a yaml file that contains the properties
necessary to configure an Ops Manager product using the om
tool. This can be used during install or update.
(Optional) A working credhub setup with its own UAA client and secret.
Retrieving products from Pivnet
Please ensure products have been procured from Pivotal Network using the reference-resources .
Installing Ops Manager and multiple products
The pipeline shows how to compose the tasks
to install Ops Manager and the Pivotal Application Service and Healthwatch products.
Its dependencies are coming from a trusted git repository,
which can be retrieved using this pipeline .
Pipeline Components
S3 Resources
These can either be uploaded manually or from the reference resources pipeline .
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52 resource_types :
- name : pivnet
type : docker-image
source :
repository : pivotalcf/pivnet-resource
tag : latest-final
resources :
- name : telemetry-collector-binary
type : s3
source :
access_key_id : ((s3_access_key_id))
secret_access_key : ((s3_secret_access_key))
region_name : ((s3_region_name))
bucket : ((s3_pivnet_products_bucket))
regexp : .*telemetry-(.*).tgz
- name : platform-automation-tasks
type : s3
source :
access_key_id : ((s3_access_key_id))
secret_access_key : ((s3_secret_access_key))
region_name : ((s3_region_name))
bucket : ((s3_pivnet_products_bucket))
regexp : .*tasks-(.*).zip
- name : platform-automation-image
type : s3
source :
access_key_id : ((s3_access_key_id))
secret_access_key : ((s3_secret_access_key))
region_name : ((s3_region_name))
bucket : ((s3_pivnet_products_bucket))
regexp : .*image-(.*).tgz
- name : pas-windows-product
type : s3
source :
access_key_id : ((s3_access_key_id))
bucket : ((s3_pivnet_products_bucket))
region_name : ((s3_region_name))
secret_access_key : ((s3_secret_access_key))
regexp : \[pas-windows,(.*)\]pas-windows-.*.pivotal
- name : pas-windows-stemcell
type : s3
source :
access_key_id : ((s3_access_key_id))
bucket : ((s3_pivnet_products_bucket))
region_name : ((s3_region_name))
secret_access_key : ((s3_secret_access_key))
regexp : pas-windows-stemcell/\[stemcells-windows-server,(.*)\]bosh-stemcell-.*-vsphere.*\.tgz
Pivotal Application Service-Windows with S3
If retrieving pas-windows
and pas-windows-stemcell
from an S3 bucket,
you must use the built in S3 concourse resource.
This is done in the example above.
The download-product-s3
task does not persist meta information
about necessary stemcell for pas-windows
because Pivotal does not distribute the Window's file system.
Alternatively, products may be downloaded using the download-product-s3
task.
In a job, specify the following task:
1
2
3
4
5
6
7
8
9
10
11
12 ...
- task : download-pas
image : platform-automation-image
file : platform-automation-tasks/tasks/download-product-s3.yml
params :
CONFIG_FILE : download-product-configs/pas.yml
input_mapping :
config : interpolated-creds
output_mapping :
downloaded-product : pas-product
downloaded-stemcell : pas-stemcell
...
Exported Installation Resource
Always Export your Installation
It is recommended to persist the zip file exported from export-installation
to an external file store (eg S3) on a regular basis.
The exported installation can restore the Ops Manager
to a working state if it is non-functional.
- name : installation
type : s3
source :
access_key_id : ((s3_access_key_id))
secret_access_key : ((s3_secret_access_key))
region_name : ((s3_region_name))
bucket : ((s3_installation_bucket))
regexp : installation-(.*).zip
These contain values for
opsman vm creation, director, product, foundation-specific vars, auth, and env files.
For more details, see the Inputs and Outputs section.
Platform Automation will not create these resources for you.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16 # VM state and foundation configuration
- name : state
type : s3
source :
access_key_id : ((s3_access_key_id))
bucket : ((s3_foundation_state_bucket))
region_name : ((s3_region_name))
secret_access_key : ((s3_secret_access_key))
versioned_file : state.yml
- name : configuration
type : git
source :
private_key : ((docs-ref-pipeline-repo-key.private_key))
uri : ((docs-ref-pipeline-repo-uri))
branch : main
Trigger Resources
# triggers used to have jobs do something in a timely manner
- name : one-time-trigger
type : time
source :
interval : 999999h
- name : daily-trigger
type : time
source :
interval : 24h
Credhub Interpolate Job
((foundation))
is a value
intended to be replaced by the filepath
of your foundation directory structure in github
(if you are not using multi-foundation, this value can be removed).
((credhub-*))
are values for accessing your Concourse Credhub.
These are set when fly
-ing your pipeline.
For more information on how to fly your pipeline
and use ((foundation))
,
please reference our How To Guides for your specific workflow.
Platform Automation will not create your Credhub or store values into your Credhub for you.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17 # This task is used in multiple jobs
# The yaml anchor "*interpolate-creds" is used in its place
interpolate-creds : &interpolate-creds
image : platform-automation-image
file : platform-automation-tasks/tasks/credhub-interpolate.yml
params :
CREDHUB_CLIENT : ((credhub-client))
CREDHUB_SECRET : ((credhub-secret))
CREDHUB_SERVER : ((credhub-server))
CREDHUB_CA_CERT : ((credhub-ca-cert))
PREFIX : '/concourse/main/reference-pipeline'
INTERPOLATION_PATHS : .
# SKIP_MISSING: true
input_mapping :
files : configuration
output_mapping :
interpolated-files : interpolated-creds
Jobs
Each job corresponds to a "box"
on the visual representation of your Concourse pipeline.
These jobs consume resources defined above.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554 jobs :
- name : test-platform-automation
serial : true
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : test-interpolate
image : platform-automation-image
file : platform-automation-tasks/tasks/test-interpolate.yml
params :
CONFIG_FILE : download-product/pas.yml
SKIP_MISSING : true
input_mapping :
config : configuration
- task : test
file : platform-automation-tasks/tasks/test.yml
image : platform-automation-image
- name : delete-opsman
serial : true
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : delete-installation
image : platform-automation-image
file : platform-automation-tasks/tasks/delete-installation.yml
input_mapping :
env : interpolated-creds
- task : delete-vm
image : platform-automation-image
file : platform-automation-tasks/tasks/delete-vm.yml
input_mapping :
config : interpolated-creds
ensure : &put-state
do :
- put : state
params :
file : generated-state/state.yml
- name : install-opsman
serial : true
serial_groups : [ install ]
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
- get : one-time-trigger
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : download-opsman-image
image : platform-automation-image
file : platform-automation-tasks/tasks/download-product-s3.yml
params :
CONFIG_FILE : download-product/opsman.yml
input_mapping :
config : interpolated-creds
output_mapping :
downloaded-product : opsman-image
- task : create-vm
image : platform-automation-image
file : platform-automation-tasks/tasks/create-vm.yml
input_mapping :
image : opsman-image
config : interpolated-creds
ensure : *put-state
- task : configure-authentication
image : platform-automation-image
file : platform-automation-tasks/tasks/configure-authentication.yml
attempts : 10
input_mapping :
env : interpolated-creds
config : interpolated-creds
- task : configure-director
image : platform-automation-image
file : platform-automation-tasks/tasks/configure-director.yml
input_mapping :
config : interpolated-creds
env : interpolated-creds
- task : apply-director-changes
image : platform-automation-image
file : platform-automation-tasks/tasks/apply-director-changes.yml
input_mapping :
env : interpolated-creds
- task : export-installation
image : platform-automation-image
file : platform-automation-tasks/tasks/export-installation.yml
input_mapping :
env : interpolated-creds
params :
INSTALLATION_FILE : installation-$timestamp.zip
- put : installation
params :
file : installation/installation*.zip
- name : staged-director-config
serial : true
plan :
- aggregate :
- get : platform-automation-tasks
params : { unpack : true }
- get : platform-automation-image
params : { unpack : true }
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : staged-director-config
image : platform-automation-image
file : platform-automation-tasks/tasks/staged-director-config.yml
input_mapping :
env : interpolated-creds
ensure : *put-state
- name : export-installation
serial : true
plan :
- aggregate :
- get : daily-trigger
trigger : true
- get : platform-automation-image
params :
unpack : true
passed : [ install-opsman ]
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : one-time-trigger
passed : [ install-opsman ]
- task : interpolate-creds
<< : *interpolate-creds
- task : export-installation
image : platform-automation-image
file : platform-automation-tasks/tasks/export-installation.yml
input_mapping :
env : interpolated-creds
params :
INSTALLATION_FILE : installation-$timestamp.zip
- put : installation
params :
file : installation/installation*.zip
- name : upgrade-opsman
serial : true
serial_groups : [ install ]
plan :
- aggregate :
- get : one-time-trigger
passed : [ export-installation ]
- get : platform-automation-image
params :
unpack : true
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : installation
passed : [ export-installation ]
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : download-opsman-image
image : platform-automation-image
file : platform-automation-tasks/tasks/download-product-s3.yml
params :
CONFIG_FILE : download-product/opsman.yml
input_mapping :
config : interpolated-creds
output_mapping :
downloaded-product : opsman-image
- task : upgrade-opsman
image : platform-automation-image
file : platform-automation-tasks/tasks/upgrade-opsman.yml
input_mapping :
image : opsman-image
config : interpolated-creds
env : interpolated-creds
ensure : *put-state
- task : configure-director
image : platform-automation-image
file : platform-automation-tasks/tasks/configure-director.yml
input_mapping :
config : interpolated-creds
env : interpolated-creds
- task : apply-director-changes
image : platform-automation-image
file : platform-automation-tasks/tasks/apply-director-changes.yml
input_mapping :
env : interpolated-creds
- task : export-installation
image : platform-automation-image
file : platform-automation-tasks/tasks/export-installation.yml
input_mapping :
env : interpolated-creds
params :
INSTALLATION_FILE : installation-$timestamp.zip
- put : installation
params :
file : installation/installation*.zip
- name : upload-and-stage-pas
serial : true
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
passed :
- upgrade-opsman
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : download-pas
image : platform-automation-image
file : platform-automation-tasks/tasks/download-product-s3.yml
params :
CONFIG_FILE : download-product/pas.yml
input_mapping :
config : interpolated-creds
output_mapping :
downloaded-product : pas-product
downloaded-stemcell : pas-stemcell
- task : upload-product
image : platform-automation-image
file : platform-automation-tasks/tasks/upload-product.yml
input_mapping :
product : pas-product
env : interpolated-creds
- task : upload-pas-stemcell
image : platform-automation-image
file : platform-automation-tasks/tasks/upload-stemcell.yml
input_mapping :
env : interpolated-creds
stemcell : pas-stemcell
- task : stage-product
image : platform-automation-image
file : platform-automation-tasks/tasks/stage-product.yml
input_mapping :
product : pas-product
env : interpolated-creds
- name : configure-pas
serial : true
serial_groups : [ install ]
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
passed :
- upload-and-stage-pas
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : configure-pas
image : platform-automation-image
file : platform-automation-tasks/tasks/configure-product.yml
input_mapping :
config : interpolated-creds
env : interpolated-creds
params :
CONFIG_FILE : pas.yml
- name : upload-and-stage-pas-windows
serial : true
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
passed :
- upgrade-opsman
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : pas-windows-product
- get : pas-windows-stemcell
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : upload-product
image : platform-automation-image
file : platform-automation-tasks/tasks/upload-product.yml
input_mapping :
product : pas-windows-product
env : interpolated-creds
- task : upload-pas-windows-stemcell
image : platform-automation-image
file : platform-automation-tasks/tasks/upload-stemcell.yml
input_mapping :
env : interpolated-creds
stemcell : pas-windows-stemcell
- task : stage-product
image : platform-automation-image
file : platform-automation-tasks/tasks/stage-product.yml
input_mapping :
product : pas-windows-product
env : interpolated-creds
- name : configure-pas-windows
serial : true
serial_groups : [ install ]
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
passed :
- upload-and-stage-pas-windows
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : configure-pas
image : platform-automation-image
file : platform-automation-tasks/tasks/configure-product.yml
input_mapping :
config : interpolated-creds
env : interpolated-creds
params :
CONFIG_FILE : pas-windows.yml
- name : upload-and-stage-healthwatch
serial : true
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
trigger : true
passed : [ "upgrade-opsman" ]
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : download-healthwatch
image : platform-automation-image
file : platform-automation-tasks/tasks/download-product-s3.yml
params :
CONFIG_FILE : download-product/healthwatch.yml
input_mapping :
config : interpolated-creds
output_mapping :
downloaded-product : healthwatch-product
downloaded-stemcell : healthwatch-stemcell
- task : upload-and-stage-product
image : platform-automation-image
file : platform-automation-tasks/tasks/upload-and-stage-product.yml
input_mapping :
product : healthwatch-product
env : interpolated-creds
- task : upload-healthwatch-stemcell
image : platform-automation-image
file : platform-automation-tasks/tasks/upload-stemcell.yml
input_mapping :
env : interpolated-creds
stemcell : healthwatch-stemcell
- name : configure-healthwatch
serial : true
serial_groups : [ install ]
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
passed :
- upload-and-stage-healthwatch
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : configure-healthwatch
image : platform-automation-image
file : platform-automation-tasks/tasks/configure-product.yml
input_mapping :
config : interpolated-creds
env : interpolated-creds
params :
CONFIG_FILE : healthwatch.yml
- name : apply-product-changes
serial : true
serial_groups : [ telemetry , install ]
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
passed :
- configure-healthwatch
- configure-pas
- configure-pas-windows
trigger : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- task : interpolate-creds
<< : *interpolate-creds
- task : pre-deploy-check
image : platform-automation-image
file : platform-automation-tasks/tasks/pre-deploy-check.yml
input_mapping :
env : interpolated-creds
- task : apply-product-changes
image : platform-automation-image
file : platform-automation-tasks/tasks/apply-changes.yml
input_mapping :
env : interpolated-creds
- task : export-installation
image : platform-automation-image
file : platform-automation-tasks/tasks/export-installation.yml
input_mapping :
env : interpolated-creds
params :
INSTALLATION_FILE : installation-$timestamp.zip
- put : installation
params :
file : installation/installation*.zip
- name : staged-pas-config
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : staged-config
image : platform-automation-image
file : platform-automation-tasks/tasks/staged-config.yml
input_mapping :
env : interpolated-creds
params :
PRODUCT_NAME : cf
ensure : *put-state
- name : expiring-certificates
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : expiring-certificates
image : platform-automation-image
file : platform-automation-tasks/tasks/expiring-certificates.yml
input_mapping :
env : interpolated-creds
params :
EXPIRES_WITHIN : 6m
- name : staged-healthwatch-config
plan :
- aggregate :
- get : platform-automation-image
params :
unpack : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : state
- task : interpolate-creds
<< : *interpolate-creds
- task : staged-config
image : platform-automation-image
file : platform-automation-tasks/tasks/staged-config.yml
input_mapping :
env : interpolated-creds
params :
PRODUCT_NAME : p-healthwatch
ensure : *put-state
- name : collect-telemetry
serial : true
serial_groups : [ telemetry ]
plan :
- aggregate :
- get : telemetry-collector-binary
params :
unpack : true
- get : platform-automation-image
params :
unpack : true
- get : platform-automation-tasks
params :
unpack : true
- get : configuration
- get : daily-trigger
trigger : true
- task : interpolate-creds
<< : *interpolate-creds
- task : collect-telemetry-data
image : platform-automation-image
file : platform-automation-tasks/tasks/collect-telemetry.yml
params :
CONFIG_FILE : telemetry.yml
input_mapping :
env : interpolated-creds
config : configuration
- task : send-telemetry-data
image : platform-automation-image
file : platform-automation-tasks/tasks/send-telemetry.yml
params :
API_KEY : no-op-test-key
DATA_FILE_PATH : collected-telemetry-data/FoundationDetails*.tar