Skip to content
Support Downloads Contact Us Sign In

Configuration Management Strategies

When building pipelines, there are many possible strategies for structuring your configuration in source control as well as in pipeline design. No single method can cover all situations. After reading this document, we hope you feel equipped to select an approach.

Single Repository for Each Foundation

This is the simplest thing that could possibly work. It's the default assumed in all our examples, unless we've articulated a specific reason to choose a different approach. It entails using a single Git repository for each foundation.

Tracking foundation changes are simple, getting started is easy, duplicating foundations is simply a matter of cloning a repository, and configuration files are not difficult to understand.

This is the strategy used throughout the Install Ops Man How to Guide and the Upgrading an Existing Ops Manager How to Guide.

Let's examine an example configuration repository that uses the "Single Repository for each Foundation" pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
├── auth.yml
├── pas.yml
├── director.yml
├── download-opsman.yml
├── download-product-configs
│   ├── healthwatch.yml
│   ├── opsman.yml
│   ├── pas-windows.yml
│   ├── pas.yml
│   └── telemetry.yml
├── env.yml
├── healthwatch.yml
├── opsman.yml
└── pas-windows.yml

Notice that there is only one subdirectory and that all other files are at the repositories base directory. This minimizes parameter mapping in the platform-automation tasks. For example, in the configure-director step:

1
2
3
4
5
6
- task: configure-director
  image: platform-automation-image
  file: platform-automation-tasks/tasks/configure-director.yml
  input_mapping:
    config: configuration
    env: configuration

We map the config files to the expected input named env of the configure-director task. Because the configure-director task's default ENV parameter is env.yml, it automatically uses the env.yml file in our configuration repo. We do not need to explicitly name the ENV parameter for the task. This also works for director.yml.

Another option for mapping resources to inputs is discussed in the Matching Resource Names and Input Names section.

For reference, here is the configure-director task:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
---
platform: linux

inputs:
- name: platform-automation-tasks
- name: config # contains the director configuration file
- name: env # contains the env file with target OpsMan Information
- name: vars # variable files to be made available
  optional: true
- name: secrets
  # secret files to be made available
  # separate from vars, so they can be store securely
  optional: true
- name: ops-files # operations files to custom configure the product
  optional: true

params:
  VARS_FILES:
  # - Optional
  # - Filepath to the Ops Manager vars yaml file
  # - The path is relative to root of the task build,
  #   so `vars` and `secrets` can be used.

  OPS_FILES:
  # - Optional
  # - Filepath to the Ops Manager operations yaml files
  # - The path is relative to root of the task build

  ENV_FILE: env.yml
  # - Required
  # - Filepath of the env config YAML
  # - The path is relative to root of the `env` input

  DIRECTOR_CONFIG_FILE: director.yml
  # - Required
  # - Filepath to the director configuration yaml file
  # - The path is relative to the root of the `config` input

run:
  path: platform-automation-tasks/tasks/configure-director.sh

Multiple Foundations with one Repository

Multiple foundations may use a single Git configuration source but have different variables loaded from a foundation specific vars file, Credhub, Git repository, etc. This approach is very similar to the Single Repository for Each Foundation described above, except that variables are loaded in from external sources.

The variable source may be loaded in a number of ways. For example, it may be loaded from a separate foundation specific Git repository, a foundation specific subdirectory in the configuration source, or even a foundation specific vars file found in the base Git configuration.

This strategy can reduce the number of overall configuration files and configuration repositories in play, and can reduce foundation drift (as the basic configuration is being pulled from a single source). However, configuration management and secrets handling can quickly become more challenging.

This is the strategy used in our Reference Pipeline

For an example repo structure using this strategy, see the config repo used by the Reference Pipeline and the Resources Pipeline

Advanced Pipeline Design

Matching Resource Names and Input Names

Alternatively, we can create resources that match the input names on our tasks and bypass the need for using input_mapping. Even if these resources map to the same git repository and branch, they can be declared as separate inputs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
- name: config
  type: git
  source:
    private_key: ((repo-key.private_key))
    uri: ((repo-uri))
    branch: develop

- name: env
  type: git
  source:
    private_key: ((repo-key.private_key))
    uri: ((repo-uri))
    branch: develop

As long as those resources have an associated get: <resource-name> in the job, they will automatically be mapped to the inputs of the tasks in that job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
- name: configure-director
  serial: true
  plan:
    - aggregate:
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: config
        passed: [previous-job]
      - get: env
        passed: [previous-job]
    - task: configure-director
      image: platform-automation-image
      file: platform-automation-tasks/tasks/configure-director.yml

Passed Constraints

If you have two resources defined with the same git repository, such as env and config, and have a passed constraint on only one of them, there is a possibility that they will not be at the same SHA for any given job in your pipeline.

Example: 

1
2
3
- get: config
- get: env
  passed: [previous-job]

Modifying Resources in-place

Concourse 5+ Only

This section uses a Concourse feature that allows inputs and outputs to have the same name. This feature is only available in Concourse 5+. The following does not work with Concourse 4.

In certain circumstances, resources can be modified by one task in a job for use later in that same job. A few tasks that offer this ability include:

For each of these tasks, output_mapping can be used to "overwrite" an input with a modified input for use with tasks later in that job.

In the following example, prepare-tasks-with-secrets takes in the platform-automation-tasks input and modifies it for the download-product task. A more in-depth explanation of this can be found on the secrets-handling page.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
- name: configure-director
  serial: true
  plan:
    - aggregate:
      - get: platform-automation-image
        params:
          unpack: true
      - get: platform-automation-tasks
        params:
          unpack: true
      - get: config
      - get: env
    - task: prepare-tasks-with-secrets
      image: platform-automation-image
      file: platform-automation-tasks/tasks/prepare-tasks-with-secrets.yml
      input_mapping:
        tasks: platform-automation-tasks
      output_mapping:
        tasks: platform-automation-tasks
      params:
        CONFIG_PATHS: config
    - task: download-product
      image: platform-automation-image
      # The following platform-automation-tasks have been modified
      # by the prepare-tasks-with-secrets task
      file: platform-automation-tasks/tasks/download-product.yml
      params:
        CONFIG_FILE: download-ops-manager.yml