LATEST VERSION: 1.12 - CHANGELOG
Pivotal Cloud Foundry v1.12

Tile Generator

This topic describes the Tile Generator tool, which helps tile authors develop, package, test, and deploy services and other add-ons to Pivotal Cloud Foundry (PCF).

Overview

Tiles are the installation package format used by Pivotal Ops Manager to deploy services and other add-ons to both public and private cloud deployments. Tile Generator uses templates and patterns that are based on years of experience integrating third-party services into Cloud Foundry and eliminates much of the need for you to have intimate knowledge of all the tools involved.

Overview

Tile Generator takes your software components and a simple configuration file that provides the minimal amount of information to describe and customize your tile. It then creates everything that’s required to deploy your software into PCF:

  • BOSH errands to deploy and delete your software, including blue/green deployments for zero-downtime upgrades
  • A BOSH release suitable for deploying your software to the Elastic Runtime or open-source Cloud Foundry
  • A Pivotal Ops Manager Tile that can be imported into Ops Manager, installed, configured, and deployed, including UI forms and automatic upgrades from previous versions
  • A Concourse pipeline configuration to enable Continuous Integration of your software with the latest versions of PCF

Use Tile Generator in combination with the pcf utility to enable rapid deploy and test cycles of your software.

The current release of Tile Generator supports tiles that have any combination of the following package types:

  • Cloud Foundry Applications
  • Cloud Foundry Buildpacks
  • Cloud Foundry Service Brokers (both inside and outside the Elastic Runtime)
  • Docker images (both inside and outside the Elastic Runtime)

Legacy Tiles and OSS-Compatible Service Brokers

Many tile authors, in both Pivotal-internal teams and at external partner companies, built their PCF tiles before Tile Generator existed.

Many other tile authors serve two markets with their service integrations, offering both a Cloud Foundry-compatible service broker to open-source users and corresponding PCF tile for PCF users. They want to continue serving both sets of users.

All of these tile authors can now use Tile Generator to simplify and speed up their development. Tile Generator can generate an OSS-compatible BOSH release service broker BOSH release in addition to a PivNet-ready PCF tile.

Screencast

For a 7-minute introduction into what Tile Generator is and does, see this screencast.

How to Use

  1. Install the tile-generator python package.

    Note: The tile-generator requires Python 2 and does not work with Python 3.

    Pivotal recommends using a virtualenv environment to avoid conflicts with other Python packages. A virtualenv is simply a directory containing dependencies for a project. When a virtual environment is active, packages install into the virtualenv instead of the system-wide Python installation. Create a virtualenv with the virtualenv command:

    virtualenv -p python2 tile-generator-env
    

    To activate the virtualenv, source the bin/activate script in the virtualenv directory:

    source tile-generator-env/bin/activate
    

    Then install the tile-generator package inside the virtualenv with:

    pip install tile-generator
    

    This puts the tile and pcf commands in your PATH when the virtualenv is active. To deactivate the virtualenv, simply run the command deactivate.

    Note: To upgrade Tile Generator, run the following command with the virtualenv activated: pip install tile-generator --upgrade

  2. Install the BOSH CLI

  3. From within the root directory of the project for which you want to create a tile, initialize it as a tile repository. Pivotal recommends that you use a git repository:

    cd YOUR-PROD-DIRECTORY
    tile init
    
  4. Edit the generated tile.yml file to define your tile.

  5. Build your tile:

    tile build
    

The generator first creates a BOSH release in the release subdirectory, then wrap that release into a Pivotal tile (in the product subdirectory). If required for the installation, it automatically pulls down the latest release version of the Cloud Foundry CLI.

Tile Generator is also available pre-installed in a Docker image on Docker Hub. This image contains the tile-generator tile and pcf commands, all the necessary Python dependencies, as well as the BOSH CLI.

You can use this in Concourse pipelines by specifying it as the base image for your tasks:

  - task: tile-build
    config:
      platform: linux
      image: cfplatformeng/tile-generator

Or, you can derive your own Docker images from this one by using it as the base image in your Dockerfile:

FROM cfplatformeng/tile-generator

Build the Sample

The tile-generator repository includes a sample tile that exercises most of the features of Tile Generator. This sample tile is used by Tile Generator’s CI pipeline to verify that things work correctly. You can build this sample using the following steps:

  1. Download the Redis BOSH release and save it to sample/resources/redis-13.1.2.tgz.

  2. Run the following commands:

    cd sample
    src/build.sh
    tile build
    

Note: The sample tile includes a Python app that is re-used in several packages, sometimes as an app, sometimes as a service broker. One of the deployments (app3) uses the sample app inside a Docker image that is currently only modified by the CI pipeline. If you modify the sample app, you have to build your own Docker image using the provided Dockerfile and change the image name in sample/tile.yml to include the modified code in app3.

Define your Tile in tile.yml

All required configuration for your tile is in the file called tile.yml. tile init creates an initial version for you that can serve as a template. The first section in the file describes the general properties of your tile:

name: tile-name # Match Pivotal Network product name, lowercase with dashes
icon_file: resources/icon.png
label: Brief Text for the Tile Icon
description: Longer description of the tile's purpose

The name should be informative, for example, your company name followed by the product name, e.g., acme-anvil. The name should match your product slug on Pivotal Network, which enables update notifications for customers. Coordinate with your product team to agree upon a name; marketing teams often care about the name because it shows up in Pivotal Network URLs.

The icon_file should be a 128x128 pixel image that appears on your tile in the Ops Manager GUI. By convention, any resources used by the tile should be placed in the resources sub-directory of your repository, although this is not mandatory. The label text appears on the tile under your icon.

Packages

Next you can specify the packages to be included in your tile. The format of each package entry depends on the type of package you are adding.

Pushed Apps

Apps (including service brokers) that are being cf pushed into the Elastic Runtime use the following format:

- name: my-application
  type: app # or app-broker
  manifest:
    # any options that you would normally specify in a cf manifest.yml, including</i>
    buildpack: # required
    command:
    domain:
    host:
    instances:
    memory:
    path:
    env:
    services:
  health_check: none                 # optional
  configurable_persistence: true     # optional
  needs_cf_credentials: true         # optional
  auto_services: p-mysql p-redis     # optional
  consumes:                          # optional
    redis:
      from: redis

For apps that are normally pushed as multiple files (node.js for example) zip up the project files plus all dependencies into a single ZIP file, then edit tile.yml to point to the zipped file:

cd <your project dir>
zip -r resources/<your project name>.zip <list of file and dirs to include in the zip>

If your app is a service broker, use app-broker as the type instead of just app. The app is then automatically registered as a broker on install, and deleted on uninstall.

health_check lets you configure the value of the cf cli --health_check_type option. Expect this option to move into the manifest as soon as CF supports it there. Currently, the only valid options are none and port.

configurable_persistence: true results in the user being able to select a backing service for data persistence. If there is a specific broker you want to use, you can use the auto-services feature described below. If you want to bind to an already existing service instance, use the services property of the manifest instead.

needs_cf_credentials causes the app to receive two additional environment variables named CF_ADMIN_USER and CF_ADMIN_PASSWORD with the admin credentials for the Elastic Runtime into which they are being deployed. This allows apps and services to interact with the Cloud Controller.

The auto_services feature is described in more detail below.

consumes specifies the BOSH links to consume and presents the hosts and properties from the links as environment variables on the app:

  • <LINK>_HOST: The address of the first instance of the link.
  • <LINK>_HOSTS: A JSON array of the addresses of all instances of the link.
  • <LINK>_PROPERTIES: A JSON object of the properties on the link.

Service Brokers

Most modern service brokers are pushed into the Elastic Runtime as normal CF apps. For these types of brokers, use the Pushed Application format specified above, but set the type to app-broker or docker-app-broker instead of just app or docker-app:

- name: my-broker
  type: app-broker
  manifest:
    buildpack: # required
    command:
    domain:
    path:
    # ...
  needs_cf_credentials: true           # optional
  auto_services: p-mysql p-redis       # optional
  enable_global_access_to_plans: true  # optional

Note: Unless you specify the enable_global_access_to_plans: true option, your broker’s services do not appear in the user’s Marketplaces. Operators have to use the cf enable-service-access command to allow specific users, orgs, and spaces to access your services.

Your broker is automatically registered with the Cloud Controller. The Cloud Controller invokes your broker’s endpoints, and it uses basic authentication to secure those API calls. The credentials it uses are passed to your broker in two environment variables:

SECURITY_USER_NAME
SECURITY_USER_PASSWORD

Your broker is expected to accept those credentials. If it doesn’t, automatic broker registration fails.

Some service brokers support operator-defined service plans, for instance when the plans reflect customer license keys. To allow operators to add plans from the tile configuration, add the following section at the top level of your tile.yml:

service_plan_forms:
- name: service_plans_1
  label: Service 1 Plans
  description: Specify the plans you want Service 1 to offer
  properties:
  - name: description
    type: string
    description: "Some Description"
    configurable: true
  - name: license_key1
    type: string
    configurable: true
    description: The license key for this plan
  - name: num_seats1
    type: integer
    configurable: true
    description: The number of available seats for this license
    default: 1
    constraints:
      min: 1
      max: 500

Name and GUID fields are supplied by default for each plan, but all other fields are optional and customizable. Multiple forms are supported. The operator-configured plans are passed to your service broker in JSON format in an environment variable named after your form but in ALL CAPS (in this case SERVICE_PLANS_1).

For an external service broker, use:

- name: my-application
  type: external-broker
  uri: http://broker3.example.com
  username: user
  password: #secret
  internal_service_names: 'service1,service2'

BOSH Releases

You can include BOSH releases in your tile with the bosh-release package type. For example, here is a package definition to include a Redis BOSH release:

- name: redis
  type: bosh-release
  path: resources/redis-13.1.2.tgz
  jobs:
  - name: redis
    templates:
    - name: redis
      release: redis
    memory: 512
    ephemeral_disk: 4096
    persistent_disk: 4096
    instances: 2
    cpu: 2
    static_ip: 0
    dynamic_ip: 1
    default_internet_connected: false
    max_in_flight: 1
    properties:
      password: red!s
  - name: sanity-tests
    templates:
    - name: sanity-tests
      release: redis
    lifecycle: errand
    post_deploy: true
    run_post_deploy_errand_default: when-changed
    memory: 512
    ephemeral_disk: 4096
    persistent_disk: 0
    cpu: 2
    dynamic_ip: 1

To include BOSH links in your bosh-release package’s deployment manifest, you can include the consumes and/or provides declarations as strings in the job’s templates section, e.g.:

# ...
  jobs:
  - name: job_name
    templates:
    - name: template_name
      consumes: |
        consumed_link: {from: foo}
      provides: |
        provided_link: {as: bar}

Buildpacks

- name: my-buildpack
  type: buildpack
  path: resources/buildpack.zip
  buildpack_order: 99     # optional, 99 means end of the list

Docker Images

Apps packages as Docker images can be deployed inside or outside the Elastic Runtime. To push a Docker image as a CF app, use the Pushed Application format specified above, but use the docker-app or docker-app-broker type instead of just app or app-broker. The Docker image to be used is then specified using the image property:

- name: app1
  type: docker-app
  image: test/dockerimage
  manifest:
    ...

If this app is also a service broker, use docker-app-broker instead of just docker-app. This option is appropriate for Docker-wrapped 12-factor apps that delegate their persistence to bound services.

Docker apps that require persistent storage can not be deployed into the Elastic Runtime. These can be deployed to separate BOSH-managed VMs instead by using the docker-bosh type:

- name: docker-bosh1
  type: docker-bosh
  cpu: 5
  memory: 4096
  ephemeral_disk: 4096
  persistent_disk: 2048
  instances: 1
  manifest: |
    containers:
    - name: redis
      image: "redis"
      command: "--dir /var/lib/redis/ --appendonly yes"
      bind_ports:
      - "6379:6379"
      bind_volumes:
      - "/var/lib/redis"
      entrypoint: "redis-server"
      memory: "256m"
      env_vars:
      - "EXAMPLE_VAR=1"
    - name: mysql
      image: "google/mysql"
      bind_ports:
      - "3306:3306"
      bind_volumes:
      - "/mysql"
    - name: elasticsearch
      image: "bosh/elasticsearch"
      links:
      - mysql:db
      depends_on:
      - mysql
      bind_ports:
      - "9200:9200"

If a Docker image cannot be downloaded by BOSH dynamically, provide a ready-made Docker image and package it as part of the BOSH release. In that case, specify the image as a local file.

- name: docker-bosh2
  type: docker-bosh
  files:
  - path: resources/cfplatformeng-docker-tile-example.tgz
  cpu: 5
  memory: 4096
  ephemeral_disk: 4096
  persistent_disk: 2048
  instances: 1
  manifest: |
    containers:
    - name: test_docker_image
      image: "cfplatformeng/docker-tile-example"
      env_vars:
      - "EXAMPLE_VAR=1"
      # See below on custom forms/variables and binding it to the Docker env variable
      - "custom_variable_name=((.properties.customer_name.value))"

To expose a container via gorouter, for example, one of the Docker containers hosts an admin webapp interface, use routes to choose a port and prefix. The external URL is [prefix]-[package.name].[system-domain]. In this case, the URL is https://admin-docker-bosh3.sys.example.com, where sys.example.com is the PCF system domain. routes is a list, so multiple containers can be exposed.

- name: docker-bosh3
  type: docker-bosh
  docker_images:
  - "cfplatformeng/database"
  - "cfplatformeng/admin_ui"
  routes:
    - prefix: admin
      port: 8080
  cpu: 5
  memory: 4096
  ephemeral_disk: 4096
  instances: 1
  manifest: |
    containers:
    - name: database
      image: "cfplatformeng/database"
      bind_ports:
      - "5432:5432"
    - name: admin_ui
      image: "cfplatformeng/admin_ui"
      bind_ports:
      - "8080:8080"

Custom Forms and Properties

You can pass custom properties to all apps deployed by your tile by adding the to the properties section of tile.yml:

properties:
- name: author
  type: string
  label: Author
  value: Tile Ninja

If you want the properties to be configurable by the tile installer, place them on a custom form instead:

forms:
- name: custom-form1
  label: Test Tile
  description: Custom Properties for Test Tile
  properties:
  - name: customer_name
    type: string
    label: Full Name
  - name: street_address
    type: string
    label: Street Address
    description: Address to use for junk mail
  - name: city
    type: string
    label: City
  - name: zip_code
    type: string
    label: ZIP+4
    default: '90310'
  - name: country
    type: dropdown_select
    label: Country
    options:
    - name: country_us
      label: US
      default: true
    - name: country_elsewhere
      label: Elsewhere
- name: account-info-1
  label: Account Info
  description: Example Account Information Form
  properties:
  - name: username
    type: string
    label: Username
  - name: password
    type: secret
    label: Password

Properties defined in either section are passed to all pushed apps as environment variables (the name of the environment variable is the same as the property name but in ALL_CAPS). They can also be referenced in other parts of the configuration file by using (( .properties.<property-name> )) instead of a hardcoded value.

All properties supported by Ops Manager may be used. The syntax is the same as used by Ops Manager, except that for simplicity property blueprints for form fields do not need to be declared separately. Instead, the declaration is included in the form itself. For a complete list of supported property types and syntax, see the Ops Manager Product Template Reference.

Properties of type secret have their value hidden on the forms and obfuscated in the installation logs (all but the first two characters are replaced by *****). But their value is passed to your apps in plain text as all other value types.

Automatic Provisioning of Services

Tile Generator automates the provisioning of services. Any app (including service brokers and Docker-based apps) that are being pushed into the Elastic Runtime can automatically be bound to services through the auto_services feature:

- name: app1
  type: app
  auto_services:
  - name: p-mysql
    plan: 100mb-dev
  - name: p-redis

You can specify any number of service names, optionally specifying a specific plan. During deployment, the generated tile creates an instance of each service if one does not already exist and then bind that instance to your package.

Service instances provisioned this way survive updates, but are deleted when the tile is uninstalled.

Note: The name is the name of the provided service, not the broker. In many cases these are not the same, and a single broker may even offer multiple services. Use cf service-access to see the services and plans offered by installed service brokers.

If you do not specify a plan, Tile Generator uses the first plan listed for the service in the broker catalog. It is a good idea to always specify a service plan. If you change the plan between versions of your tile, Tile Generator attempts to update the plan while preserving the service (thus not causing data loss during upgrade). If the service does not support plan changes, this causes the upgrade to fail.

configurable_persistence is really just a special case of auto_services, letting the user choose between some standard brokers.

Declaring Product Dependencies

When your product has dependencies on others, you can have Ops Manager enforce that dependency by declaring it in your tile.yml file as follows:

requires_product_versions:
- name: p-mysql
  version: '~> 1.7'

If the required product is not present in the PCF installation, Ops Manager displays a message saying <your-tile> requires 'p-mysql' version '~> 1.7' as a dependency and refuses to install your tile until that dependency is satisfied.

When using automatic provisioning of services as described above, it is often appropriate to add those products as a dependency. Tile Generator can not do this automatically as it can’t always determine which product provides the requested service.

Orgs and Spaces

By default, Tile Generator creates a single new org and space for any packages that install into the Elastic Runtime, using the name of the tile and appending -org and -space, respectively. The default memory quota for a newly created org is 1024 (1 G). You can change any of these defaults by specifying the following properties in tile.yml:

org: test-org
org_quota: 4096
space: test-space

Security

If your cf packages need outbound access (including access to other packages within the same tile), you need to apply an appropriate security group. The following option removes all constraints on outbound traffic:

apply_open_security_group: true

Stemcells

Tile Generator defaults to a recent stemcell supported by Ops Manager. In most cases the default is fine, because the stemcell is only used to execute CF command lines and/or the Docker daemon. But if you have specific stemcell requirements, you can override the defaults in your tile.yml file by including a stemcell-criteria section and replacing the appopriate values:

stemcell_criteria:
  os: 'ubuntu-trusty'
  version: '3146.5'     #NOTE: You must quote the version to force the type to be string

Custom Errands

Tile Generator supplies standard errands to deploy and delete CF type packages. You can replace or augment those errands by specifying errand shell commands in your tile.yml file. For example:

packages:
- name: meta-buildpack
  type: buildpack
  buildpack_order: 0 # Go to head of list
  path: meta_buildpack.zip
  deploy: |
    cp meta_buildpack.zip meta_buildpack-v{{context.version}}.zip
    existing=`cf buildpacks | grep '^meta_buildpack'`
    if [ -z "$existing" ]; then
      cf create-buildpack meta_buildpack meta_buildpack-v{{context.version}}.zip 0
    else
      semver=`echo "$existing" | sed 's/.* meta_buildpack-v\(.*\)\.zip/\1/'`
      if is_newer "{{context.version}}" "$semver"; then
        cf update-buildpack meta_buildpack -p meta_buildpack-v{{context.version}}.zip
      else
        echo "Newer version ($semver) of meta_buildpack is already present"
      fi
      cf update-buildpack meta_buildpack -i 0
    fi
  delete: |
    # Intentional no-op, as others may have a dependency on this

deploy and delete completely replace the standard errand commands for the package in which you include them. If you want to keep the standard commands, but add additional commands to execute before or after the standard errand, use pre_deploy, post_deploy, pre_delete, and/or post_delete instead.

Versioning

Tile Generator uses semver versioning. By default, tile build generates the next patch release. Major and minor releases can be generated by explicitly specifying tile build major or tile build minor. Or to override the version number completely, specify a valid semver version on the build command, e.g. tile build 3.4.5.

No-op content migration rules are generated for every prior release to the current release, so that Ops Manager allows tile upgrades from any version to any newer version. This depends on the existence of the file tile-history.yml. In a pinch, if you need to be able to upgrade from a random old version to a new one, you can edit that file, or do:

tile build <old-version>
tile build <new-version>

The new tile then supports upgrades from old-version.

Upgrades

By default, Tile Generator produces all code necessary to do a blue/green, zero-downtime deployment of all tile components when installing a newer version over an older one. For most tile versions this is all that is needed.

Ops Manager has support for performing upgrade actions, like database migrations, during a tile upgrade, but this capability is not yet exposed through tile generator.

Example

$ tile build
name: tibco-bwce
icon: icon.png
label: TIBCO BusinessWorks Container Edition
description: BusinessWorks edition that supports deploying to Cloud Foundry
version: 0.0.2

bosh init-release --dir=cf
bosh generate-package cf_cli
bosh generate-package bwce_buildpack
bosh generate-job install_bwce_buildpack
bosh generate-job remove_bwce_buildpack
bosh create-release --final --tarball=cf_incubator --version 0.0.2

tile generate release
tile generate metadata
tile generate errand install_bwce_buildpack
tile generate errand remove_bwce_buildpack
tile generate content-migrations

created tile tibco-bwce-0.0.2.pivotal

This tile includes a single large buildpack and takes less than 15 seconds to build including the CF CLI download and the BOSH release generation.

Supported Commands

tile init [<tile-name>]
tile build [patch|minor|major|<version>]

Credits

Create a pull request or raise an issue on the source for this page in GitHub