LATEST VERSION: 1.0 - CHANGELOG
Spring Cloud Data Flow v1.0

Operator Information

See below for information about Spring Cloud┬« Data Flow’s deployment model and other information which may be useful in administering Data Flow service instances or deployed applications.

Tile Configuration Options

The Spring Cloud Data Flow tile includes settings for various options. You can configure these by visiting the Installation Dashboard of Pivotal Cloud Foundry® Operations Manager and clicking the Spring Cloud Data Flow tile.

Configure Broker Database Service and Plan

By default, the Spring Cloud Data Flow product uses the MySQL for PCF v2 product and its default db-small service plan to provision a database service instance for the Data Flow service broker. If you are using the MySQL for PCF v2 product and it does not have an active db-small plan or if you wish to use an alternative service, you must configure the Spring Cloud Data Flow product for the service and service plan you wish to use. You can configure an alternative relational database service and service plan in the Service Broker pane of the Spring Cloud Data Flow settings.

Configuring MySQL Service

The broker’s database service should be configured only during the Spring Cloud Data Flow tile installation process. If you have already completed the tile installation process, do not alter that setting (you may still configure the broker’s database service plan after the tile installation).

WARNING: Configuring the service broker’s dependent relational database service after the tile installation has finished can result in orphaned Data Flow service instances or in multiple data sets and corruption of the broker database’s data.

See below for more information about the results of configuring the service broker’s relational database.

If you and change then
install the tile the service name or plan the tile uses the specified service and plan. You may also use the default values of p.mysql and db-small.
update the tile the service plan the tile’s relational database service instance is changed to use the specified plan. No data is lost.
update the tile the service name the service name change is ignored. If you have also changed the service plan, this may leave the broker in an inoperative state. To resolve this issue, revert your changes.

Configure Data Flow Server Database Service and Plan

By default, the Spring Cloud Data Flow product uses the MySQL for PCF v2 product and its default db-small service plan to provision a database service instance for the Data Flow server application deployed for each Data Flow service instance. If you are using the MySQL for PCF v2 product and it does not have an active db-small plan or if you wish to use an alternative service, you must configure the service and service plan you wish to use for the Data Flow server. You can configure an alternative relational database service and service plan in the Data Flow Server pane of the Spring Cloud Data Flow settings.

Configuring Data Flow Server Database Service

Configure Skipper Database Service and Plan

By default, the Spring Cloud Data Flow product uses the MySQL for PCF v2 product and its default db-small service plan to provision a database service instance for the Skipper application deployed for each Data Flow service instance. If you are using the MySQL for PCF v2 product and it does not have an active db-small plan or if you wish to use an alternative service, you must configure the service and service plan you wish to use for Skipper. You can configure an alternative relational database service and service plan in the Skipper pane of the Spring Cloud Data Flow settings.

Configuring Skipper Database Service

Enable or Disable Global Marketplace Access

By default, the Spring Cloud Data Flow product enables access to its p-dataflow service offering across all orgs in the PCF deployment as part of the tile installation process. If you wish to manually grant service access to specific organizations, you can configure the default global access in the Service Access pane of the Spring Cloud Data Flow settings.

Configuring Service Access

To disable the default of service access enabled for all orgs, clear the Enable global access to plans of service p_dataflow checkbox. Click Save, return to the Installation Dashboard, and apply your changes. You can now enable or disable access to the Data Flow service offering for specific orgs.

Service Orchestration

Spring Cloud Data Flow provides a Spring Cloud Data Flow server as a Managed Service on Pivotal Cloud Foundry (PCF). It uses Cloud Foundry’s Service Broker API to manage this service. See below for information about Spring Cloud Data Flow’s broker implementation.

The Service Broker

The Spring Cloud Data Flow service broker’s functionality is contained in the following Spring Boot application instance, which is deployed in the “system” organization to the “p-dataflow” space.

  • p-dataflow-[version]: Implements the Service Broker API to act on provision, deprovision, bind, and unbind requests.

The broker relies on the MySQL for Pivotal Cloud Foundry v2 product for the following service instance.

  • p_dataflow-p-mysql: A MySQL database used as a backing store for the service broker.

Note: You can configure an alternate relational database service for the broker to use. See the Configure Broker Database Service and Plan section above.

Service Broker Upgrades

The Spring Cloud Data Flow product upgrade process checks before redeploying the service broker to see whether the product’s version has changed. If the version has not changed, the upgrade process will continue without redeploying the service broker.

The service broker application is deployed using a blue-green deployment strategy. During an upgrade of the service broker, the broker will continue processing requests to provision, deprovision, bind, and unbind service instances, without downtime.

Access Via Apps Manager

To view the broker applications in Pivotal Cloud Foundry® Apps Manager, log into Apps Manager as an admin user and select the “system” org.

Viewing system Org

The applications are deployed in the “p-dataflow” space.

Viewing p-dataflow Space

Service Instance Management

Product Architecture

For each Spring Cloud Data Flow service instance created, the service broker provisions the following resources, all within the space from which the service instance was created (“the user space”) unless noted otherwise.

  • A new space within the “p-dataflow” org, named using the service instance GUID and containing:
    • A Data Flow server application.
    • A Data Flow metrics application.
    • A Skipper package management application.
    • A relational database service, bound to the Data Flow server application.

      Note: This relational database service is a “p.mysql” service instance by default. You can configure an alternate relational database when you create the service instance.

    • A relational database service, bound to the Skipper package management application.
    • A messaging data service, bound to the Data Flow metrics application.

      Note: This messaging data service is a “p-rabbitmq” service instance by default. You can configure an alternate messaging service when you create the service instance.

    • A Redis database service, bound to the Data Flow server application.
  • The following resources are created in the originating user space where the service instance command was targeted at:
    • A “p-dataflow” service instance.
    • A “p-dataflow” relational database service instance (providing access to the relational database service created in the service instance’s space within the “p-dataflow” org).
    • A “p-dataflow” messaging service instance (providing access to the messaging data service created in the service instance’s space within the “p-dataflow” org).
    • A “p-dataflow” analytics service instance (providing access to the analytics service created in the service instance’s space within the “p-dataflow” org).

Capacity Requirements

Spring Cloud Data Flow for PCF deploys:

  • A service broker and its associated relational database
  • Backing applications and their associated data service instances

Below are the usage requirements of the service broker.

Application Memory Allocated Disk Allocation
Service Broker 2 GB 1 GB

In addition, the service broker binds to a relational database to hold data about its service instances. This is configurable as described in The Service Broker section above.

Below are usage requirements of the Data Flow server and metrics applications that back each Spring Cloud Data Flow service instance.

Backing Application Memory Allocation / App Instance Disk Allocation / App Instance
Data Flow Server 2 GB 2 GB
Data Flow Metrics 1 GB 1 GB
Skipper 1 GB 2 GB

Spring Cloud Data Flow service instances may be backed by other service instances. These may be either services from PCF data service products or custom services provided to a Data Flow service instance during its creation. These data services will include a relational, messaging and analytics instance for each Data Flow service instance created. For Skipper there is a MySQL database that is provisioned and that it binds to as its backing store.

Create a pull request or raise an issue on the source for this page in GitHub