Spring Cloud Data Flow for VMware Tanzu
Spring Cloud Data Flow is an open-source toolkit that can deploy streaming and batch data pipelines on Cloud Foundry (see the project home page). The data pipelines are composed of Spring Cloud Stream or Spring Cloud Task applications. A streaming pipeline DSL makes it easy to specify which apps to deploy and how to connect outputs and inputs. The composed task DSL is useful for when a series of task apps must be run as a directed graph.
Spring Cloud Data Flow provides REST APIs for composing and deploying data pipelines programmatically. It also provides a dashboard that includes a graphical pipeline editor for building new pipelines interactively, as well as views of deployable apps and running apps with metrics using Prometheus, InfluxDB, or other monitoring systems. Spring Cloud Data Flow also includes a shell application for working with the API from the command line.
Spring Cloud Data Flow for VMware Tanzu automates the deployment of SCDF and its dependent services so that developers can use Apps Manager to deploy their own SCDF instances. They can then access their Data Flow server, either via the SCDF dashboard UI or using their VMware Tanzu Application Service for VMs (TAS) credentials with the SCDF shell.
Spring Cloud Data Flow for VMware Tanzu includes the following key features:
- Addition of the Spring Cloud Data Flow server to the Marketplace as a managed service (see Managing Service Instances
- Automatic integration with dependent TAS services (see Configure Data Flow Server Dependent Services):
- VMware Tanzu SQL [MySQL] (or a user-provided relational database service) for apps, pipelines, and task history
- RabbitMQ for VMware Tanzu (or a user-provided RabbitMQ or Kafka service) for event messaging
- Integration of Data Flow server with the TAS UAA security model
- One-step upgrade of Spring Cloud Data Flow servers and related components (see Updating an Instance)
The following table provides version and version-support information about Spring Cloud Data Flow for VMware Tanzu.
|Release date||November 3, 2021|
|Software component version||Spring Cloud Data Flow Pro Server v1.4.1|
|Open-source version||Spring Cloud Data Flow v2.9.1|
|Compatible Ops Manager version(s)||v2.4.x–v2.10.x|
|Compatible VMware Tanzu Application Service for VMs version(s)||v2.4.x–v2.13.x|
|IaaS support||All supported by TAS|
The following table provides information about the OSS versions used in each version of Spring Cloud Data Flow for VMware Tanzu.
|SCDF Tile||SCDF OSS||Spring Cloud Skipper OSS|
Spring Cloud Data Flow for VMware Tanzu has the following requirements:
- Java Cloud Foundry buildpack version 3.8 or later (see the Prerequisites section of Installation for more information)
- VMware Tanzu SQL [MySQL], or an alternative relational database service
- If you are using TLS with VMware Tanzu SQL [MySQL], VMware recommends using VMware Tanzu SQL [MySQL] v2.5.x. If using VMware Tanzu SQL [MySQL] v2.4.x, you must manually enable TLS on the Data Flow broker’s backing VMware Tanzu SQL [MySQL] service instance when installing Spring Cloud Data Flow for VMware Tanzu (see Using TLS with VMware Tanzu SQL [MySQL] v2.4.x).
- RabbitMQ for VMware Tanzu, or an alternative RabbitMQ or Kafka service
- Cloud Foundry Command Line Interface (cf CLI)
To use the Spring Cloud Data Flow shell interface with Spring Cloud Data Flow for VMware Tanzu service instances, install the following cf CLI plugins:
Spring Cloud Data Flow for VMware Tanzu cf CLI plugin. To install the plugin, run the following command:
$ cf install-plugin -r CF-Community "spring-cloud-dataflow-for-pcf"
Service Instance Logging cf CLI plugin. To install the plugin, run the following command:
$ cf install-plugin -r CF-Community "Service Instance Logging"