Preparing to Install Spring Cloud Data Flow for Kubernetes

This topic describes how to prepare dependencies and your local machine for the installation of Spring Cloud Data Flow for Kubernetes (SCDF for Kubernetes).

Download and Extract the Installation Resources

To download and extract the SCDF for Kubernetes installation resources, follow the steps below.

  1. Visit VMware Tanzu Network and log in.

  2. Navigate to the Spring Cloud Data Flow for Kubernetes product listing.

  3. Download “SCDF for Kubernetes installation files”. Place the downloaded file in a local directory of your choice.

  4. Extract the contents of the data-flow-1.1.4.tgz file:

    $ tar xzvf data-flow-1.1.4.tgz

Relocate Distribution Container Images to Another Registry (Optional)

SCDF for Kubernetes relies on several container images. If you wish, you can re-locate these images to a different container image registry. Supported registries include Harbor, Google Container Registry (GCR), and DockerHub.

If you use Harbor as your container registry, ensure that Harbor is deployed in your environment and is accessible at a domain name of your choice. To deploy Harbor using Ops Manager, you can use the VMware Harbor Registry product.

To download and relocate the SCDF for Kubernetes installation images, follow the steps below.

  1. Visit VMware Tanzu Network and log in.

  2. Navigate to the Spring Cloud Data Flow for Kubernetes product listing.

  3. Download “SCDF for Kubernetes installation images”. Place the downloaded file in the same directory as the “SCDF for Kubernetes installation files” archive file described earlier.

  4. Extract the contents of the data-flow-images-1.1.4.tgz file:

    $ tar xzvf data-flow-images-1.1.4.tgz

  5. Use the Docker command-line interface (CLI) tool to authenticate to your registry.

  6. Change to the extracted spring-cloud-data-flow directory and run the image relocation script.

    The script is in the bin directory and is named relocate-image.sh. It takes two arguments: --app and --repository. Its usage is described below.

    Usage: ./bin/relocate-image.sh [-r|--repository <arg>] [-a|--app <arg>] [-h|--help]
  -r, --repository: Import images into given image repository (no default)
  -a, --app: Application to relocate, data-flow or skipper (default: data-flow)
  -h, --help: Prints help

    To relocate the Data Flow server image, run the following command:

    $ ./bin/relocate-image.sh --app data-flow --repository <registry-repo-prefix>/spring-cloud-dataflow-server

    To relocate the Composed Task Runner image, run the following command:

    $ ./bin/relocate-image.sh --app composed-task-runner --repository <registry-repo-prefix>/spring-cloud-dataflow-composed-task-runner

    To relocate the Skipper server image, run the following command:

    $ ./bin/relocate-image.sh --app skipper --repository <registry-repo-prefix>/spring-cloud-skipper-server

    The relocation script updates the kustomization.yaml files to match the new relocated image repository locations.

Use Custom-Built Images for Data Flow and Skipper Servers (Optional)

If you wish, you can replace the SCDF for Kubernetes container images with your own custom-built images.

In order to use these custom-built images, you must replace image references in some of the configuration files. See the following sections for more information.

Data Flow Server

To replace the image used for the Data Flow server, you must edit:

  • The apps/data-flow/kustomize/overlays/production/kustomization.yaml file, for the production environment
  • The apps/data-flow/kustomize/overlays/dev/kustomization.yaml file, for the development environment

In both files, locate the images section and change the newName, newTag, and digest entries to match your custom image. See the following example.

images:
- name: springcloud/spring-cloud-dataflow-server # used for Kustomize matching
  newName: registry.example.com/my-project/scdf-pro-server
  newTag: my-1.0.4
  digest: sha256:c63d57e7f70130c97aaf605f17130a28a658f7cc29ea37a2bd1c5b21e59c6c74
configMapGenerator:
- name: scdf-server
  files:
  - bootstrap.yaml
  - application.yaml
bases:
- ../../base
patches:
- deployment-patch.yaml
- service-patch.yaml

The digest must match the digest of your custom image. If you want the image to be pulled by the tag, remove the digest line.

Composed Task Runner

To replace the image used for the Composed Task Runner, you must edit:

  • The apps/data-flow/kustomize/overlays/production/application.yaml file, for the production environment
  • The apps/data-flow/kustomize/overlays/dev/application.yaml file, for the development environment

In both files, change the spring.cloud.dataflow.task.composedTaskRunnerUri entry to match your custom image. The value should be prefixed with docker://. See the following example.

spring:
  cloud:
    config:
      enabled: false
    dataflow:
      server:
        uri: http://${SCDF_SERVER_SERVICE_HOST}:${SCDF_SERVER_SERVICE_PORT}
      features:
        schedules-enabled: true
      task:
        composedTaskRunnerUri: docker://registry.example.com/my-project/spring-cloud-dataflow-composed-task-runner:my-2.6.1

...

You can provide either a tag or a digest that matches your custom image.

Skipper Server

To replace the image used for the Skipper server, you must edit:

  • The apps/skipper/kustomize/overlays/production/kustomization.yaml file, for the production environment
  • The apps/skipper/kustomize/overlays/dev/kustomization.yaml file, for the development environment

In both files, locate the images section and change the newName, newTag, and digest entries to match your custom image. See the following example.

images:
- name: springcloud/spring-cloud-skipper-server # used for Kustomize matching
  newName: registry.example.com/my-project/spring-cloud-skipper-server
  newTag: my-2.5.2
  digest: sha256:327181e455b869dce61e05b53cec493e054a3bfb1788576eab39a7ca3eebfe5e
configMapGenerator:
- name: skipper
  files:
  - bootstrap.yaml
  - application.yaml
bases:
- ../../base
patches:
- deployment-patch.yaml
- service-patch.yaml

The digest must match the digest of your custom image. If you want the image to be pulled by the tag, remove the digest line.

Next: Install Command-Line Tools

To install SCDF for Kubernetes, your Linux or macOS machine must have the following command-line tools installed:

  • kubectl, the Kubernetes CLI (for more information about installing kubectl, see Installing the Kubernetes CLI in the VMware Enterprise PKS documentation)
  • The following command-line utilities:
    • kapp
    • kbld
    • yq
    • sed (should already be installed on your machine)

To continue with the installation of SCDF for Kubernetes, see Installing Command-Line Tools.