Configuring Multiple Platforms Support for VMware Spring Cloud® Data Flow for Kubernetes

This topic describes how to configure the multiple platform support in VMware Spring Cloud® Data Flow for Kubernetes (SCDF for Kubernetes).

Multiple Platforms

SCDF for Kubernetes can deploy streaming and batch appplications to different Kubernetes namespaces or clusters. For information about scenarios that benefit from this approach, see the Spring Cloud Data Flow documentation about the Role of Multiple Platform Deployments.

To configure multiple platform support, you must edit the application.yaml file for the relevant SCDF for Kubernetes application. The Data Flow server deploys Spring Cloud Task applications, and the Skipper server deploys Spring Cloud Stream applications.

To configure one or more deployment platforms for Spring Cloud Task applications, you must edit the application.yaml file for the Data Flow application. In the spring.cloud.dataflow.task.platform.kubernetes section, the accounts element is a map, where a key is an account name and a value is the properties of the KubernetesDeployerProperties Java object. For a list of object properties, with descriptions, see the Deployer Properties section of the Spring Cloud Data Flow Reference Documentation.

The following example configures two accounts: a default account that targets the default namespace and has a memory limit of 1024Mi, and a highmemory account that targets the highmemory-namespace namespace and has a memory limit of 4096Mi.

spring:
  cloud:
    dataflow:
      task:
        platform:
          kubernetes:
            accounts:
              default:
                namespace: default
                limits:
                  memory: 1024Mi
              highmemory:
                namespace: highmemory-namespace
                limits:
                  memory: 4096Mi

SCDF for Kubernetes uses the Fabric8 Kubernetes client library to connect to Kubernetes. The KubernetesDeployerProperties object contains a property named fabric8, which exposes the Fabric8 client configuration properties. You can use this to configure connection properties, such as the Kubernetes master location and certificate information.

The following example configures an account that references another cluster, which contains GPU resources.

spring:
  cloud:
    dataflow:
      task:
        platform:
          kubernetes:
            accounts:
              default:
                namespace: default
                limits:
                  memory: 1024Mi
              gpuzone:
                fabric8:
                  masterUrl: <k8s-master-api-url>
                  namespace: gpuzone-namespace
                  trustCerts: true

To configure one or more deployment platforms for Spring Cloud Stream applications, you must edit the application.yaml file for the Skipper application. Add similar configuration to the accounts map in the spring.cloud.dataflow.stream.platform.kubernetes section.

The following example configures two accounts that reference both a default namespace and a namespace where a Kafka broker is available. Because this configuration sets environment variables at the account level, all Spring Cloud Stream applications deployed to the kafkazone account will be configured to use the Kafka broker available in the kafka-namespace namespace.

spring:
  cloud:
    skipper:
      server:
        platform:
          kubernetes:
            accounts:
              default:
                namespace: default
                limits:
                  memory: 1024Mi
                  cpu: 500m
                readinessProbeDelay: 120
                livenessProbeDelay: 90
              kafkazone:
                namespace: kafka-namespace
                environmentVariables: 'SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=${KAFKA_SERVICE_HOST}:${KAFKA_SERVICE_PORT},SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES=${KAFKA_ZOOKEEPER_SERVICE_HOST}:${KAFKA_ZOOKEEPER_SERVICE_PORT}'
                limits:
                  memory: 2048Mi
                  cpu: 500m
                readinessProbeDelay: 180
                livenessProbeDelay: 120

Next: Install SCDF for Kubernetes

After completing the configuration steps, you can install SCDF for Kubernetes. For installation instructions, see Installing SCDF for Kubernetes.