Using Stark & Wayne kStreams™ for PCF

This page provides a walkthrough of the following basic features:

  • Deploying sample producer and sample consumer applications to Pivotal Cloud Foundry (PCF)
  • Provisioning Stark & Wayne kStreams database and binding to the two apps
  • Using Apache Kafka to observe the apps’ communication
  • Cleanup

Preparation

Create a new Pivotal Cloud Foundry space using the following commands:

bash
cf create-space kafka-tutorial
cf t -s kafka-tutorial

Clone the repository containing the two sample apps using the following commands:

bash
git clone https://github.com/dingotiles/dingo-kafka-sample-apps.git

Discover Stark & Wayne kStreams Service

Confirm that Stark & Wayne kStreams has been installed by your platform operators using the following command:

bash
$ cf marketplace

service                 plans      description
starkandwayne-kstreams     topic      Share a single topic on shared Kafka
                        shared     Create your own topics on shared Kafka

Provision Shared Apache Kafka Topic

Provision a shared Apache Kafka topic using the following command:

$ cf create-service starkandwayne-kstreams topic kafka-service

Deploy Producer App to Pivotal Cloud Foundry

Build and deploy the sample producer application in the root directory of the dingo-kafka-simple-apps Git repository you downloaded above using the following commands:

bash
mvn -Dmaven.test.skip=true install
cd kafka-sample-producer
cf push

This application is a headless application – it continously pushes events to the Kafka topic.

To observe it in action, watch its logs:

bash
cf logs kafka-sample-producer

An example of the logs might be:

2017-05-15T13:07:16.88+1000 [APP/PROC/WEB/0] OUT 2017-05-15 03:07:16 [pool-1-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version : 0.10.0.1
2017-05-15T13:07:16.88+1000 [APP/PROC/WEB/0] OUT 2017-05-15 03:07:16 [pool-1-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId : a7a17cdec9eaa6c5
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT 2017-05-15 03:07:26 [pool-1-thread-1] INFO  i.p.c.s.p.app.KafkaSampleProducer - sending message: the time is now 03:07:26
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT 2017-05-15 03:07:26 [pool-1-thread-1] INFO  o.a.k.c.producer.ProducerConfig - ProducerConfig values:
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT    metric.reporters = []
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT    metadata.max.age.ms = 300000
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT    reconnect.backoff.ms = 50
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT    sasl.kerberos.ticket.renew.window.factor = 0.8
2017-05-15T13:07:26.87+1000 [APP/PROC/WEB/0] OUT    bootstrap.servers = [10.244.0.148:9092, 10.244.0.149:9092, 10.244.0.155:9092]
...

Deploying Consumer App

Open another terminal window. You should have already done a mvn install during the kafka-sample-producer phase, but if you have not, run mvn -Dmaven.test.skip=true install in the base directory of dingo-kafka-simple-apps.

Build and deploy the sample consumer application using the following commands:

cd -
cd kafka-sample-consumer
cf push

This application is a headless application – it continously pushes events to the Kafka topic.

To observe it in action, watch its logs:

bash
cf logs kafka-sample-consumer

An example of each message received from the kafka-sample-producer through Apache Kafka:

2017-05-15T13:12:56.98+1000 [APP/PROC/WEB/0] OUT 2017-05-15 03:12:56 [-kafka-consumer-1] INFO  i.p.c.s.c.app.ConsumerListener - received: ConsumerRecord(topic = 23ff851e-8642-4682-839b-1d1b04125f19, partition = 1, offset = 26, CreateTime = 1494817976982, checksum = 535612902, serialized key size = -1, serialized value size = 49, key = null, value = hello from the KafkaSampleProducer: 1494817976878)

Service Credentials

When the Stark & Wayne kStreams service instance is bound to the app, it provides a set of connection credentials. Each application uses those credentials to connect to Apache Kafka with a Kafka client library.

To see the credentials, use the following commands:

bash
cf env kafka-sample-producer

The output will include a snippet like the one below:

json
{
 "VCAP_SERVICES": {
  "starkandwayne-kstreams": [
   {
    "credentials": {
      "hostname": "10.213.10.35:9092,10.213.10.36:9092,10.213.10.34:9092",
      "topicName": "67f85c81-ee6f-4791-b169-864f2e18d929",
      "uri": "kafka://10.213.10.35:9092,10.213.10.36:9092,10.213.10.34:9092/67f85c81-ee6f-4791-b169-864f2e18d929",
      "zkPeers": "10.213.10.31:2181,10.213.10.32:2181,10.213.10.33:2181"
    },
    "label": "starkandwayne-kstreams",
    "name": "kafka-service",
    "plan": "topic",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [
     "kafka"
    ],
    "volume_mounts": []
   }
  ]
 }
}

Cleanup

To delete the sample apps, use the following command:

cf delete kafka-sample-consumer
cf delete kafka-sample-producer
cf delete-service kafka-service
Create a pull request or raise an issue on the source for this page in GitHub