Getting Started: ODB on a Local Development Environment

This topic describes how to create and manage an on-demand service broker (ODB) on a local development machine using Pivotal Cloud Foundry (PCF) Dev and BOSH Lite.

For more information about the components in this topic, see PCF Dev and BOSH-Lite.

Note: The examples in this topic are based on Kafka open source messaging. See the following sample code directories:

Prerequisites

Before you set up and use ODB on your local machine, install and configure the following components:

  • BOSH Lite v9000.131.0 or later. To install BOSH Lite, see Install in the BOSH Lite documentation.

  • PCF Dev file pcfdev-v0.19.1-rc.46. To install PCF Dev, see Installing PCF Dev. Record the PCF Dev domain for later. The default is local.pcfdev.io.

Part 1: Set Up

This section details how to prepare BOSH Lite and set up the Kafka example service, the Kakfa example service adapter, and ODB.

Step 1: Prepare BOSH Lite

To prepare BOSH Lite, do the following:

  1. Target your BOSH Lite installation.

    bosh alias-env lite -e 192.168.50.4
    

  2. Upload the BOSH Lite stemcell.

    bosh -e lite upload-stemcell \
    https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent?v=3262.2
    

Step 2: Set Up the Kafka Example Service

To set up the Kafka example service, do the following:

  1. Clone the Kafka example service into your workspace.

    git clone \
    https://github.com/pivotal-cf-experimental/kafka-example-service-release.git
    

  2. In the kafka-example-service-release directory, create and upload the Kafka example service.

    cd kafka-example-service-release
    bosh create-release --name kafka-example-service
    

  3. Upload the service to the BOSH director.

    bosh -e lite upload-release
    

Step 3: Set Up the Kafka Example Service Adapter

To set up the Kafka example service adapter, do the following:

  1. Clone the Kafka example service adapter.

    git clone \
    https://github.com/pivotal-cf-experimental/kafka-example-service-adapter-release.git
    

  2. Update service adapter dependencies.

    cd kafka-example-service-adapter-release
    git submodule update --init --recursive
    

  3. Create the example service adapter.

    bosh create-release --name kafka-example-service-adapter
    

  4. Upload the example service adapter to the BOSH director.

    bosh -e lite upload-release
    

Step 4: Set Up ODB

To set up ODB, do the following:

  1. Download the on-demand service broker from Pivotal Network. To download, see Pivotal Cloud Foundry On Demand Services SDK.

  2. Upload the on-demand-service-broker release.

    bosh -e lite upload-release on-demand-service-broker-X.Y.Z.tgz
    
    Where:

    • X.Y.Z is the ODB release version.

      For example:
      $ bosh -e lite upload-release on-demand-service-broker-0.21.1.tgz
      

Part 2: Create

This section describes how to create a BOSH deployment and a service broker on PCF Dev.

Step 1: Create a BOSH Deployment

To create a BOSH Lite deployment, do the following:

  1. Create a new directory in your workspace and a cloud_config.yml for the BOSH Lite Director. For example:

    vm_types:
    - name: container
      cloud_properties: {}
    
    networks:
    - name: kafka
      type: manual
      subnets:
      - range: 10.244.1.0/24
        gateway: 10.244.1.1
        az: lite
        cloud_properties: {}
    
    disk_types:
    - name: ten
      disk_size: 10_000
      cloud_properties: {}
    
    azs:
    - name: lite
      cloud_properties: {}
    
    compilation:
      workers: 2
      reuse_compilation_vms: true
      network: kafka
      az: lite
      cloud_properties: {}
    
  2. Update the BOSH Lite cloud config using the deployment manifest.

    bosh -e lite update-cloud-config cloud_config.yml
    

  3. Record the URL and UUID of your BOSH Lite director.

    bosh environment
    
    See the following example output:
    $ bosh environment
    Config
                 /Users/pivotal/.bosh_config

    Director Name Bosh Lite Director URL https://192.168.50.4:25555 Version 1.3215.0 (00000000) User admin UUID 17a45148-1d00-43bc-af28-9882e5a6535a CPI warden_cpi dns disabled compiled_package_cache enabled (provider: local) snapshots disabled

  4. Create a BOSH Lite deployment manifest in a file called deployment_manifest.yml using the following as a base:

    name: kafka-on-demand-broker
    
    director_uuid: BOSH-LITE-UUID
    
    releases:
    - name: &broker-release on-demand-service-broker
      version: latest
    - name: &service-adapter-release kafka-example-service-adapter
      version: latest
    - name: &service-release kafka-example-service
      version: latest
    
    stemcells:
    - alias: trusty
      os: ubuntu-trusty
      version: STEMCELL-VERSION
    
    instance_groups:
    - name: broker
      instances: 1
      vm_type: container
      persistent_disk_type: ten
      stemcell: trusty
      azs: [lite]
      networks:
      - name: kafka
      jobs:
      - name: kafka-service-adapter
        release: *service-adapter-release
      - name: admin_tools
        release: *service-release
      - name: broker
        release: *broker-release
        properties:
          port: 8080
          username: broker    # or replace with your own
          password: password  # or replace with your own
          disable_ssl_cert_verification: true
          bosh:
            url: BOSH-LITE-URL
            authentication:
              basic:
                username: admin
                password: admin
          cf:
            url: https://api.PCF-DEV-DOMAIN
            authentication:
              url: https://uaa.PCF-DEV-DOMAIN
              user_credentials:
                username: admin
                password: admin
          service_adapter:
            path: /var/vcap/packages/odb-service-adapter/bin/service-adapter
          service_deployment:
            releases:
            - name: *service-release
              version: SERVICE-RELEASE-VERSION
              jobs: [kafka_server, zookeeper_server]
            stemcells:
              - os: ubuntu-trusty
                version: STEMCELL-VERSION
          service_catalog:
            id: D94A086D-203D-4966-A6F1-60A9E2300F72
            service_name: kafka-service-with-odb
            service_description: Kafka Service
            bindable: true
            plan_updatable: true
            tags: [kafka]
            plans:
            - name: small
              plan_id: 11789210-D743-4C65-9D38-C80B29F4D9C8
              description: A Kafka deployment with a single instance of each job and persistent disk
              instance_groups:
              - name: kafka_server
                vm_type: container
                instances: 1
                persistent_disk_type: ten
                azs: [lite]
                networks: [kafka]
              - name: zookeeper_server
                vm_type: container
                instances: 1
                persistent_disk_type: ten
                azs: [lite]
                networks: [kafka]
              properties:
                auto_create_topics: true
                default_replication_factor: 1
    
    update:
      canaries: 1
      canary_watch_time: 30000-180000
      update_watch_time: 30000-180000
      max_in_flight: 4
    

    Where:

    • BOSH-LITE-UUID is the UUID value you recorded in the BOSH environment step above.
    • BOSH-LITE-URL is the URL value you recorded in the BOSH environment step above.
    • PCF-DEV-DOMAIN is the PCF Dev domain you recorded in the Prerequisites above.
  5. Deploy the broker.

    bosh -e lite -d kafka-on-demand-broker deployment_manifest.yml
    

  6. Record the IP address of the deployed broker.

    bosh -e lite -d kafka-on-demand-broker instances
    

    See the following example output:

    $ Using environment 'lite' as user 'admin' (openid, bosh.admin)
    
    Task 54727. Done
    
    Deployment 'redis-on-demand-broker-dev2'
    
    Instance                                     Process State  AZ  IPs
    broker/84294753-84b9-4be1-a338-37c1f3e71919  running        z1  10.244.1.2
    
    1 instances
    
    Succeeded
    

Step 2: Create a Service Broker on PCF Dev

To create a service broker on PCF Dev, do the following:

  1. Create a service broker on PCF Dev and enable access to its service offering.

    cf create-service-broker kafka-broker USERNAME PASSWORD http://BROKER-IP:8080
    
    Where:

    • USERNAME and PASSWORD are the broker’s credentials set under properties in the broker job.
    • BROKER-IP is the value obtained in the step above. See the last step in Create a BOSH Deployment.

    For example:

    $ cf create-service-broker kafka-broker broker password http://10.244.1.2:8080
    

  2. Enable access to the broker’s service plans.

    cf enable-service-access kafka-service-with-odb
    

  3. View the broker-offered services in the Marketplace.

    cf marketplace
    

    See the following example output:

    Getting services from Marketplace in org pcfdev-org / space pcfdev-space as admin...
    OK
    
    service                  plans        description
    kafka-service-with-odb   small        Kafka Service
    p-mysql                  512mb, 1gb   MySQL databases on demand
    p-rabbitmq               standard     RabbitMQ is a robust and scalable high-performance multi-protocol messaging broker.
    p-redis                  shared-vm    Redis service to provide a key-value store
    
  4. Create a service instance using the Kafka on-demand broker.

    cf create-service kafka-service-with-odb small k1
    

Part 3: Verify and Use

To verify and use your on-demand service, do the following:

Step 1: Verify Your BOSH Deployment and On-Demand Service

  1. Check the status of your service.

    cf service k1
    
    See the state change from create in progress to create succeeded.

  2. Verify that the on-demand service is provisioned in the BOSH deployment

    bosh -e lite deployments
    

    See the following example output:

    Name                                                    Release(s)                              Stemcell(s)                                        Cloud Config
    kafka-on-demand-broker                                  kafka-example-service-adapter/0+dev.2   bosh-warden-boshlite-ubuntu-trusty-go\_agent/3262.2   latest
                                                            on-demand-service-broker/0.2.0+dev.1
    service-instance_2715262c-8564-4cd9-b629-0ae99e6aa4b9   kafka-example-service/0+dev.2           bosh-warden-boshlite-ubuntu-trusty-go\_agent/3262.2   latest
    

    This example shows that the service instance is provisioned and the service releases are specified in the ODB deployment manifest.

Step 2: Use Your On-Demand Service

To use the service instance that you created, do the following:

  1. Clone the Kafka example app.

    git clone https://github.com/pivotal-cf-experimental/kafka-example-app.git
    

  2. Push the app.

    cd kafka-example-app
    cf push --no-start
    

  3. Bind the app to your service instance.

    cf bind-service kafka-example-app k1
    

  4. Start the app.

    cf start kafka-example-app
    

Step 3: Read and Write to Your Service Instance

Now the app runs at https:/kafka-example-app.PCF-DEV-DOMAIN. You can use it to read and write to your on-demand Kafka service instance.

For example:

  • To write data, run the following.

    curl -XPOST http://kafka-example-app.PCF-DEV-DOMAIN/queues/my-queue -d SOME-DATA
    
  • To read data, run the following.

    curl http://kafka-example-app.PCF-DEV-DOMAIN/queues/my-queue