LATEST VERSION: 0.17.2 - CHANGELOG

Getting Started: ODB on a Local Development Environment

This guide describes how to create and manage an on-demand service using PCF Dev and BOSH lite, which are tools that allow you run BOSH and Pivotal Cloud Foundry on a local development machine. This tutorial bases its example service on Kafka open source messaging and uses the following sample code directories:

Prerequisites

Before setting up and using ODB on your local development environment, install and configure the following components:

Note: For PCF Dev to route requests to the deployments on BOSH Lite ensure you run the script bin/add-route in the BOSH Lite repository. You may need to run this again if your networking is reset (e.g. reboot, or connecting to a different network).

      _______  _______  _______    ______   _______  __   __
    |       ||       ||       |  |      | |       ||  | |  |
    |    _  ||       ||    ___|  |  _    ||    ___||  |_|  |
    |   |_| ||       ||   |___   | | |   ||   |___ |       |
    |    ___||      _||    ___|  | |_|   ||    ___||       |
    |   |    |     |_ |   |      |       ||   |___  |     |
    |___|    |_______||___|      |______| |_______|  |___|
    is now running.
    To begin using PCF Dev, please run:
       cf login -a https://api.local.pcfdev.io --skip-ssl-validation
    Apps Manager URL: https://local.pcfdev.io
    Admin user => Email: admin / Password: admin

Make a note of the PCF Dev domain, you will need this later on. The default is local.pcfdev.io.

Step 1: Set Up BOSH Lite

  1. Target your BOSH Lite installation:

    $ bosh target
    Current target is https://192.168.50.4:25555 (Bosh Lite Director)
    

  2. Upload the BOSH Lite stemcell:

    $ bosh upload stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent?v=3262.2
    

Step 2: Set Up the Kafka Example Service

  1. Clone the Kafka example service into your workspace:

    $ git clone https://github.com/pivotal-cf-experimental/kafka-example-service-release.git
    

  2. In the kafka-example-service-release directory, create and upload the kafka example service:

    $ cd kafka-example-service-release
    $ bosh create release --name kafka-example-service
    

  3. Upload the service to the BOSH director:

    $ bosh upload release
    

Step 3: Set Up the Kafka Example Service Adapter

  1. Clone the Kafka example service adapter and run git submodule update --init to bring in the adapter’s dependencies

    $ git clone https://github.com/pivotal-cf-experimental/kafka-example-service-adapter-release.git
    

  2. Update the service adapter dependencies:

    $ cd kafka-example-service-adapter-release
    $ git submodule update --init --recursive
    

  3. Create the example service adapter:

    $ bosh create release --name kafka-example-service-adapter
    

  4. Upload the example service adapter to the BOSH director:

    $ bosh upload release
    

Step 4: Set Up the On-Demand Service Broker

  1. Download the on-demand service broker from PivNet

  2. Upload the on-demand-service-broker release (replace X.Y.Z with the ODB release version):

    $ bosh upload release on-demand-service-broker-X.Y.Z.tgz
    

Step 5: Create a BOSH Deployment

  1. Create a new directory in your workspace and a cloud_config.yml for the BOSH Lite Director. For example:

    vm_types:
    - name: container
      cloud_properties: {}
    
    networks:
    - name: kafka
      type: manual
      subnets:
      - range: 10.244.1.0/24
        gateway: 10.244.1.1
        az: lite
        cloud_properties: {}
    
    disk_types:
    - name: ten
      disk_size: 10_000
      cloud_properties: {}
    
    azs:
    - name: lite
      cloud_properties: {}
    
    compilation:
      workers: 2
      reuse_compilation_vms: true
      network: kafka
      az: lite
      cloud_properties: {}
    
  2. Update the BOSH Lite cloud config using the deployment manifest created in the previous step:

    $ bosh update cloud-config cloud_config.yml
    

  3. Obtain the URL and UUID BOSH Lite director information:

    $ bosh status
    
    This command produces output similar to the following:
    → bosh status
    Config
                 /Users/pivotal/.bosh_config

    Director Name Bosh Lite Director URL https://192.168.50.4:25555 Version 1.3215.0 (00000000) User admin UUID 17a45148-1d00-43bc-af28-9882e5a6535a CPI warden_cpi dns disabled compiled_package_cache enabled (provider: local) snapshots disabled

    Record the URL and UUID from your output. You will add them to the deployment_manifest.yml in the next step.

  4. Create a BOSH Lite deployment manifest in a file called deployment_manifest.yml using the following as a base.

    Replace BOSH_LITE_UUID, BOSH_LITE_URL and PCF_DEV_DOMAIN with the values obtained in the previous steps.

    name: kafka-on-demand-broker
    
    director_uuid: <BOSH_LITE_UUID>
    
    releases:
    - name: &broker-release on-demand-service-broker
      version: latest
    - name: &service-adapter-release kafka-example-service-adapter
      version: latest
    - name: &service-release kafka-example-service
      version: latest
    
    stemcells:
    - alias: trusty
      os: ubuntu-trusty
      version: <STEMCELL_VERSION>
    
    instance_groups:
    - name: broker
      instances: 1
      vm_type: container
      persistent_disk_type: ten
      stemcell: trusty
      azs: [lite]
      networks:
      - name: kafka
      jobs:
      - name: kafka-service-adapter
        release: *service-adapter-release
      - name: admin_tools
        release: *service-release
      - name: broker
        release: *broker-release
        properties:
          port: 8080
          username: broker #or replace with your own
          password: password #or replace with your own
          disable_ssl_cert_verification: true
          bosh:
            url: <BOSH_LITE_URL>
            authentication:
              basic:
                username: admin
                password: admin
          cf:
            url: https://api.<PCF_DEV_DOMAIN>
            authentication:
              url: https://uaa.<PCF_DEV_DOMAIN>
              user_credentials:
                username: admin
                password: admin
          service_adapter:
            path: /var/vcap/packages/odb-service-adapter/bin/service-adapter
          service_deployment:
            releases:
            - name: *service-release
              version: <SERVICE_RELEASE_VERSION>
              jobs: [kafka_server, zookeeper_server]
            stemcell:
              os: ubuntu-trusty
              version: <STEMCELL_VERSION>
          service_catalog:
            id: D94A086D-203D-4966-A6F1-60A9E2300F72
            service_name: kafka-service-with-odb
            service_description: Kafka Service
            bindable: true
            plan_updatable: true
            tags: [kafka]
            plans:
            - name: small
              plan_id: 11789210-D743-4C65-9D38-C80B29F4D9C8
              description: A Kafka deployment with a single instance of each job and persistent disk
              instance_groups:
              - name: kafka_server
                vm_type: container
                instances: 1
                persistent_disk_type: ten
                azs: [lite]
                networks: [kafka]
              - name: zookeeper_server
                vm_type: container
                instances: 1
                persistent_disk_type: ten
                azs: [lite]
                networks: [kafka]
              properties:
                auto_create_topics: true
                default_replication_factor: 1
    
    update:
      canaries: 1
      canary_watch_time: 30000-180000
      update_watch_time: 30000-180000
      max_in_flight: 4
    
  5. Change the BOSH deployment to use the deployment manifest created in the previous step:

    $ bosh deployment deployment_manifest.yml
    

  6. Deploy the manifest:

    $ bosh deploy
    

  7. Obtain the IP address of the deployed broker:

    $ bosh instances
    

    This command produces output similar to the following:

    Acting as client 'admin' on deployment 'kafka-on-demand-broker' on 'Bosh Lite Director'
    
    Director task 147
    
    Task 147 done
    
    +--------------------------------------------------+---------+-----+-----------+------------+
    | Instance                                         | State   | AZ  | VM Type   | IPs        |
    +--------------------------------------------------+---------+-----+-----------+------------+
    | broker/0 (59231277-d7b8-46bb-8bbb-8154b6bae347)* | running | n/a | container | 10.244.1.2 |
    +--------------------------------------------------+---------+-----+-----------+------------+
    
    (*) Bootstrap node
    
    Instances total: 1
    

    Record the IP address of the broker. You will use this in the next step to create a service broker.

Step 6: Create a Service Broker on PCF Dev

  1. Create a service broker on PCF Dev and enable access to its service offering. You will need the broker’s credentials set in the deployment manifest and the IP of the broker VM.

    Replace BROKER_IP with the value obtained in the previous step.

    $ cf create-service-broker kafka-broker broker password http://<BROKER_IP>:8080
    

    For more details on service brokers see here.

  2. Enable access to the broker’s service plans:

    $ cf enable-service-access kafka-service-with-odb
    

  3. View the services offered by the broker in the marketplace:

    $ cf marketplace
    

    This command produces output similar to the following:

    Getting services from marketplace in org pcfdev-org / space pcfdev-space as admin...
    OK
    
    service                  plans        description
    kafka-service-with-odb   small        Kafka Service
    p-mysql                  512mb, 1gb   MySQL databases on demand
    p-rabbitmq               standard     RabbitMQ is a robust and scalable high-performance multi-protocol messaging broker.
    p-redis                  shared-vm    Redis service to provide a key-value store
    
  4. Create a service instance using the Kafka on-demand broker.

    $ cf create-service kafka-service-with-odb small k1
    

Step 7: Verify Your BOSH Deployment and On-Demand Service

  1. Check the status of your service:

    $ cf service k1
    
    Initially, the service status state is: create in progress. After the service is created, the status changes to: create succeeded.

  2. Verify that the on-demand service is provisioned in the BOSH deployment

    $ bosh deployments
    

    The output of this command appears as follows:

    +-------------------------------------------------------+---------------------------------------+--------------------------------------------------+--------------+
    | Name                                                  | Release(s)                            | Stemcell(s)                                      | Cloud Config |
    +-------------------------------------------------------+---------------------------------------+--------------------------------------------------+--------------+
    | kafka-on-demand-broker                                | kafka-example-service-adapter/0+dev.2 | bosh-warden-boshlite-ubuntu-trusty-go\_agent/3262.2 | latest       |
    |                                                       | on-demand-service-broker/0.2.0+dev.1  |                                                  |              |
    +-------------------------------------------------------+---------------------------------------+--------------------------------------------------+--------------+
    | service-instance\_2715262c-8564-4cd9-b629-0ae99e6aa4b9 | kafka-example-service/0+dev.2         | bosh-warden-boshlite-ubuntu-trusty-go\_agent/3262.2 | latest       |
    +-------------------------------------------------------+---------------------------------------+--------------------------------------------------+--------------+
    

    This example shows that the service instance is provisioned and the service releases are specified in the ODB deployment manifest.

Step 8: Use Your On-Demand Service

The Kafka Example App shows how you can use the service instance that you created with cf create-service.

  1. Clone the Kafka example app in your workspace:

    $ git clone https://github.com/pivotal-cf-experimental/kafka-example-app.git
    

  2. Push the app.

    $ cd kafka-example-app
    $ cf push --no-start
    

  3. Bind the app to your service instance:

    $ cf bind-service kafka-example-app k1
    

    Note: You can use cf bs as an alias for cf bind-service

  4. Start the app:

    $ cf start kafka-example-app
    

Now the app runs at https:/kafka-example-app.<PCF_DEV_DOMAIN>, and you can use it to read and write to your on-demand Kafka service instance. For example:

  • To write data, run curl -XPOST http://kafka-example-app.<PCF_DEV_DOMAIN>/queues/my-queue -d SOME-DATA
  • To read data, run curl http://kafka-example-app.<PCF_DEV_DOMAIN>/queues/my-queue
Create a pull request or raise an issue on the source for this page in GitHub