Deploying a Nozzle to the Loggregator Firehose

Page last updated:

This topic describes deploying a nozzle application to the Cloud Foundry (CF) Loggregator Firehose. The Cloud Foundry Loggregator team created an example nozzle application for use with this tutorial.

The procedure described below deploys this example nozzle to the Firehose of a Cloud Foundry installation deployed locally with BOSH Lite. For more information about BOSH Lite, see the BOSH Lite GitHub repository.

Prerequisites

Note: Deploying Cloud Foundry can take up to several hours, depending on your internet bandwith, even when using the automated provision_cf script.

Step 1: Download Cloud Foundry BOSH Manifest

  1. Run bosh -e MY-ENV deployments to identify the names of all deployments in the environment you specify. Replace MY-ENV with the alias you set for your BOSH Director. For example:

    $ bosh -e dev deployments
    Using environment '192.168.15.4' as client 'admin'
    
    Name                                Release(s)                Stemcell(s)                                         Team(s)  Cloud Config
    cf-example                          binary-buildpack/1.0.9    bosh-warden-boshlite-ubuntu-trusty-go_agent/3363.9  -        latest
                                        capi/1.21.0
                                        cf-mysql/34
                                        cf-smoke-tests/11
                                        cflinuxfs2-rootfs/1.52.0
                                        consul/155
                                        diego/1.8.1
                                        etcd/94
                                        garden-runc/1.2.0
                                        loggregator/78
                                        nats/15
                                        routing/0.145.0
                                        statsd-injector/1.0.20
                                        uaa/25
    service-instance_0d4140a0-42b7-...  mysql/0.6.0               bosh-warden-boshlite-ubuntu-trusty-go_agent/3363.9  -        latest
    
    2 deployments
    
    Succeeded
    
  2. Run bosh -e MY-ENV -d MY-DEPLOYMENT manifest > MY-MANIFEST.yml to download and save the current BOSH deployment manifest. Replace MY-ENV with your BOSH Director alias, MY-DEPLOYMENT with the deployment name from the output of the previous step, and MY-MANIFEST.yml with a name you choose for the saved manifest file. You need this manifest to locate information about your databases.

    $ bosh -e dev -d cf-example manifest cf.yml

Step 2: Add UAA client

You must authorize the example nozzle as a UAA client for your CF deployment. To do this, add an entry for the example nozzle as client for uaa under the properties key in your CF deployment manifest. You must enter the example nozzle object in the correct location in the manifest, and with the correct indentation, as described below.

Deployment manifests are YAML files. Visit YAML to learn about YAML syntax.

  1. Open the deployment manifest in a text editor.
  2. Locate the left-aligned properties key.
  3. Under the properties key, locate uaa at the next level of indentation.
  4. Under the uaa key, locate the clients key at the next level of indentation.
  5. Enter properties for the example-nozzle at the next level of indentation, exactly as shown below. The ... in the text below indicate other properties that may populate the manifest at each level in the hierarchy.

    properties:
      ...
      uaa:
      ...
        clients:
        ...
          example-nozzle:
            access-token-validity: 1209600
            authorized-grant-types: client_credentials
            override: true
            secret: example-nozzle
            authorities: oauth.login,doppler.firehose
    
  6. Save the deployment manifest file.

Step 3: Redeploy Cloud Foundry

  1. Use the bosh deployment command to set the edited manifest file for your deployment.

    $ bosh deployment cf.yml
    Deployment set to `/Users/example_user/workspace/bosh-lite/cf.yml'
    

  2. Deploy Cloud Foundry with BOSH.

    $ bosh deploy
    Acting as user 'admin' on deployment 'cf-warden' on 'Bosh Lite Director'
    Getting deployment properties from director...
    Detecting deployment changes ---------------------------- Releases No changes
    Compilation No changes
    Update No changes
    Resource pools No changes
    Disk pools No changes
    Networks No changes
    Jobs No changes
    Properties uaa clients example-nozzle + access-token-validity: 1209600 + authorized-grant-types: authorization_code,client_credentials,refresh_token + override: true + secret: example-nozzle + scope: openid,oauth.approvals,doppler.firehose + authorities: oauth.login,doppler.firehose
    Meta No changes
    Please review all changes carefully
    Deploying --------- Are you sure you want to deploy? (type 'yes' to continue):yes

Step 4: Clone Example Release

The Cloud Foundry Loggregator team created an example nozzle application for use with this tutorial.

  1. Run git clone to clone the main release repository from GitHub.

    $ git clone git@github.com:cloudfoundry-incubator/example-nozzle-release.git
    Cloning into 'example-nozzle-release'...
    

  2. Run git submodule update --init --recursive to update all of the included submodules.

    $ git submodule update --init --recursive
    Submodule 'src/github.com/cloudfoundry-incubator/example-nozzle' (git@github.com:cloudfoundry-incubator/example-nozzle.git) registered for path 'src/github.com/cloudfoundry-incubator/example-nozzle'
    Submodule 'src/github.com/cloudfoundry-incubator/uaago' (git@github.com:cloudfoundry-incubator/uaago.git) registered for path 'src/github.com/cloudfoundry-incubator/uaago'
    ...
    Cloning into 'src/github.com/cloudfoundry-incubator/example-nozzle'...
    ...
    

  3. Navigate to the example-release directory.

    $ cd example-nozzle-release
    

Step 5: Prepare Nozzle Manifest

Complete the following steps to prepare the nozzle deployment manifest:

  1. In the example-nozzle-release directory, navigate to the templates directory.

    $ cd templates
    
    Within this directory, examine the two YAML files. bosh-lite-stub.yml contains the values used to populate the missing information in template.yml. By combining these two file, we create a deployment manifest for our nozzle.

  2. Create a tmp directory for the compiled manifest.

  3. Use Spiff to compile a deployment manifest from the template and stub, and save this manifest.

    $ spiff merge templates/template.yml templates/bosh-lite-stub.yml > tmp/manifest_bosh_lite.yml
    

  4. Run bosh status --uuid to obtain your BOSH director UUID.

    $ bosh status --uuid
    

  5. In the compiled nozzle deployment manifest, locate the director_uuid property. Replace PLACEHOLDER-DIRECTOR-UUID with your BOSH director UUID.

    compilation:
    cloud_properties:
      name: default
    network: example-nozzle-net
    reuse_compilation_vms: true
    workers: 1
    director_uuid: PLACEHOLDER-DIRECTOR-UUID  # replace this
    

Note: If you do not want to see the complete deployment procedure, run the following command to automatically prepare the manifest: scripts/make_manifest_spiff_bosh_lite

Step 6: Set Nozzle Deployment Manifest

Use the bosh deployment command to set the deployment manifest for the nozzle.

  $ bosh deployment tmp/manifest_bosh_lite.yml
  Deployment set to `/Users/example_user/workspace/example-nozzle-release/templates/tmp/manifest_bosh_lite.yml'

Step 7: Create Nozzle BOSH Release

Use the bosh create release --name RELEASE-NAME command to create a BOSH release. Replace RELEASE-NAME with example-nozzle to match the UAA client that you created in the CF deployment manifest.

  $ bosh create release --name example-nozzle
  Syncing blobs...
  ...

Step 8: Upload Nozzle BOSH Release

Run bosh upload release to upload the release that you created in Step 7: Create Nozzle BOSH Release.

  $ bosh upload release
  Acting as user 'admin' on 'Bosh Lite Director'
Copying packages ---------------- example-nozzle golang1.7
Copying jobs ------------ example-nozzle
Generated /var/folders/4n/qs1rjbmd1c5gfb78m3_06j6r0000gn/T/d20151009-71219-17a5m49/d20151009-71219-rts928/release.tgz Release size: 59.2M
Verifying release... ... Release info ------------ Name: nozzle-test Version: 0+dev.2
Packages - example-nozzle (b0944f95eb5a332e9be2adfb4db1bc88f9755894) - golang1.7 (b68dc9557ef296cb21e577c31ba97e2584a5154b)
Jobs - example-nozzle (112e01c6ee91e8b268a42239e58e8e18e0360f58)
License - none
Uploading release

Step 9: Deploy Nozzle

Run bosh deploy to deploy the nozzle.

  $ bosh deploy
  Acting as user 'admin' on deployment 'example-nozzle-lite' on 'Bosh Lite Director'
  Getting deployment properties from director...
  Unable to get properties list from director, trying without it...
  Cannot get current deployment information from director, possibly a new deployment
  Please review all changes carefully
Deploying --------- Are you sure you want to deploy? (type 'yes' to continue):yes

Step 10: View Nozzle Output

The example nozzle outputs all of the data originating coming from the Firehose to its log files. To view this data, SSH into the example-nozzle VM and examine the logs.

  1. Run bosh ssh to access the nozzle VM at the IP configured in the nozzle’s manifest template stub ./templates/bosh-lite-stub.yml.

    $ bosh ssh example-nozzle
    Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.19.0-25-generic x86_64)
    Documentation: https://help.ubuntu.com/
    Last login: Wed Sep 23 21:29:50 2015 from 192.0.2.1
    

  2. Use the cat command to output the stdout log file.

    $ cat /var/vcap/sys/log/example-nozzle/example-nozzle.stdout.log
    ===== Streaming Firehose (will only succeed if you have admin credentials)
    origin:"DopplerServer" eventType:ValueMetric timestamp:1443046217700750747 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910193187 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910360012 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910252169 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910294255 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910318582 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910339088 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910379199 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910394886 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"router__0" eventType:HttpStartStop timestamp:1443046219105062148 deployment:"cf-warden" job:"router_z1" index:"0" ip:"203.0.113.22" httpStartStop: peerType:Client method:POST uri:"uaa.service.cf.internal:8080/oauth/token" remoteAddress:"" userAgent:"" statusCode:200 contentLength:-1 >
    origin:"api_z1_0" eventType:HttpStartStop timestamp:1443046219109842455 deployment:"cf-warden" job:"api_z1" index:"0" ip:"203.0.113.134" httpStartStop: peerType:Server method:GET uri:"routing-api.service.cf.internal:3000/v1/routes" remoteAddress:"203.0.113.15:49577" userAgent:"Go 1.1 package http" statusCode:200 contentLength:108 >
    origin:"router__0" eventType:HttpStartStop timestamp:1443046219110064368 deployment:"cf-warden" job:"router_z1" index:"0" ip:"203.0.113.22" httpStartStop: peerType:Client method:GET uri:"routing-api.service.cf.internal:3000/v1/routes" remoteAddress:"" userAgent:"" statusCode:200 contentLength:108 >
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177165446 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177288325 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177346726 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177274975 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177310389 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177330214 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177353454 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177360052 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177481456 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880585603 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880895040 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.146" counterEvent:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881017888 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881011670 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880929574 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881004417 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:
    origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880929568 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:
    origin:"MetronAgent" eventType:CounterEvent timestamp:1443046220058280679 deployment:"cf-warden" job:"api_z1" index:"0" ip:"203.0.113.134" counterEvent:
    
Create a pull request or raise an issue on the source for this page in GitHub