Deploying a Nozzle to the Loggregator Firehose

Page last updated:

This topic describes deploying a nozzle app to the Pivotal Application Service (PAS) Loggregator Firehose. For more information about nozzles and the Loggregator Firehose, see Loggregator Architecture. The PAS Loggregator team created an example nozzle app for use with this tutorial.

Overview

The procedure described below deploys this example nozzle to the Firehose of a PAS installation deployed locally with BOSH Lite. For more information about BOSH Lite, see the BOSH Lite repository on GitHub.

To lighten the load on custom nozzles you develop, you can request Firehose subscriptions that emit only whitelisted metrics. For examples, see rlpreader and rlptypereader in the loggregator-tools repository and V2 Subscriptions in the loggregator-release repository.

Prerequisites

Before you deploy a nozzle to the Loggregator Firehose, you must have:

  • BOSH CLI. For more information, see BOSH CLI in the BOSH documentation.

  • Spiff installed locally and added to the load path of your shell. For more information, see the Spiff repository on GitHub.

  • BOSH Lite deployed locally using VirtualBox. For more information, see the BOSH Lite repository on GitHub.

  • A working PAS deployment, including Loggregator, deployed with your local BOSH Lite. This serves as the source of data. For more information, see Deploying Cloud Foundry with BOSH Lite v2, or use the provision_cf script included in the BOSH Lite release.

Note: Deploying PAS can take several hours depending on your Internet bandwith, even when using the automated provision_cf script.

Step 1: Download Cloud Foundry BOSH Manifest

To download the BOSH manifest:

  1. Run bosh -e MY-ENV deployments to identify the names of all deployments in the environment you specify. Replace MY-ENV with the alias you set for your BOSH Director. For example:

    $ bosh -e dev deployments
    Using environment '192.168.15.4' as client 'admin'
    
    Name                                Release(s)                Stemcell(s)                                         Team(s)  Cloud Config
    cf-example                          binary-buildpack/1.0.9    bosh-warden-boshlite-ubuntu-trusty-go_agent/3363.9  -        latest
                                        capi/1.21.0
                                        cf-mysql/34
                                        cf-smoke-tests/11
                                        cflinuxfs3-rootfs/1.52.0
                                        consul/155
                                        diego/1.8.1
                                        etcd/94
                                        garden-runc/1.2.0
                                        loggregator/78
                                        nats/15
                                        routing/0.145.0
                                        statsd-injector/1.0.20
                                        uaa/25
    service-instance_0d4140a0-42b7-...  mysql/0.6.0               bosh-warden-boshlite-ubuntu-trusty-go_agent/3363.9  -        latest
    
    2 deployments
    
    Succeeded
    
  2. Run bosh -e MY-ENV -d MY-DEPLOYMENT manifest > MY-MANIFEST.yml to download and save the current BOSH deployment manifest. Replace MY-ENV with your BOSH Director alias, MY-DEPLOYMENT with the deployment name from the output of the previous step, and MY-MANIFEST.yml with a name you choose for the saved manifest file. You need this manifest to locate information about your databases.

    $ bosh -e dev -d cf-example manifest > cf.yml

Step 2: Add UAA Client

You must authorize the example nozzle as a UAA client for your PAS deployment. To do this, add an entry for the example nozzle as client for uaa under the properties key in your PAS deployment manifest YAML file. You must enter the example nozzle object in the correct location in the manifest, and with the correct indentation.

To add the nozzle as a UAA client for your deployment:

  1. Open the deployment manifest in a text editor.

  2. Locate the left-aligned properties key.

  3. Under the properties key, locate uaa at the next level of indentation.

  4. Under the uaa key, locate the clients key at the next level of indentation.

  5. Enter properties for the example-nozzle at the next level of indentation, exactly as shown below. The ... in the text below indicate other properties that may populate the manifest at each level in the hierarchy.

    properties:
      ...
      uaa:
      ...
        clients:
        ...
          example-nozzle:
            access-token-validity: 1209600
            authorized-grant-types: client_credentials
            override: true
            secret: example-nozzle
            authorities: oauth.login,doppler.firehose
    
  6. Save the deployment manifest file.

Step 3: Redeploy PAS

To deploy PAS with BOSH, run:

bosh -e MY-ENV deploy

WHERE MY-ENV is the alias you set for your BOSH Director.

For example:

  $ bosh -e dev deploy
  Acting as user 'admin' on deployment 'cf-warden' on 'Bosh Lite Director'
  Getting deployment properties from director...
Detecting deployment changes ---------------------------- Releases No changes
Compilation No changes
Update No changes
Resource pools No changes
Disk pools No changes
Networks No changes
Jobs No changes
Properties uaa clients example-nozzle + access-token-validity: 1209600 + authorized-grant-types: authorization_code,client_credentials,refresh_token + override: true + secret: example-nozzle + scope: openid,oauth.approvals,doppler.firehose + authorities: oauth.login,doppler.firehose
Meta No changes
Please review all changes carefully
Deploying --------- Are you sure you want to deploy? (type 'yes' to continue):yes

Step 4: Clone Example Release

The PAS Loggregator team created an example nozzle app for use with this tutorial.

To clone the example nozzle release:

  1. Run git clone to clone the main release repository from the example-nozzle-release repository on GitHub.

    $ git clone https://github.com/cloudfoundry-incubator/example-nozzle-release.git
    Cloning into 'example-nozzle-release'...
    

  2. Navigate to the example-nozzle-release directory by running:

  cd example-nozzle-release
  1. Run git submodule update --init --recursive to update all of the included submodules.
    $ git submodule update --init --recursive
    Submodule 'src/github.com/cloudfoundry-incubator/example-nozzle' (git@github.com:cloudfoundry-incubator/example-nozzle.git) registered for path 'src/github.com/cloudfoundry-incubator/example-nozzle'
    Submodule 'src/github.com/cloudfoundry-incubator/uaago' (git@github.com:cloudfoundry-incubator/uaago.git) registered for path 'src/github.com/cloudfoundry-incubator/uaago'
    ...
    Cloning into 'src/github.com/cloudfoundry-incubator/example-nozzle'...
    ...
    

Step 5: Prepare Nozzle Manifest

To prepare the nozzle deployment manifest:

  1. In the example-nozzle-release directory, navigate to the templates directory by running:

    cd templates
    

    Within this directory, examine the two YAML files. bosh-lite-stub.yml contains the values used to populate the missing information in template.yml. By combining these two files, you can create a deployment manifest for the nozzle.

  2. Create a tmp directory for the compiled manifest.

  3. Use Spiff to compile a deployment manifest from the template and stub by running:

    spiff merge templates/template.yml templates/bosh-lite-stub.yml > tmp/manifest_bosh_lite.yml
    

    Save this manifest.

  4. Run bosh -e MY-ENV deployments to identify the names of all deployments in the environment you specify. Replace MY-ENV with the alias you set for your BOSH Director.

  5. Run bosh -e MY-ENV env --column=uuid to obtain your BOSH Director UUID. Replace MY-ENV with the alias you set for your BOSH Director. For example:

    bosh -e dev env --column=uuid
    
  6. In the compiled nozzle deployment manifest, locate the director_uuid property. Replace PLACEHOLDER-DIRECTOR-UUID with your BOSH Director UUID.

    compilation:
      cloud_properties:
        name: default
    network: example-nozzle-net
    reuse_compilation_vms: true
    workers: 1
    director_uuid: PLACEHOLDER-DIRECTOR-UUID
    

    Where PLACEHOLDER-DIRECTOR-UUID is your BOSH Director UUID.

Note: If you do not want to see the complete deployment procedure, run scripts/make_manifest_spiff_bosh_lite to automatically prepare the manifest.

Step 6: Create Nozzle BOSH Release

To create a nozzle BOSH release, run:

bosh -e MY-ENV create-release --name RELEASE-NAME

WHERE: * MY-ENV is the alias you set for your BOSH Director. * RELEASE-NAME is example-nozzle to match the UAA client that you created in the PAS deployment manifest.

For example:

  $ bosh -e dev create-release --name example-nozzle
  Syncing blobs...
  ...

Step 7: Upload Nozzle BOSH Release

Upload the nozzle BOSH release that you created as part of Step 6: Create Nozzle BOSH Release.

To upload the BOSH release, run:

bosh -e MY-ENV upload-release

WHERE MY-ENV is the alias you set for your BOSH Director.

For example:

  $ bosh -e dev upload-release
  Acting as user 'admin' on 'Bosh Lite Director'
Copying packages ---------------- example-nozzle golang1.7
Copying jobs ------------ example-nozzle
Generated /var/folders/4n/qs1rjbmd1c5gfb78m3_06j6r0000gn/T/d20151009-71219-17a5m49/d20151009-71219-rts928/release.tgz Release size: 59.2M
Verifying release... ... Release info ------------ Name: nozzle-test Version: 0+dev.2
Packages - example-nozzle (b0944f95eb5a332e9be2adfb4db1bc88f9755894) - golang1.7 (b68dc9557ef296cb21e577c31ba97e2584a5154b)
Jobs - example-nozzle (112e01c6ee91e8b268a42239e58e8e18e0360f58)
License - none
Uploading release

Step 8: Deploy Nozzle

To deploy the nozzle, run:

bosh -e MY-ENV deploy

WHERE MY-ENV is the alias you set for your BOSH Director.

For example:

  $ bosh -e dev deploy
  Acting as user 'admin' on deployment 'example-nozzle-lite' on 'Bosh Lite Director'
  Getting deployment properties from director...
  Unable to get properties list from director, trying without it...
  Cannot get current deployment information from director, possibly a new deployment
  Please review all changes carefully
Deploying --------- Are you sure you want to deploy? (type 'yes' to continue):yes

Step 9: View Nozzle Output

The example nozzle outputs all of the data originating coming from the Firehose to its log files. To view this data, SSH into the example-nozzle VM and examine the logs.

To view nozzle output:

  1. Run bosh -e MY-ENV ssh to access the nozzle VM at the IP configured in the nozzle’s manifest template stub ./templates/bosh-lite-stub.yml. Replace MY-ENV with the alias you set for your BOSH Director. For example:

    $ bosh -e dev ssh example-nozzle
    Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.19.0-25-generic x86_64)
    Documentation: https://help.ubuntu.com/
    Last login: Wed Sep 23 21:29:50 2015 from 192.0.2.1
    

  2. Use the cat command to output the stdout log file.

  $ cat /var/vcap/sys/log/example-nozzle/example-nozzle.stdout.log
  ===== Streaming Firehose (will only succeed if you have admin credentials)
  origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.swap.percent" value:0 unit:"Percent" >
  origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.swap.kb" value:0 unit:"Kb" >
  origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.disk.ephemeral.percent" value:3 unit:"Percent" >
  origin:"bosh-system-metrics-forwarder" eventType:ValueMetric timestamp:1541091851000000000 deployment:"cf-c42ae2c4dfb6f67b6c27" job:"loggregator_trafficcontroller" index:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" ip:"" tags:>key:"id" value:"d1dffe15-5894-44de-b7f0-ad43161a0a7b" > tags:>key:"source_id" value:"bosh-system-metrics-forwarder" > valueMetric:>"system.healthy" value:1 unit:"b" >
  origin:"gorouter" eventType:ValueMetric timestamp:1541091851218590916 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"router" index:"d5d1b5a4-2497-4679-8d3b-66ffc978d829" ip:"10.0.4.13" tags:>key:"source_id" value:"gorouter" > valueMetric:>"uptime" value:3.273478e+06 unit:"seconds" >
  origin:"netmon" eventType:ValueMetric timestamp:1541091851234217334 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>name:"numGoRoutines" value:13 unit:"count" >
  origin:"netmon" eventType:ValueMetric timestamp:1541091851234129669 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>"numCPUS" value:2 unit:"count" >
  origin:"netmon" eventType:ValueMetric timestamp:1541091851234292367 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>"memoryStats.numBytesAllocated" value:542328 unit:"count" >
  origin:"netmon" eventType:ValueMetric timestamp:1541091851234279470 deployment:"cf-c56ab7c4dfb6f67b6c28" job:"diego_cell" index:"8007afda-3bff-4856-857f-a47a43cbf994" ip:"10.0.4.18" tags:>key:"source_id" value:"netmon" > valueMetric:>"memoryStats.numBytesAllocatedStack" value:655360 unit:"count" >
  . . .