Nozzles

This topic explains how to build a nozzle that directs desired data from the aggregated Firehose of component and app logs and metrics to services or apps that consume them.

Overview

Cloud Foundry’s logging system, Loggregator, has a feature called Firehose. The Firehose includes the combined stream of logs and metrics from every apps as well as the Cloud Foundry platform itself. Building a nozzle could be a solution for

Firehose-to-syslog is a real world, production example of a nozzle.

Develop a Nozzle

Developing a nozzle should be done in Go, as this allows leveraging the NOAA library. NOAA does the heavy lifting of establishing an authenticated websocket connection to the logging system as well as de-serializing the protocol buffers.

Draining the logs consists of:

  1. Authenticating
  2. Establishing a connection to the logging system
  3. Forwarding events on to their ultimate destination

Authenticate by fetching a token from uaa with a client having the doppler.firehose scope:

    uaaClient, err := uaago.NewClient(uaaUrl)
    if err != nil {
        panic(err)
    }

    token, err := uaaClient.GetAuthToken(username, password, skipSSL)
    if err != nil {
        panic(err)
    }

Using the token, create a consumer and connect to the Firehose with a subscription id. The id is important, since the Firehose looks for connections having the same id and only sends an event to one of those connections. This is how a nozzle developer can prevent message loss during upgrades an other deployments: run at least two instances.

    consumer := consumer.New(config.TrafficControllerURL, &tls.Config{
        InsecureSkipVerify: config.SkipSSL,
    }, nil)
    events, errors := consumer.Firehose(firehoseSubscriptionID, token)

Firehose will give back two channels: one for events and a second for errors.

The events channel receives six different types of events.

  • ValueMetric: Some platform metric at a point in time, emitted by platform components. For example, how many 2xx responses the router has sent out.
  • CounterEvent: An incrementing counter, emitted by platform components. For example, a Diego cell’s remaining memory capacity.
  • Error: An error.
  • HttpStartStop: HTTP request details, including both application and platform requests.
  • LogMessage: A log message for an individual app.
  • ContainerMetric: Application container information. For example, memory used.

For the full details on events, see the dropsonde protocol.

The above events show how this data targets two different personae: platform operators and application developers. Keep this in mind when designing an integration.

Having doppler.firehose scope gets a nozzle data for every application as well as the platform. Any filtering based on the event payload is the nozzle implementor’s responsibility. An advanced integration could do something like combine a service broker with a nozzle to:

  • Let application developers opt-in to logging (implementing filtering in the nozzle)
  • Establish SSO exchange for authentication such that developers only can access logs for their space’s apps

For a full working example, see firehose-nozzle.

Deploy a Nozzle

Once you’ve build a nozzle, you can deploy it as either a managed service or as an app.

As a Managed Service

Visit managed service for more details on what it means to be a managed service.

See also this example nozzle BOSH release.

As an App

You can also deploy the nozzle as an app on Elastic Runtime. Visit the Tile Generator’s section on pushed applications for more details.

Example Nozzles

There are several open source examples you could use as a reference for building your nozzle

firehose-nozzle

  • Example that simply writes to standard out
  • Useful starting point: scaffolding, tests, etc are in place

example-nozzle

  • A single file implementation with no tests: as minimal as things can get

gcp-tools-release

  • In addition to Nozzle data, it drains component syslogs and health data
  • Shows how to do a bosh-addon (for additional data outside a nozzle)
  • Nozzle is managed via bosh
  • Raw logs and metrics data take different paths in the source

firehose-to-syslog

  • Includes implementation code that adds additional metadata (potentially needed for acl)
    • Application name
    • Space guid & name
    • Org guid & name
  • logsearch-for-cloudfoundry packages this nozzle as a BOSH release

datadog-firehose-nozzle

  • A simpler implementation than other real examples

Other Resources

Create a pull request or raise an issue on the source for this page in GitHub