Using Splunk Firehose Nozzle for PCF

This topic describes how to use Splunk Firehose Nozzle for Pivotal Cloud Foundry (PCF).

After installing and configuring Splunk Firehose Nozzle for PCF, PCF operators can navigate to the URL of their existing Splunk Enterprise deployment to immediately search, report, visualize, and alert on PCF Firehose data.

The following assumes basic familiarity on how to run searches, save reports, and create dashboards in Splunk Enterprise. If you are new to Splunk, start with the Search Tutorial which shows you how to search data and create simple dashboards.

The following is an example Splunk dashboard for operational intelligence.

Report Example: Sample Ops Dashboard

Search Firehose Data

The Firehose event types forwarded by the Splunk Nozzle are assigned the following Splunk sourcetypes by default:

Firehose event type Splunk sourcetype
Error cf:error
HttpStartStop cf:httpstartstop
LogMessage cf:logmessage
ContainerMetric cf:containermetric
CounterEvent cf:counterevent
ValueMetric cf:valuemetric

In addition, logs from the Nozzle are of sourcetype cf:splunknozzle.

You can use these Splunk sourcetypes to search and retrieve the relevant events from the index, use statistical commands to calculate metrics and generate reports, search for specific conditions within a rolling time window, identify patterns in your data, predict future trends, and so on. Here are some sample search commands that you can use:

  • Search for any errors from your deployment:

    sourcetype=cf:error

  • Search for router bad gateways:

    sourcetype=cf:counterevent name="bad_gateways"

  • Search for metrics of remaining cell disk capacity or memory:

    sourcetype=cf:valuemetric name=CapacityRemainingDisk sourcetype=cf:valuemetric name=CapacityRemainingMemory

You must have access to the index specified to receive these Firehose events. You might need to explicitly specify the index in your search if that index is not part of the indexes searched by default for your Splunk role. These searches can be saved as reports and used to power dashboard panels, as shown in the next section.

Report and Visualize Firehose Data

To monitor your environment operational health, you can build on these same searches to generate reports and visualizations. For example:

  • Report available memory per cell:

    sourcetype=cf:valuemetric name=CapacityRemainingMemory
      | eval valueGB=round(case(unit=="MiB", value/1024, unit=="KiB", value/(1024*1024), unit=="GiB", value),2)
      | stats min(valueGB) as mem by job_instance
      | rename mem as "Available Memory (GB)", job_instance as "Job Instance"
    

    Report Example: Available mem per cell

  • Report available memory per cell over time:

    sourcetype=cf:valuemetric name=CapacityRemainingMemory
      | eval valueGB=round(case(unit=="MiB", value/1024, unit=="KiB", value/(1024*1024), unit=="GiB", value),2)
      | timechart min(valueGB) by job_instance
    

    Report Example: Available mem per cell over time

  • Report number of routes registered with trend indicator:

    sourcetype=cf:valuemetric name=RoutesTotal
      | timechart avg(value) as numRoutes
    

    Report Example: Total routes with trend

For more information about searching and reporting with Splunk, see Splunk Docs

Create a pull request or raise an issue on the source for this page in GitHub