Splunk Firehose Nozzle for PCF Architecture

Architecture Overview

The Splunk Firehose Nozzle for Pivotal Cloud Foundry (PCF) has been developed as a Pivotal Application Service app. The Nozzle subscribes to the Loggregator endpoint and writes events to an external Splunk environment.

Report Example: Basic Nozzle Architecture

Figure 1: Basic Nozzle Architecture

Inside each nozzle:

  • A Golang routine collects events from the Loggregator endpoint.
  • An in-memory queue buffers events for availability to multiple Splunk HTTP Event Collector clients.
  • The Splunk HTTP Event Collector clients run with their own Golang routines that consume events from the queue to enrich PCF events by attaching metadata fields.
  • Finally, the events are batched together and sent to Splunk via the HEC endpoint.

Report Example: Internal Nozzle Architecture

Figure 2: Nozzle Internal Architecture

Splunk Load Balancing Overview

In Splunk environments with multiple servers, each with their own configured HEC endpoints; a mechanism needs to be used to distribute these requests to all the necessary Splunk servers. The collection of HEC endpoints can reside on either a cluster of Splunk Heavy Forwarders or Splunk Indexers depending on the specific environment configuration.

A physical or logical load balancer is used to perform this function. If using a load balancer, the address of the load balancer needs to be configured in the setting for HTTP Event Collector Endpoint (HEC) URL found on the Splunk Settings configuration page.

Performance and Sizing Guide

Splunk has tested the Splunk Firehose Nozzle running as a single deployed nozzle on the AWS instance type c4.4xlarge. This EC2 instance type has 8 CPU and 32 GB memory. Storage is EBS-Only and has dedicated EBS Bandwidth of 2,000 Mbps.

Tests were performed with structured and unstructured data with 2 different event sizes 256 and 1024 bytes. Averaged performance metrics over an extended period of time resulted in the following results:

Tests Events per Second (EPS) Mbps Nozzle CPU (%) Nozzle Mem Total (MB)
Structured Data (JSON) - 256 byte event size 9444 7.30 530.6 57.2
Structured Data (JSON) - 1024 byte event size 6775 10.90 567.9 79.9
Unstructured Data - 256 byte event size 11189 9.10 502.3 58.3
Unstructured Data - 1024 byte event size 9487 16.03 491.5 82.8

Note: Nozzle CPU (%) 100% means that a single core is wholly dedicated to the Nozzle.
500% means five cores are wholly dedicated.
Factoring these optimal testing numbers, Splunk recommends that to run a single Splunk nozzle instance, you use a system with at least 8 CPUs and 300 MB RAM.

Note: Performance results should be taken as a guide only as different configurations and environments may vary results.

Create a pull request or raise an issue on the source for this page in GitHub