Loggregator Architecture

Page last updated:

This topic describes the Loggregator architecture and components. It also describes the Pivotal Cloud Foundry (PCF) components that send BOSH-reported VM metrics to Loggregator.

Overview

Loggregator gathers and streams logs and metrics from user apps in a PCF deployment. It also gathers and streams metrics from PCF components and health metrics from BOSH-deployed VMs. Loggregator allows you to view these logs and metrics through the Loggregator CLI plugins or through a third-party service. For more information, see loggregator in GitHub.

The Loggregator architecture includes components for collecting, storing, and forwarding logs and metrics.

Loggregator Architecture and Components

The diagram below shows the architecture of Loggregator, including the PCF components with which it interacts.

Loggregatornew

View a larger version of this image.

The following are Loggregator components, as shown in the diagram above:

  • Metron Agent: Loggregator Agents run on both PCF component VMs and Diego cell VMs. They receive logs and metrics from the apps and PCF components located on those VMs. Loggregator Agents then forward the logs and metrics to Dopplers.
  • Doppler: Dopplers receive logs and metrics from Loggregator Agents, store them in temporary buffers, and forward them to Traffic Controllers.
  • Traffic Controller: Traffic Controllers poll Doppler servers for logs and metrics, then translate these messages from the Doppler servers as necessary for external and legacy APIs. The Loggregator Firehose is located on the Traffic Controller.
  • Reverse Log Proxy: Reverse Log Proxies (RLPs) collect logs and metrics from Dopplers and forward them to Syslog Adapters for syslog drains. Operators can scale up the number of RLPs based on overall log volume.
  • Syslog Adapter: Syslog Adapters manage connections with and write to syslog drains. Operators can scale the Syslog Adapter component based on the number of syslog drains.
  • Firehose: The Firehose is a WebSocket endpoint that streams all the event data from a PCF deployment. The data stream includes HTTP events, app logs, container metrics from apps, and metrics from PCF platform components. The Firehose cf CLI plugin allows you to view the output of the Firehose. For more information about the Firehose plugin, see Installing the Loggregator Firehose Plugin for cf CLI.
  • Log Cache: The Log Cache allows you to view logs and metrics from the Firehose over a specified period of time. The Log Cache includes API endpoints and a CLI plugin to query and filter logs and metrics. To download the Log Cache CLI plugin, see Cloud Foundry Plugins. The Log Cache API endpoints are available by default. For more information about using the Log Cache API, see Log Cache on GitHub.
  • Nozzles: Nozzles are programs that consume data from the Loggregator Firehose. Nozzles can be configured to select, buffer, and transform data, and to forward it to other apps and services. For more information about creating and deploying nozzles, see Deploying a Nozzle to the Loggregator Firehose.

This section describes the PCF components that forward BOSH-reported VM metrics to Loggregator. BOSH-reported VM metrics measure the health of BOSH-deployed VMs on which apps and PCF components are deployed. Loggregator streams BOSH-reported VM metrics through the Firehose.

The following are the PCF components that send BOSH-reported VM metrics to Loggregator:

  • BOSH Agent: BOSH Agents are located on component VMs and Diego cell VMs. They collect metrics, such as cell capacity remaining, from the VM and forward them to the BOSH Health Monitor.

  • BOSH Health Monitor: The BOSH Health Monitor receives metrics from the BOSH Agents. It then forwards the metrics to a third-party service or to the BOSH System Metrics Forwarder.

  • BOSH System Metrics Plugin: This plugin reads health events, such as VM heartbeats and alerts from the BOSH Health Monitor, and streams them to the BOSH System Metrics Server.

  • BOSH System Metrics Server: The BOSH System Metrics Server streams metrics and heartbeat events to the BOSH System Metrics Forwarder over gRPC.

  • BOSH System Metrics Forwarder: The BOSH System Metrics Forwarder is located on the Loggregator Traffic Controller. It forwards heartbeat events from the BOSH System Metrics Server as envelopes to Loggregator through a Loggregator Agent.