Loggregator Architecture

This topic describes the Loggregator architecture and components. It also describes the Pivotal Cloud Foundry (PCF) components that send BOSH-reported VM metrics to Loggregator.

Overview

Loggregator gathers and streams logs and metrics from user apps in a PCF deployment. It also gathers and streams metrics from PCF components and health metrics from BOSH-deployed VMs. Loggregator allows you to view these logs and metrics through the Loggregator CLI plugins or through a third-party service. For more information, see loggregator in GitHub.

The Loggregator architecture includes components for collecting, storing, and forwarding logs and metrics.

Loggregator Architecture and Components

This section includes diagrams of Loggregator architecture.

In Pivotal Application Service (PAS) v2.6, platform operators can deploy a Loggregator architecture that uses either Syslog Adapters or Syslog Agents. Selecting Syslog Agents increases the number of syslog drain service bindings and reduces the workload for Loggregator VMs.

Platform operators can also enable System Metrics Agents on all VMs deployed with PAS to collect system-level metrics.

For more information, see Loggregator Syslog Agent Increases Scale For Syslog Drains and Monitor System Metrics with System Metrics Agent in the PAS v2.6 Release Notes.

Loggregator Architecture with Syslog Adapter

The following diagram shows the architecture of a Loggregator deployment that uses Syslog Adapter.

A Loggregator Agent appears in squares that depict 'Host VM' and 'Cloud Controller VMs'. Represented by arrows, the Loggregator Agent receives logs from a statsd Injector, which communicates with components like Diego, Cloud Controller, UAA. Loggregator Agents also obtain logs from Diego Executors. Loggregator Agents send logs to Dopplers over gRPC communication. Dopplers then forward the logs to Traffic Controllers and Reverse Log Proxies. Reverse Log Proxies send logs to Log Caches and Syslog Adapters. For more descriptions of all of the components in Loggregator, see the 'Loggregator Components' section below.

View a larger version of this image.

Loggregator Architecture with Syslog Agents

The following diagram shows the architecture of a Loggregator deployment that uses Syslog Agents.

A Loggregator Agent appears in squares that depict 'Host VM' and 'Cloud Controller VMs'. Represented by arrows, the Loggregator Agent receives logs from a Forwarder Agent, which forwards logs received from a Syslog Agent. Loggregator Agents send the logs to Dopplers over gRPC communication. Dopplers then forward the logs to Traffic Controllers. For more descriptions of all of the components in Loggregator, see the 'Loggregator Components' section below.

View a larger version of this image.

Loggregator Architecture with System Metrics Agents

The following diagram shows the architecture of a Loggregator deployment that uses System Metrics Agents to collect VM and system-level metrics.

System Metrics Agent appear in squares that depict 'Host VMs'. Represented by arrows, the System Metrics Agents send VM system-level metrics to a System Metrics Scraper, which forwards these metrics to Loggregator Agents over mTLS. For more descriptions of all of the components in Loggregator, see the 'Loggregator Components' section below.

View a larger version of this image.

Loggregator Components

The following are Loggregator components, as shown in the diagrams above:

  • Loggregator Agent: Loggregator Agents run on both PCF component VMs and Diego cell VMs. They receive logs and metrics from the apps and PCF components located on those VMs. Loggregator Agents then forward the logs and metrics to Dopplers.
  • System Metrics Agent: A standalone agent to provide VM system metrics using a Prometheus-scrapable endpoint.
  • Doppler: Dopplers receive logs and metrics from Loggregator Agents, store them in temporary buffers, and forward them to Traffic Controllers.
  • Traffic Controller: Traffic Controllers poll Doppler servers for logs and metrics, then translate these messages from the Doppler servers as necessary for external and legacy APIs. The Loggregator Firehose is located on the Traffic Controller.
  • Reverse Log Proxy: Reverse Log Proxies (RLPs) collect logs and metrics from Dopplers and forward them to Syslog Adapters for syslog drains. Operators can scale up the number of RLPs based on overall log volume.
  • Syslog Adapter: Syslog Adapters manage connections with and write to syslog drains. Operators can scale the Syslog Adapter component based on the number of syslog drains.
  • Syslog Agents: When used instead of Syslog Adapters, Syslog Agents increase the number of syslog drain service bindings supported by the Loggregator system and reduce the workload for Loggregator VMs. Agents run on the Loggregator Trafficcontroller VM.
  • Syslog Binding Cache: Syslog Agents can overwhelm CAPI when querying for existing bindings. This component acts a a proxy for the CAPI Binding query.
  • Firehose: The Firehose is a WebSocket endpoint that streams all the event data from a PCF deployment. The data stream includes HTTP events, app logs, container metrics from apps, and metrics from PCF platform components. The Firehose cf CLI plugin allows you to view the output of the Firehose. For more information about the Firehose plugin, see Installing the Loggregator Firehose Plugin for cf CLI.
  • Log Cache: The Log Cache allows you to view logs and metrics from the Firehose over a specified period of time. The Log Cache includes API endpoints and a CLI plugin to query and filter logs and metrics. To download the Log Cache CLI plugin, see Cloud Foundry Plugins. The Log Cache API endpoints are available by default. For more information about using the Log Cache API, see Log Cache on GitHub.
  • Nozzles: Nozzles are programs that consume data from the Loggregator Firehose. Nozzles can be configured to select, buffer, and transform data, and to forward it to other apps and services. For more information about creating and deploying nozzles, see Deploying a Nozzle to the Loggregator Firehose.

This section describes the PCF components that forward BOSH-reported VM metrics to Loggregator. BOSH-reported VM metrics measure the health of BOSH-deployed VMs on which apps and PCF components are deployed. Loggregator streams BOSH-reported VM metrics through the Firehose.

The following are the PCF components that send BOSH-reported VM metrics to Loggregator:

  • BOSH Agent: BOSH Agents are located on component VMs and Diego cell VMs. They collect metrics, such as cell capacity remaining, from the VM and forward them to the BOSH Health Monitor.

  • BOSH Health Monitor: The BOSH Health Monitor receives metrics from the BOSH Agents. It then forwards the metrics to a third-party service or to the BOSH System Metrics Forwarder.

  • BOSH System Metrics Plugin: This plugin reads health events, such as VM heartbeats and alerts from the BOSH Health Monitor, and streams them to the BOSH System Metrics Server.

  • BOSH System Metrics Server: The BOSH System Metrics Server streams metrics and heartbeat events to the BOSH System Metrics Forwarder over gRPC.

  • BOSH System Metrics Forwarder: The BOSH System Metrics Forwarder is located on the Loggregator Traffic Controller. It forwards heartbeat events from the BOSH System Metrics Server as envelopes to Loggregator through a Loggregator Agent.