PCF Metrics Product Architecture

This topic describes the product architecture of Pivotal Cloud Foundry (PCF) Metrics.

Overview

The diagram below shows the components of PCF Metrics and the Cloud Foundry components that the PCF Metrics system interacts with.

The bold rectangles represent Cloud Foundry apps that PCF Metrics deploys. The cylinders represent the data storage components of PCF Metrics.

PCF Metrics 1.5 architecture

The following sections explain processes that run within the PCF Metrics system.

How Data Flows from the Firehose to the Datastores

This section describes how PCF Metrics fills its datastores. PCF Metrics uses three datastores:

  • The MySQL component stores metric and event data from the apps running on your PCF deployment.
    • Examples of events are start and stop.
    • Examples of metrics are container metrics such as CPU and network metrics such as Requests.
  • The PostgreSQL component stores logs from the apps running on your PCF deployment.
  • The Redis component caches data ingested from the Firehose and data queried by the Metrics API.

Components

The diagram below highlights the components that convey metric and log data into the PostgreSQL and MySQL datastore.

Ingesting firehose data into PCF Metrics 1.5 datastores

Process

The following table describes how the components act during each stage.

Stage Description
1 The metrics-ingestor app receives app logs, container metrics, and network metrics from the Firehose and forwards them to Redis.

Redis does the following:
  • Acts as a buffer for events and metrics data
  • Acts as a cache
2 Each of the logqueues acts independently, writing information to the datastores:

PostgreSQL logqueue

The postgres-logqueue app reads data from Redis and checks whether the PostgreSQL datastore is available. If available, the app writes logs to the PostgreSQL datastore.

The postgres-logqueue prunes data older than the Logs Retention Window at UTC±00:00 every day or when the postgres-logqueue is restarted.

The postgres-logqueue will also prune logs data when the database surpasses the Logs Max Retention Percentage and checks this percentage at the Logs Disk Size Pruning Interval.

MySQL logqueue

The mysql-logqueue app reads data from Redis and checks whether the MySQL datastore is available. If the datastore is available, the app does the following:

  • Inserts container and network metrics into MySQL with 1-minute granularity
  • Parses app log messages for an app event name and inserts the event into MySQL
The mysql-logqueue prunes data older than the Metrics Retention Window at UTC±00:00 every day or when the mysql-logqueue is restarted.

How the PCF Metrics UI Retrieves Data from the Datastores

This section describes the flow of data through the system when you interact with the PCF Metrics UI.

Components

The diagram below highlights the components involved in this process.

Displaying logs, events, and metrics data in the PCF Metrics 1.5 UI

Process

The following table describes how the components act during each stage.

StageDescription
1 A user launches metrics.SYSTEM-DOMAIN in a browser and enters their UAA credentials.
2 After the UAA authorizes the user, the browser does the following:

  1. Retrieves through the Cloud Controller API a list of apps that the user can access
  2. Displays a page in which the user can select any app returned by the Cloud Controller API
3 A user selects an app from the dropdown menu, which does the following:

  1. Opens an AJAX connection to the metrics API to retrieve metrics and logs for the specified time frame
4 The metrics API receives the requests from the browser and does the following:

  1. Communicates with the UAA and Cloud Controller to confirm that the user can access data for the requested app
  2. Fetches the data from MySQL and PostgreSQL and then streams it to the browser