LATEST VERSION: 1.4 - CHANGELOG
PCF Metrics v1.4

PCF Metrics Product Architecture

This topic describes the product architecture of Pivotal Cloud Foundry (PCF) Metrics.

Overview

The diagram below displays the components of PCF Metrics, as well as the Cloud Foundry components that the PCF Metrics system interacts with.

PCF Metrics deploys several Cloud Foundry apps as part of the install process. These components are the bold rectangles in the diagram. The cylinders represent the data storage components of PCF Metrics.

Architecture

See the following sections to understand the processes that happen within the PCF Metrics system.

How Data Flows from the Firehose to the Datastores

This section describes how PCF Metrics fills its datastores. PCF Metrics uses three datastores:

  • The MySQL component stores metric and event data from the apps running on your PCF deployment.
    • Examples of events are start and stop.
    • Examples of metrics are container metrics such as CPU and network metrics such as Requests.
  • The Elasticsearch component stores logs from the apps running on your PCF deployment.
  • The Redis component is used to cache data ingested from the Firehose and data queried by the Metrics API.

Components

The diagram below highlights the components involved in the process of getting metric and log data into the Elasticsearch and MySQL datastore.

process one

Process

The following table describes how the components act during each stage.

Stage Description
1 The metrics-ingestor app receives app logs, container metrics, and network metrics from the Firehose and forwards them to Redis.

Redis does the following:
  • Acts as a buffer for events and metrics data
  • Acts as a cache
2 Each of the logqueues acts independently, writing information to the datastores:

Elasticsearch logqueue

The elasticsearch-logqueue app reads data from Redis and checks whether the Elasticsearch datastore is available. If available, the app writes logs to the Elasticsearch datastore.

MySQL logqueue

The mysql-logqueue app reads data from Redis and checks whether the MySQL datastore is available. If the datastore is available, the app does the following:

  • Inserts container and network metrics into MySQL with 1-minute granularity
  • Parses app log messages for an app event name and inserts the event into MySQL

How the PCF Metrics UI Retrieves Data from the Datastores

This section describes the flow of data through the system when you interact with the PCF Metrics UI.

Components

The diagram below highlights the components involved in this process.

Process 2

Process

The following table describes how the components act during each stage.

StageDescription
1 A user launches metrics.SYSTEM-DOMAIN in a browser and enters their UAA credentials.
2 After the UAA authorizes the user, the browser does the following:

  1. Retrieves through the Cloud Controller API a list of apps that the user can access
  2. Displays a page in which the user can select any app returned by the Cloud Controller API
3 A user selects an app from the dropdown menu, which does the following:

  1. Opens an AJAX connection to the metrics API to retrieve metrics and logs for the specified time frame
4 The metrics API receives the requests from the browser and does the following:

  1. Communicates with the UAA and Cloud Controller to confirm that the user can access data for the requested app
  2. Fetches the data from MySQL and Elasticsearch and then streams it to the browser
Create a pull request or raise an issue on the source for this page in GitHub