Indicator Protocol Dashboard (Beta)

Warning: This feature is in Beta. To provide feedback to the Indicator Protocol team, please email indicator-protocol@pivotal.io.

Overview

The Indicator Protocol dasboard allows you to define and monitor custom scaling indicator metrics for the services in your deployment. The dashboard displays charts of indicator metrics, which are defined in an indicator document. The dashboard is populated by data from the Log Cache component of Loggregator using a PromQL query.

For more information about PromQL, see Querying Prometheus in the Prometheus documentation.

For more information about Log Cache, see Loggregator Architecture.

Note: The charts on the Indicator Protocol dashboard are populated by Log Cache, which is a component of Loggregator. Charts on the Indicator Protocol dashboard may display the following error: "Error fetching graph data.". This error is caused because Log Cache times out while it attempts to process data. No corrective action is required. The error will resolve itself if possible.

Architecture

The Indicator Protocol dashboard is enabled by a Monitoring Indicator Protocol Registration Agent that runs on each BOSH VM. The Monitoring Indicator Protocol Registration Agent watches and inspects the /var/vcap/jobs for a document called indicators.yml in a config folder.

If the agent locates this indicator document, it parses the document and sends the results to the Indicator Registry. This process of registering documents is repeated every 30 seconds.

In the following architecture diagram, the indicator-registration-agent runs on the RabbitMQ Server VM. The agent finds a document at /var/vcap/jobs/rabbit/config/indicators.yml and sends the data to an indicator registry.

This indicator registry defines indicators and runs on the Healthwatch Forwarder VM. Documents are temporarily stored in memory in the registry.

The indicator registry exposes a /indicator-documents endpoint, which returns a JSON representation of the indicator documents. Pivotal Healthwatch consumes the results of this endpoint and creates a dashboard for each document. If the indicator registry does not receive a re-registered document within 2 hours, the document is evicted and that service’s dashboard will not appear.

Indicator protocol architecture

Enable and Configure the Indicator Protocol Dashboard

This section provides instructions for enabling and configuring the Indicator Protocol dashboard.

To enable the Indicator Protocol dashboard, do the following:

  1. Enable the Indicator Protocol runtime config add-on for BOSH Director. See Enable the Indicator Protocol Add-on.

  2. Register an indicator document. See Register an Indicator Document.

Enable the Indicator Protocol Add-on

To enable the Indicator Protocol dashboard, you must enable a global BOSH Director runtime config add-on for Indicator Protocol. The BOSH Director applies runtime config changes to each deployment during the next BOSH deploy.

Note: After you update the runtime config, all deployments are outdated.

For more information about runtime config, see Director Runtime Config.

To enable the Indicator Protocol runtime config add-on, do the following:

  1. On the Ops Manager Installation Dashboard, click the Pivotal Healthwatch tile.

  2. Navigate to the Healthwatch Component Config pane.

  3. Select Yes on the BETA: Enable the Indicator Protocol registration agent add-on? field.

  4. Click Save.

  5. On the Ops Manager Installation Dashboard, click Review Pending Changes.

  6. Enable only Pivotal Healthwatch. If any other products are enabled, disable them.

    Note:BOSH Director cannot be disabled.

  7. Click Apply Changes.

Register an Indicator Document

The Indicator Protocol dashboard displays charts of indicator metrics, which are defined in an indicator document.

To register an indicator document for the Indicator Protocol dashboard, whether Pre-Provisioned or On-Demand, you must co-locate the Monitoring Indicator Protocol Registration Agent on each existing services’ VM.

For more information about the Monitoring Indicator Protocol Registration Agent, see Architecture.

To co-locate the Monitoring Indicator Protocol Registration Agent on each of your existing services’ VM, re-deploy each service through BOSH.

For example, to enable RabbitMQ dashboards for all On-Demand service instances, use the upgrade-all-service-instances errand.

For information about the errand, including implications, see Upgrade all service instances errand.

Access the Indicator Protocol Dashboard

To verify that the Indicator Protocol dashboards appear in Pivotal Healthwatch, view the Pivotal Healthwatch UI at https://healthwatch.SYSTEM-DOMAIN/indicator-dashboards, where SYSTEM-DOMAIN is your system domain.