Indicator Protocol Dashboard (Beta)

Warning: This feature is in beta.

Overview

The Indicator Protocol dashboard allows you to define and monitor custom scaling indicator metrics for the services in your deployment. The dashboard displays charts of indicator metrics, which are defined in an indicator document. The dashboard is populated by data from the Log Cache component of Loggregator using a PromQL query.

For more information about PromQL, see Querying Prometheus in the Prometheus documentation.

For more information about Log Cache, see Loggregator Architecture.

Note: The charts on the Indicator Protocol dashboard are populated by Log Cache, which is a component of Loggregator. Charts on the Indicator Protocol dashboard might display the following error: "Error fetching graph data.". This error is caused because Log Cache times out while it attempts to process data. No corrective action is required. The error resolves itself if possible.

Architecture

The Indicator Protocol dashboard is enabled by a Monitoring Indicator Protocol Registration Agent that runs on each BOSH VM. The Monitoring Indicator Protocol Registration Agent watches and inspects the /var/vcap/jobs for a document called indicators.yml in a config directory.

If the agent locates this indicator document, it parses the document and sends the results to the Indicator Registry. This process of registering documents is repeated every 30 seconds.

In the following architecture diagram, the indicator-registration-agent runs on the RabbitMQ Server VM. The agent finds a document at /var/vcap/jobs/rabbit/config/indicators.yml and sends the data to an indicator registry.

This indicator registry defines indicators and runs on the Healthwatch Forwarder VM. Documents are temporarily stored in memory in the registry.

The indicator registry exposes a /indicator-documents endpoint, which returns a JSON representation of the indicator documents. Pivotal Healthwatch consumes the results of this endpoint and creates a dashboard for each document. If the indicator registry does not receive a re-registered document within 2 hours, the document is evicted and that service’s dashboard does not appear.

Indicator protocol architecture

Enable and Configure the Indicator Protocol Dashboard

This section provides instructions for enabling and configuring the Indicator Protocol dashboard.

To enable the Indicator Protocol dashboard:

  1. Enable the Indicator Protocol runtime config add-on for BOSH Director. For more information, see Enable the Indicator Protocol Add-on below.

  2. Register an indicator document. For more information, see Register an Indicator Document below.

Enable the Indicator Protocol Add-on

To enable the Indicator Protocol dashboard, you must enable a global BOSH Director runtime config add-on for Indicator Protocol. The BOSH Director applies runtime config changes to each deployment during the next BOSH deploy.

Note: After you update the runtime config, all deployments are outdated.

For more information about runtime config, see Director Runtime Configin the BOSH documentation.

To enable the Indicator Protocol runtime config add-on:

  1. Navigae to the Ops Manager Installation Dashboard.

  2. Click the Pivotal Healthwatch tile.

  3. Select Healthwatch Component Config.

  4. Under BETA: Enable the Indicator Protocol registration agent add-on?, select Yes.

  5. Click Save.

  6. Return to the Ops Manager Installation Dashboard.

  7. Click Review Pending Changes.

  8. Enable only Pivotal Healthwatch. If any other products are enabled, disable them.

    Note: BOSH Director cannot be disabled.

  9. Click Apply Changes.

Register an Indicator Document

The Indicator Protocol dashboard displays charts of indicator metrics, which are defined in an indicator document.

To register an indicator document for the Indicator Protocol dashboard, whether Pre-Provisioned or On-Demand, you must co-locate the Monitoring Indicator Protocol Registration Agent on each existing service’s VM.

For more information about the Monitoring Indicator Protocol Registration Agent, see Architecture above.

To co-locate the Monitoring Indicator Protocol Registration Agent on each of your existing services’ VM, re-deploy each service through BOSH.

For example, to enable VMware Tanzu RabbitMQ dashboards for all On-Demand service instances, use the upgrade-all-service-instances errand.

For information about the errand, including implications, see Upgrade All Service Instances in Troubleshooting and FAQs for On-Demand VMware Tanzu RabbitMQ [VMs] in the VMware Tanzu RabbitMQ documentation.

Access the Indicator Protocol Dashboard

To verify that the Indicator Protocol dashboards appear in Pivotal Healthwatch, view the Pivotal Healthwatch UI at https://healthwatch.SYSTEM-DOMAIN/indicator-dashboards, where SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile.