Monitoring Pivotal Healthwatch

This topic explains how to monitor the health of Pivotal Healthwatch using the metrics and key performance indicators (KPIs) generated by the service.

For general information about monitoring Pivotal Platform, see Monitoring Pivotal Cloud Foundry.

About Metrics

Pivotal Healthwatch emits metrics in the following format:

origin:"healthwatch" eventType:ValueMetric timestamp:1509638101820496694 deployment:"healthwatch-app-dev-v1-3"
job:"healthwatch-forwarder" index:"097f4b1e-5ca8-4866-82d5-00883798dad4" ip:"10.0.16.29" 
valueMetric:<name:"metrics.published" value:38 unit:"count">

All Pivotal Healthwatch-emitted metrics have the healthwatch origin.

Service Level Indicators for Pivotal Healthwatch

Service Level Indicators monitor that key features of the Pivotal Healthwatch product are working as expected. These SLIs are the most important operational metrics emitted about Healthwatch itself, as they indicate the reliability of the assessments Healthwatch is making.

CLI Health Test Availability

Description

Use: Indicates that Pivotal Healthwatch is assessing the health of the Cloud Foundry Command Line Interface (cf CLI) commands. If these continuous validation tests fail to make up-to-date assessments, they are no longer a reliable warning mechanism.

Metrics
name: health.check.cliCommand.probe.available firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(health_check_cliCommand_probe_available{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: <= 0.6
Red critical: <= 0.4

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response
  1. Ensure the cf-health-check app is running in the healthwatch space of the system org.
  2. Check the app logs for any obvious errors.

Canary App Health Test Availability

Description

Use: Indicates that Pivotal Healthwatch is assessing the current state of health for the canary app. If this continuous validation test fails to make up-to-date assessments, it is no longer a reliable warning mechanism.

Metrics
name: health.check.CanaryApp.probe.available firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(health_check_CanaryApp_probe_available{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: <= 0.6
Red critical: <= 0.4

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response
  1. Ensure the canary-health-check app is running in the healthwatch space of the system org. Check the app logs for any obvious errors.
  2. Verify that Apps Manager is running and accessible through the URL configured in the CANARY_URL environment variable of the canary-health-check app.

BOSH Director Health Test Availability

Description

Use: Indicates that Pivotal Healthwatch is assessing the current state of health for the BOSH Director. If this continuous validation test fails to make up-to-date assessments, it is no longer a reliable warning mechanism.

Metrics
name: health.check.bosh.director.probe.available firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(health_check_bosh_director_probe_available{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: <= 0.6
Red critical: <= 0.4

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response
  1. Ensure the bosh-health-check app is running in the healthwatch space of the system org. Check the app logs for any obvious errors.
  2. SSH into the running bosh-health-check app and copy the BOSH manifest from /home/vcap/app/health_check_manifest.yml. Try to deploy it manually on the BOSH Director and check for errors.

Ops Manager Health Test Availability

Description

Use: Indicates that Pivotal Healthwatch is assessing the current state of health for Ops Manager. If this continuous validation test fails to make up-to-date assessments, it is no longer a reliable warning mechanism.

Metrics
name: health.check.OpsMan.probe.available firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(health_check_OpsMan_probe_available{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: <= 0.6
Red critical: <= 0.4

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response
  1. Ensure the opsmanager-health-check app is running in the healthwatch space of the system org. Check the app logs for any obvious errors.
  2. Verify that Ops Manager is running and accessible through the URL configured in the OPSMANAGER_URL environment variable of the opsmanager-health-check app.

Pivotal Healthwatch UI Availability

Description

Use: Indicates that the Healthwatch UI is running and available to product users. While an issue with the UI does not impact the assessments that Pivotal Healthwatch is making, loss of the UI can impact user ability to visually reference these assessments.

Metrics
name: healthwatch.ui.available firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(healthwatch_ui_available{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: <= 0.6
Red critical: <= 0.4

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response
  1. Ensure the healthwatch app is running in the healthwatch space of the system org.
  2. Check the app logs for any obvious errors.
  3. Verify that the /info endpoint is available on the healthwatch app route.

Key Performance Indicators for Pivotal Healthwatch

This section describes the KPIs that you can use to monitor the health of Pivotal Healthwatch.

Number of Pivotal Healthwatch Nozzle Disconnects from Firehose

Description

Use: An unusual increase in the number of disconnects from the Firehose typically indicates that you need to scale the nozzle up. The Firehose disconnects nozzles that are slow consumers to protect apps from backpressure. This metric can also spike during a Pivotal Platform deployment because the Traffic Controller VMs restart, logging a disconnect.

A prolonged period of losing metrics as a result of disconnects can endanger the assessments that Pivotal Healthwatch makes using platform metrics from the Firehose.

Metrics
name: ingestor.disconnects firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(ingestor_disconnects{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: >= 20
Red critical: >= 30

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response

If no known deployment occurred and the spike is sustained, increase the number of Pivotal Healthwatch Ingestor instances and monitor this metric to ensure that it returns to a normal state.

You can scale Ingestor instances in the Healthwatch Component Config tab of the Pivotal Healthwatch tile or using the cf scale healthwatch-ingestor command. While cf scale helps you to quickly scale the instances, you should also update the tile configuration so that the next deployment does not override the manual scaling.

Number of Ingestor Dropped Metrics

Description

Use: An unusual increase in the number of dropped messages by the Pivotal Healthwatch Ingestor likely indicates that you need to scale up this component and verify the health of Redis. A prolonged period of dropping messages can endanger the assessments that Pivotal Healthwatch makes using platform metrics from the Firehose.

Metrics
name: ingestor.dropped firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL avg_over_time(ingestor_dropped{source_id="healthwatch-forwarder",deployment="$deployment"}[5m])
Thresholds Yellow warning: >= 10
Red critical: >= 20

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response

Verify the health of the Redis VM and increase the number of Pivotal Healthwatch Ingestor instances. Monitor this metric to ensure that it returns to a normal state.

You can scale Ingestor instances using the cf scale healthwatch-ingestor command. While cf scale helps you to quickly scale the instances, you should also update the Ingestor Count in the tile configuration located in Healthwatch Component Config tab. Otherwise, the next Apply Changes will override the manual scaling.

Redis Queue Size

Description

Use: An unusual spike in the number of queued metrics can indicate that Pivotal Healthwatch Workers are unable to keep up with the volume of metrics from the Firehose. A large Redis queue will result in value metrics and counter events being delayed; if the queue becomes completely full, metrics will be lost altogether. This will also adversely affect Pivotal Healthwatch’s ability to calculate super value metrics.

Metrics
name: redis.valueMetricQueue.size firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

name: redis.counterEventQueue.size firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL redis_valueMetricQueue_size{source_id="healthwatch-forwarder",deployment="$deployment"} + redis_counterEventQueue_size{source_id="healthwatch-forwarder",deployment="$deployment"}
Thresholds Red critical: >= 10000
Measurement

Average over last 5 minutes

Recommended Response

If the spike is sustained, increase the number of Pivotal Healthwatch Worker instances and monitor this metric to ensure that it returns to a normal state.

You can scale Worker instances in the Healthwatch Component Config tab of the Pivotal Healthwatch tile or using the cf scale healthwatch-worker command. While cf scale helps you to quickly scale the instances, you should also update the tile configuration so that the next deployment does not override the manual scaling.

Number of Healthwatch Super Metrics Published to Firehose

Description

Use: If an operator has not made changes that impact the number or frequency of assessments, an unusual drop in the number of metrics published can indicate that Pivotal Healthwatch may be experiencing a computation or publication issue.

Metrics
name: metrics.published firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL sum(metrics_published{source_id="healthwatch-forwarder",deployment="$deployment"}) by (job)
Thresholds Yellow warning: <= 20
Red critical: <= 10

These are environment specific

Measurement

Average over last 5 minutes

Recommended Response
  1. Verify that the healthwatch-forwarder VM is running.
  2. Check all of the logs in /var/vcap/sys/log on the VM.
  3. Verify that the *-health-check apps are running and the logs in the healthwatch space of the system org are not receiving any obvious errors from them.

Other Metrics Available

This section describes other metrics that you can use to monitor Pivotal Healthwatch.

Number of Healthwatch Events Published to PCF Event Alerts

Description

Number of Pivotal Healthwatch Event Alerts triggered and published to [PCF Event Alerts](http://docs.pivotal.io/event-alerts/index.html.

Use: This metric is primarily interesting for informational purposes. As the number of alerting events could vary greatly, it is not recommended to alert on this metric itself.

name: events.published firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL events_published{source_id="healthwatch-forwarder",deployment="$deployment"}

BOSH Deployment Check Probe Count

Description

Number of Pivotal Healthwatch BOSH Deployment Occurrence probes completed in the measured time interval.

Use: When monitoring this metric, the primary indicator of concern is an unexpected negative variance from the normal pattern of checks per test type. If an operator has not made changes that impact the number of checks being made, such as scaling the test runner or changing the frequency of the test, an unexpected variance from normal likely indicates problems in the test runner functionality.

In the default installation, these tests run every 30 seconds across 2 runner apps.

name: health.bosh.deployment.probe.count firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL health_bosh_deployment_probe_count{source_id="healthwatch-forwarder",deployment="$deployment"}

CLI Command Health Probe Count

Description

Number of Pivotal Healthwatch CLI Command Health probe assessments completed in the measured time interval.

Use: For alerting purposes, Pivotal suggests using health.check.cliCommand.probe.available instead. This metric is most helpful for additional diagnostics or secondary alerting.

When monitoring this metric, the primary indicator of concern is an unexpected negative variance from the normal pattern of checks per test type. If an operator has not made changes that impact the number of checks being made, such as scaling the test runner or changing the frequency of the test, an unexpected variance from normal likely indicates problems in the test runner functionality.

In the default installation, these tests run every 5 minutes across 2 runner apps.

name: health.check.cliCommand.probe.count firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL health_check_cliCommand_probe_count{source_id="healthwatch-forwarder",deployment="$deployment"}

Ops Manager Health Probe Count

Description

Number of Pivotal Healthwatch Ops Manager Health probe assessments completed in the measured time interval.

Use: For alerting purposes, Pivotal suggests using health.check.OpsMan.probe.available instead. This metric is most helpful for additional diagnostics or secondary alerting.

When monitoring this metric, the primary indicator of concern is an unexpected negative variance from the normal pattern of checks per test type. If an operator has not made changes that impact the number of checks being made, such as scaling the test runner or changing the frequency of the test, an unexpected variance from normal likely indicates problems in the test runner functionality.

In the default installation, these tests run every 1 minute across 2 runner apps.

name: health.check.OpsMan.probe.count firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL health_check_OpsMan_probe_count{source_id="healthwatch-forwarder",deployment="$deployment"}

Canary App Health Probe Count

Description

Number of Pivotal Healthwatch Canary App Health probe assessments completed in the measured time interval.

Use: For alerting purposes, Pivotal suggests using health.check.CanaryApp.probe.available instead. This metric is most helpful for additional diagnostics or secondary alerting.

When monitoring this metric, the primary indicator of concern is an unexpected negative variance from the normal pattern of checks per test type. If an operator has not made changes that impact the number of checks being made, such as scaling the test runner or changing the frequency of the test, an unexpected variance from normal likely indicates problems in the test runner functionality.

In the default installation, these tests run every 1 minutes across 2 runner apps.

name: health.check.CanaryApp.probe.count firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL health_check_CanaryApp_probe_count{source_id="healthwatch-forwarder",deployment="$deployment"}

BOSH Director Health Probe Count

Description

Number of Pivotal Healthwatch BOSH Director Health probe assessments completed in the measured time interval.

Use: For alerting purposes, Pivotal suggests using health.check.bosh.director.probe.available instead. This metric is most helpful for additional diagnostics or secondary alerting.

When monitoring this metric, the primary indicator of concern is an unexpected negative variance from the normal pattern of checks per test type. If an operator has not made changes that impact the number of checks being made, such as scaling the test runner or changing the frequency of the test, an unexpected variance from normal likely indicates problems in the test runner functionality.

In the default installation, these tests run every 10 minutes using 1 runner app.

name: health.check.bosh.director.probe.count firehose origin: healthwatch log-cache source id: healthwatch-forwarder type: gauge frequency: 60s

PromQL health_check_bosh_director_probe_count{source_id="healthwatch-forwarder",deployment="$deployment"}