Configuring Pivotal Healthwatch Alerts

This topic describes how to use the Pivotal Healthwatch API to retrieve and configure alert configurations. It also provides information about configuring Pivotal Event Alerts to receive push notifications when a Pivotal Healthwatch alert occurs.

Note: Pivotal Healthwatch stores all data points for 25 hours and then prunes them. Any active alerts that are pruned reissue an alert every 24 hours if the related metrics are not yet recovered to a normal state.

Currently, two main types of alert configurations are supported: Out-of-the-Box and Deployment-specific. Through this API, consumers can do the following:

• View current alert configurations
• Enable/disable current alert configurations
• Update Out-of-the-Box threshold values
• Create or delete deployment-specific configurations based on existing Out-of-the-Box configurations

Prerequisites and Assumptions

The steps in this document assume that you can generate bearer tokens for a UAA client with the healthwatch.read (GET only) and healthwatch.admin (both GET and POST) scopes.

After creating a user that has healthwatch.read or healthwatch.admin scopes, authenticate against UAA by running:

uaac token client get HEALTHWATCH-ADMIN-CLIENT -s HEALTHWATCH-ADMIN-CLIENT-SECRET

Where:

• HEALTHWATCH-ADMIN-CLIENT is the UAA client with the healthwatch.read or healthwatch.admin scopes.

• HEALTHWATCH-ADMIN-CLIENT-SECRET is the UAA client secret.

• SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

At this point you are properly authenticated and ready to start using the Healthwatch Alerts API.

Healthwatch API Status

To test the availability of the Healthwatch API, send a GET request to /info endpoint by running:

curl https://healthwatch-api.SYSTEM-DOMAIN/info

Where SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

The expected response is a 200/OK with the message "HAPI is happy".

View All Alert Configurations

GET /v1/alert-configurations

To view a list of alert configurations, send a GET request to the /alert-configurations endpoint by running:

uaac curl https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations

Where SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

This returns a JSON array of alert configurations:

[
    {
        "query": "origin == 'some_origin' and name == 'Some.Metric.Name'",
        "enabled": true,
        "threshold": {
            "critical": 95,
            "warning": 85,
            "type": "UPPER"
        }
    },
    {
        "query": "origin == 'some_origin' and name == 'Another.Metric.Name'",
        "enabled": false,
        "threshold": {
            "critical": 0,
            "warning": 0,
            "type": "LOWER"
        }
    },
    {
        "query": "origin == 'another_origin' and name == 'Some.Metric.Name'",
        "enabled": true,
        "threshold": {
            "critical": 1,
            "type": "EQUALITY"
        }
    }
]

The query and threshold properties are covered in detail below.

View Specific Alert Configurations

GET /v1/alert-configurations?q=…

To narrow the results of a GET request, add a query to the URL in a parameter named q:

uaac curl "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations?q=origin == 'some_origin' and name == 'Some.Metric.Name'"

Where SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

This returns a JSON array of alert configurations, filtered against the provided query:

[
    {
        "query": "origin == 'some_origin' and name == 'Some.Metric.Name'",
        "enabled": true,
        "threshold": {
            "critical": 95,
            "warning": 85,
            "type": "UPPER"
        }
    }
]

Update Alert Configurations

POST /v1/alert-configurations

To update an existing alert configuration, make a POST request to the alert-configurations endpoint with the updated data:

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'some_origin' and name == 'Some.Metric.Name'\",\"threshold\":{\"critical\":90,\"warning\":80},\"enabled\":true}"

Where SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

See the following example output:

{
    "query": "origin == 'some_origin' and name == 'Some.Metric.Name'",
    "enabled": true,
    "threshold": {
        "critical": 95,
        "warning": 85,
        "type": "UPPER"
    }
}

Warning: These alert configurations cannot be deleted. To revert your changes, update the alert back to its default values. For more information, see Supported Alerts below.

Create Alert Configurations for Isolation Segments

POST /v1/alert-configurations

Specific thresholds can be set for an isolation segment by extending existing alert configurations with a deployment specifier. For example, to create an isolation segment alert configuration for the above alert, run:

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'some_origin' and name == 'Some.Metric.Name' and deployment == 'ISOLATION-SEGMENT-NAME'\",\"threshold\":{\"critical\":55,\"warning\":45}}"

Where:

• SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

• ISOLATION-SEGMENT-NAME is the name of the isolation segment for which you want to create an alert configuration.

A new alert configuration will be enabled by default. You can optionally include \"enabled\": false in the request body to override this behavior.

The threshold type will be inferred from the deployment-agnostic alert configuration and cannot be changed.

The created alert configuration is echoed back in the following response:

{
    "query": "origin == 'some_origin' and name == 'Some.Metric.Name' and deployment == 'ISOLATION-SEGMENT-NAME'",
    "enabled": true,
    "threshold": {
        "critical": 55,
        "warning": 45,
        "type": "UPPER"
    }
}

Note: You can delete isolation segment alert configurations only if you created them with the above method.

Delete Isolation Segment Alert Configurations

To delete a user-created alert configuration for an isolation segment, add a query to the URL in a parameter named q:

DELETE /v1/alert-configurations?q=...

See the following example:

uaac curl -X DELETE "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations?q=origin == 'some_origin' and name == 'Some.Metric.Name' and deployment == 'ISOLATION-SEGMENT-NAME'"

Where:

• SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

• ISOLATION-SEGMENT-NAME is the name of the isolation segment for which you want to create an alert configuration.

This returns the number of deleted alert configurations. See the following example output:

1

Note: You can delete isolation segment alert configurations only if you created them through the Healthwatch API.

Queries

The query field is used in two ways:

• GET requests: The query filters the alert configurations being queried.
• POST requests: The query specifies the alert configuration being updated.
• DELETE requests: The query filters the alert configurations being deleted.

The query associated with an alert configuration denotes the trigger conditions for the alert.

A well-formed query is a valid Spring Expression Language (SpEL) expression that uses only the equality, "==", and conjunction, "and", operators. For example:

"origin == 'some_origin' and name == 'Some.Metric.Name'"  # valid
"origin  > 'some_origin' and name == 'Some.Metric.Name'"  # invalid (uses '>')
"origin == 'some_origin'  or name == 'Some.Metric.Name'"  # invalid (uses 'or')

The following fields can be queried:

• (Required) origin   
• (Required) name       
• job
• deployment

Thresholds

The threshold field contains a threshold type, as well as critical and warning threshold values. The type can be "UPPER", "LOWER", "EQUALITY", or "INEQUALITY".

Alert configurations whose thresholds are of the UPPER type trigger their alerts when the actual metric value is above the warning value, and again when above the critical value.

Thresholds with the LOWER type work the same way, except the alerts trigger when the metric falls below the thresholds.

The EQUALITY alerts trigger when the metric value is not exactly equal to the critical threshold. These alerts do not have warning thresholds.

The INEQUALITY alerts trigger when the metric value is exactly equal to the critical threshold. These alerts do not have warning thresholds.

Supported Alerts

By default, Pivotal Healthwatch includes the following configurable alerts: This section describes the default Warning and Critical alerts for Pivotal Healthwatch. Each alert includes recommended thresholds for metrics monitored by Healthwatch.

Performance Alerts

For more information about the metrics Pivotal Healthwatch emits, see Pivotal Healthwatch Metrics.

¹ Healthwatch alerts by Diego Cell for this metric. Healthwatch notifies at a critical level when any Diego Cell has been unhealthy for 5 minutes.

Scaling Alerts

² This handles two jobs to accommodate different job names between PAS and SRT*

Service Level Indicators

Environment Specific Alerts

The alerts in the table below are highly variable depending on your environment. Pivotal recommends customizing their critical and warning thresholds based on your environment. These alerts are disabled by default (the enabled property on the alert configuration is false) and the default threshold values are 0.

You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health. Once you have monitored your system and established baseline values, you can update the threshold values and enable these alerts.

For more information about updating alert configurations in Pivotal Healthwatch, see Update Alert Configurations.

Warning: Since these alerts have default threshold values of 0, ensure the values are updated before they are enabled.

Errors

This section lists common error messages and their causes.

Error Message: “Unsupported condition in query expression”

Possible Cause: Key in query string is something other than name, origin, job, or deployment

Error Message: “Unsupported query expression”

Possible Causes:

• Query string does not include AT LEAST name and origin
• Found operator other than and or ==
• Expression in query string not in format of property == 'value' and ...

Error Message: “Invalid query expression”

Possible Cause: Invalid query string format

Error Message: “Invalid threshold type for metric ‘some_origin.some_name’”)

Possible Cause: Given threshold type does not match expected (see table above)

Error Message: “Must provide a warning threshold for an Upper/Lower Threshold”

Possible Cause: Threshold missing required values for type

Error Message: “name = 'some_name’ and origin = 'some_origin’ is not a supported alert configuration” Error Message: “name = 'some_name’, origin = 'some_origin’, and job = 'some_job’ is not a supported alert configuration”

Possible Cause: An alert configuration does not exist for the targeted metric

Error Message: “A valid threshold is required when creating an alert configuration”

Possible Cause: An attempt to create a new alert configuration (ie, for an isolation segment) did not include a valid threshold/s.

Walkthrough Example

A best practice deployment of Cloud Foundry includes at least three availability zones (AZs). For these types of deployments, Pivotal recommends that you have enough capacity to suffer failure of an entire AZ.

By default, Healthwatch sends an alert if the Diego.TotalPercentageAvailableMemoryCapacity.5M metric falls below 35%, or one in three.

However, if your environment has been scaled up to five AZs, you may want to adjust the alert configuration accordingly to 20%, or one in five, by running:

uaac token client get HEALTHWATCH-ADMIN-CLIENT -s HEALTHWATCH-ADMIN-CLIENT-SECRET

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'healthwatch' and name == 'Diego.TotalPercentageAvailableMemoryCapacity.5M'\",\"threshold\":{\"critical\":0.2,\"warning\":0.3}}"

Where:

• HEALTHWATCH-ADMIN-CLIENT is the UAA client with the healthwatch.read or healthwatch.admin scopes.

• HEALTHWATCH-ADMIN-CLIENT-SECRET is the UAA client secret.

• SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

To simply disable the alert, run:

uaac token client get HEALTHWATCH-ADMIN-CLIENT -s HEALTHWATCH-ADMIN-CLIENT-SECRET

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'healthwatch' and name == 'Diego.TotalPercentageAvailableMemoryCapacity.5M'\",\"enabled\": false}"

Where:

• HEALTHWATCH-ADMIN-CLIENT is the UAA client with the healthwatch.read or healthwatch.admin scopes.

• HEALTHWATCH-ADMIN-CLIENT-SECRET is the UAA client secret.

• SYSTEM-DOMAIN is the system domain URL configured in the TAS for VMs tile. For example, sys.example.com.

The response body contains the updated alert configuration:

uaac curl "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations?q=origin == 'healthwatch' and name == 'Diego.TotalPercentageAvailableMemoryCapacity.5M'"

Configure Pivotal Healthwatch Alert Notifications

You can configure Pivotal Event Alerts to receive push notifications when a Pivotal Healthwatch alert occurs. For example, if you configured a Pivotal Healthwatch alert for memory on a VM, you can use Pivotal Event Alerts to receive a message on Slack if memory on the VM exceeds the threshold defined in the Pivotal Healthwatch alert.

For more information about configuring Pivotal Event Alerts for Pivotal Healthwatch, see Pivotal Event Alerts.