Configuring PCF Healthwatch Alerts

Warning: PCF Healthwatch v1.6 is no longer supported or available for download. PCF Healthwatch v1.6 has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to use the Pivotal Cloud Foundry (PCF) Healthwatch API to retrieve and configure alert configurations. It also provides information about configuring PCF Event Alerts to receive push notifications when a PCF Healthwatch alert occurs.

Note: PCF Healthwatch stores all data points for 25 hours and then prunes them. Any active alerts that are pruned reissue an alert every 24 hours if the related metrics are not yet recovered to a normal state.

Currently, two main types of alert configurations are supported: Out-of-the-Box and Deployment-specific. Through this API, consumers can do the following:

• View current alert configurations
• Update Out-of-the-Box threshold values
• Create or delete deployment-specific configurations based on existing Out-of-the-Box configurations

Prerequisites/Assumptions

The steps in this document assume that you can generate bearer tokens for a UAA client with the healthwatch.read (GET only) and healthwatch.admin (both GET and POST) scopes.

After creating a user that has healthwatch.read or healthwatch.admin scopes, follow these steps to authenticate against UAA:

uaac token client get <my_healthwatch_admin_client> -s <my_healthwatch_admin_secret>

At this point you are properly authenticated and ready to start using the Healthwatch Alerts API.

Healthwatch API Status

Test the availability of the Healthwatch API by hitting the /info endpoint with a GET request:

curl https://healthwatch-api.SYSTEM-DOMAIN/info

The expected response is a 200/OK with the message "HAPI is happy".

View All Alert Configurations

GET /v1/alert-configurations

To view a list of alert configurations, send a GET request to the /alert-configurations endpoint:

uaac curl https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations

This returns a JSON array of alert configurations:

[
    {
        "query": "origin == 'some_origin' and name == 'Some.Metric.Name'",
        "threshold": {
            "critical": 95,
            "warning": 85,
            "type": "UPPER"
        }
    },
    {
        "query": "origin == 'some_origin' and name == 'Another.Metric.Name'",
        "threshold": {
            "critical": 9,
            "warning": 28,
            "type": "LOWER"
        }
    },
    {
        "query": "origin == 'another_origin' and name == 'Some.Metric.Name'",
        "threshold": {
            "critical": 1,
            "type": "EQUALITY"
        }
    }
]

The query and threshold properties are covered in detail below.

View Specific Alert Configurations

GET /v1/alert-configurations?q=...

To narrow the results of a GET request, add a query to the URL in a parameter named q:

uaac curl "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations?q=origin == 'some_origin' and name == 'Some.Metric.Name'" 

This returns a JSON array of alert configurations, filtered against the provideded query:

[
    {
        "query": "origin == 'some_origin' and name == 'Some.Metric.Name'",
        "threshold": {
            "critical": 95,
            "warning": 85,
            "type": "UPPER"
        }
    }
]

Update Alert Configurations

POST /v1/alert-configurations

To update an existing alert configuration, make a POST request to the alert-configurations endpoint with the updated data:

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'some_origin' and name == 'Some.Metric.Name'\",\"threshold\":{\"critical\":90,\"warning\":80,\"type\":\"UPPER\"}}"

See the following example output:

{
    "query": "origin == 'some_origin' and name == 'Some.Metric.Name'",
    "threshold": {
        "critical": 95,
        "warning": 85,
        "type": "UPPER"
    }
}

Warning: These alert configurations cannot be deleted. In order to revert your changes, update the alert back to its default values.

Create Alert Configurations for Isolation Segments

POST /v1/alert-configurations

Specific thresholds can be set for an Isolation Segments by extending existing alert configurations with a deployment specifier. For example, to create an isolation segment alert configuration for the above alert, run the following:

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'some_origin' and name == 'Some.Metric.Name' and deployment == 'Some-Isolated-Deployment'\",\"threshold\":{\"critical\":55,\"warning\":45,\"type\":\"UPPER\"}}"

The created alert configuration is echoed back in the following response:

{
    "query": "origin == 'some_origin' and name == 'Some.Metric.Name' and deployment == 'Some-Isolated-Deployment'",
    "threshold": {
        "critical": 55,
        "warning": 45,
        "type": "UPPER"
    }
}

Note: You can delete Isolation Segment alert configurations only if you created them with the above method.

Delete Isolation Segment Alert Configurations

To delete a user-created alert configuration for an isolation segment, add a query to the URL in a parameter named q.

DELETE /v1/alert-configurations?q=...

See the following example:

uaac curl -X DELETE "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations?q=origin == 'some_origin' and name == 'Some.Metric.Name' and deployment == 'Some-Isolated-Deployment'"

This returns the number of deleted alert configurations. See the following example output.

1

Note: You can delete Isolation Segment alert configurations only if you created them through the Healthwatch API.

Disable Alerts on a Metric

PCF Healthwatch does not support disabling alerts on a specific metric. To ensure that PCF Healthwatch does not alert on a metric, update the threshold of the alert to a value that will never trigger an alert.

For more information about thresholds, see Thresholds.

For more information about updating alert configurations, see Update Alert Configurations.

Queries

The query field is used in two ways:

• GET requests: The query filters the alert configurations being queried.
• POST requests: The query specifies the alert configuration being updated.
• DELETE requests: The query filters the alert configurations being deleted.

The query associated with an alert configuration denotes the trigger conditions for the alert.

A well-formed query is a valid Spring Expression Language (SpEL) expression that uses only the equality, "==", and conjunction, "and", operators. For example:

"origin == 'some_origin' and name == 'Some.Metric.Name'"  # valid
"origin  > 'some_origin' and name == 'Some.Metric.Name'"  # invalid (uses '>')
"origin == 'some_origin'  or name == 'Some.Metric.Name'"  # invalid (uses 'or')

The following fields can be queried:

• origin   (required)
• name       (required)
• job
• deployment

Thresholds

The threshold field contains a threshold type, as well as critical and warning threshold values. The type can be "UPPER", "LOWER", "EQUALITY", or "INEQUALITY".

Alert configurations whose thresholds are of the UPPER type trigger their alerts when the actual metric value is above the warning value, and again when above the critical value.

Thresholds with the LOWER type work the same way, except the alerts trigger when the metric falls below the thresholds.

The EQUALITY alerts trigger when the metric value is not exactly equal to the critical threshold. These alerts do not have warning thresholds.

The INEQUALITY alerts trigger when the metric value is exactly equal to the critical threshold. These alerts do not have warning thresholds.

Supported Alerts

This section describes the default Warning and Critical alerts for PCF Healthwatch. Each alert includes recommended thresholds for metrics monitored by Healthwatch.

Pivotal recommends customizing the default thresholds for alerts indicated with a ¹ in the following tables based on your environment. You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health. For more information about updating alert configurations in PCF Healthwatch, see Update Alert Configurations.

By default, PCF Healthwatch includes the configurable alerts in the tables below. You can learn more about the metrics PCF Healthwatch emits here: PCF Healthwatch Metrics.

Performance Alerts

¹ Pivotal recommends customizing the default thresholds for these alerts based on your environment. You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health. For more information about updating alert configurations in PCF Healthwatch, see Update Alert Configurations.
² We are alerting by cell for this metric. We will notify at a critical level when any Diego Cell has been unhealthy for 5 minutes.

Scaling Alerts

¹ Pivotal recommends customizing the default thresholds for these alerts based on your environment. You can determine the best threshold for your environment by monitoring the metrics over time and noting the metric values that indicate acceptable and unacceptable system performance and health. For more information about updating alert configurations in PCF Healthwatch, see Update Alert Configurations.
³ This handles two jobs to accommodate different job names between PAS and SRT*

Service Level Indicators

Errors

This section lists common error messages and their causes.

Error Message: “Unsupported condition in query expression”

Possible Cause: Key in query string is something other than name, origin, job, or deployment

Error Message: “Unsupported query expression”

Possible Causes:

• Query string does not include AT LEAST name and origin
• Found operator other than and or ==
• Expression in query string not in format of property == 'value' and ...

Error Message: “Invalid query expression”

Possible Cause: Invalid query string format

Error Message: “Invalid threshold type for metric ‘some_origin.some_name’”)

Possible Cause: Given threshold type does not match expected (see table above)

Error Message: “Must provide a warning threshold for an Upper/Lower Threshold”

Possible Cause: Threshold missing required values for type

Error Message: “name = 'some_name’ and origin = 'some_origin’ is not a supported alert configuration” Error Message: “name = 'some_name’, origin = 'some_origin’, and job = 'some_job’ is not a supported alert configuration”

Possible Cause: An alert configuration does not exist for the targeted metric

Walkthrough Example

A best practice deployment of Cloud Foundry includes at least three availability zones (AZs). For these types of deployments, Pivotal recommends that you have enough capacity to suffer failure of an entire AZ.

By default, Healthwatch sends an alert if the Diego.TotalPercentageAvailableMemoryCapacity.5M metric falls below 35%, or one in three.

However, if your environment has been scaled up to five AZs you may wish to adjust the alert configuration accordingly to 20%, or more in five.

uaac token client get <my_healthwatch_admin_client> -s <my_healthwatch_admin_secret>
export token=$(uaac context | grep access_token | awk '{print $2}')

uaac curl -X POST "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations"  \
      -H "Content-Type: application/json" \
      --data "{\"query\":\"origin == 'healthwatch' and name == 'Diego.TotalPercentageAvailableMemoryCapacity.5M'\",\"threshold\":{\"critical\":0.2,\"warning\":0.3,\"type\":\"LOWER\"}}"

The response body contains the updated alert configuration. You can then confirm the change:

uaac curl "https://healthwatch-api.SYSTEM-DOMAIN/v1/alert-configurations?q=origin == 'healthwatch' and name == 'Diego.TotalPercentageAvailableMemoryCapacity.5M'"

Configure PCF Healthwatch Alert Notifications

You can configure PCF Event Alerts to receive push notifications when a PCF Healthwatch alert occurs. For example, if you configured a PCF Healthwatch alert for memory on a VM, you can use PCF Event Alerts to receive a message on Slack if memory on the VM exceeds the threshold defined in the PCF Healthwatch alert.

For more information about configuring PCF Event Alerts for PCF Healthwatch, see PCF Event Alerts.