Using PCF Event Alerts

Page last updated:

This topic describes how to configure alert destinations and subscribe to alerts. It explains how to set alert destinations as targets, and how to subscribe those targets to alert topics. An example workflow is included below these procedures: see Example Workflow.

PCF users with admin permissions can run event alert commands and subscribe to alerts. Admins can configure event alerts using the CF CLI.

Prerequisite

You must install both the PCF Event Alerts tile and plugin as described in the following sections:

Create Targets for Alerts

Targets are destinations for alerts and are unique to each user. A target can be an email address, a Slack channel, or a webhook.

To create an alert target, do the following:

  1. Run the command:

    cf eva-create-target TARGET-NAME TARGET-TYPE VALUE
    

    Where:

    • TARGET-NAME is a unique name for this target
    • TARGET-TYPE is one of the following:
      • email
      • slack
      • webhook
    • VALUE is an appropriate value for the target type, for example, an email address

Examples for using this command to create each of these targets appear below.

Create Email Targets

To create an email target, do the following:

  1. Run the command:

    cf eva-create-target TARGET-NAME email EMAIL-ADDRESS
    

    For example:

    $ cf eva-create-target user-email email user@example.com

    This creates a target named user-email for the email address user@example.com. Email addresses that do not match the currently logged in user are verified by sending an email containing a link. The target receives alerts after the user verifies the email.

    Note: You can use any email address as an alert target; email alert recipients do not necessarily have to be other admins or cf users.

Create Slack Targets

To create a Slack target, do the following:

  1. Create an incoming Slack webhook using the instructions in the Slack documentation.

    Record the Slack webhook URL, which is used in the next step.

  2. Run the command:

    cf eva-create-target TARGET-NAME slack URL
    

    For example:

    $ cf eva-create-target my-slack-channel slack https://hooks.slack.com/services/T024LQKAS/B9XFKQG59/YPwmF0cEXAMPLEc7T2iEr0b
    

    This creates a target named my-slack-channel where alerts go to the Slack webhook URL in the command.

Event alerts to a Slack channel, for example a Critical Threshold Event alert and a Healthy Threshold Event alert, might appear as follows:

Sample Slack Alerts

Create Webhook Targets

Webhook targets provide many possibilities for using event alerts. For example, a simple webserver proxy can forward events to Pager Duty, New Relic, Datadog, and other products. You can also use webhook targets to log or store alerts for auditing purposes.

About Webhook Request Formats

Webhook requests are not a standardized format. This means that the request body from Event Alerts must be changed to a format compatible with the webhook receiver. For example, for PagerDuty, the JSON request payload from Event Alerts must be changed to the format required for PagerDuty webhooks. For information about PagerDuty webhook payloads, see the PagerDuty documentation.

The format change can be done through a web server proxy. The proxy receives the webhook request from Event Alerts, transforms the request payload to the new format, and forwards the request to the end receiver, for example, PagerDuty.

Procedure

To create a webhook target, do the following:

  1. Run the command:

    cf eva-create-target TARGET-NAME webhook URL
    

    Where URL is the URL of the end receiver or, if you are using a proxy, it is the URL of the proxy app. See About Webhook Request Formats above.

    For example:

    $ cf eva-create-target my-webhook webhook https://example.com/pcf-events/foundation-prod

    This creates a target named my-webhook. Event Alerts posts a JSON webhook payload to https://example.com/pcf-events/foundation-prod in the following format:

    {
      "publisher": "healthwatch",
      "topic": "system.cpu.user",
      "timestamp": "2018-03-22T10:41:31-06:00",
      "metadata": {
        "level": "CRITICAL",
        "foundation": "pcf.example.com",
        "job": "router"
      },
      "subject": "CRITICAL Threshold Event: VM Memory Used: router; pcf.example.com",
      "body": "<b>level: CRITICAL</b>"
    }
    

    In the above example:

    • healthwatch is the alert publisher.
    • system.cpu.user is the alert topic.
    • The metadata keys shown, level foundation and job, represent what the publisher sends to PCF Event Alerts and can be different for each alert topic.

Subscribe to Alert Topics

In order for a target to start receiving alerts, you must subscribe it to available alert topics. A topic is an event that is relevant to your system.

Note: For PCF Event Alerts v1.2.1, you must have Pivotal Healthwatch v1.3 or later installed if you want to set up and subscribe to Healthwatch topics.

To subscribe a target to one or more alert topics, do the following:

  1. To see a list of available topics, run the cf eva-topics command:

    $ cf eva-topics
    Getting topics as admin...
    OK
    Publisher      Topic                            Description
    healthwatch    system.mem.percent               VM Memory Used
    healthwatch    system.cpu.user                  VM CPU Utilization
    healthwatch    system.disk.persistent.percent   VM Persistent Disk Used
    healthwatch    system.disk.ephemeral.percent    VM Ephemeral Disk Used
    healthwatch    system.disk.system.percent       VM Disk Used
    healthwatch    system.healthy                   VM Health Check
    

  2. To subscribe a target to a topic, use either of the commands below.

    • To subscribe a target to one or more topics from a publisher:

      cf eva-subscribe TARGET-NAME PUBLISHER --topics TOPIC1,TOPIC2,...
      
    • To subscribe a target to all topics from a publisher:

      cf eva-subscribe TARGET-NAME PUBLISHER --all
      

      Note: If you subscribe to all topics from a publisher, you can subscribe to a single topic from that publisher by running cf eva-subscribe TAEGET-NAME PUBLISHER --topics TOPIC1.

    Find the PUBLISHER and the TOPIC name(s) from the topic list in the previous step. Enter topic names in the command separated by commas.

    For example, to subscribe the target user-email to two topics from the publisher healthwatch, run:

    $ cf eva-subscribe user-email healthwatch --topics system.mem.percent,system.cpu.user

    Or, to subscribe the target user-email to all topics from the publisher healthwatch, run:

    $ cf eva-subscribe user-email healthwatch --all

Unsubscribe from Alert Topics

To unsubscribe a target from alert topics, do one of the following:

  • Unsubscribe a target from one or more topics from a publisher by running the following command:

    cf eva-unsubscribe TARGET-NAME PUBLISHER --topics TOPIC1,TOPIC2,...
    


    For example, to unsubscribe the target user-email from the system.cpu.user topic from the publisher healthwatch, run the following command:

    $ cf eva-unsubscribe user-email healthwatch --topics system.cpu.user

    Note: If you subscribe to all topics from a publisher, you can unsubscribe from a single topic from that publisher using the above command. If you want to resubscribe to the topic, run cf eva-subscribe TAEGET-NAME PUBLISHER --topics TOPIC1.

  • Unsubscribe a target from all topics from a publisher by running the following command:

    cf eva-unsubscribe TARGET-NAME PUBLISHER --all
    


    For example, to unsubscribe the target user-email from all topics from the publisher healthwatch, run the following command:

    $ cf eva-unsubscribe user-email healthwatch --all

To find PUBLISHER and TOPIC names run cf eva-topics. The resulting list is shown in the procedure, Subscribe to Alert Topics, above. You must separate topic names with commas in the command.

Example Workflow: Send all Healthwatch Alerts to a Webhook

In this example workflow, you create a webhook target named webhook-ops that has the URL http://example.com/healthwatch-events/foundation-prod. You then subsribe this target to all alert topics from the publisher healthwatch, and you publish a test alert to one of the healthwatch topics.

  1. Create a webhook target named webhook-ops:

    $ cf eva-create-target webhook-ops webhook http://example.com/healthwatch-events/foundation-prod
    
  2. To verify that the target is created correctly, run cf eva-targets.

  3. Subscribe the webhook-ops target to all healthwatch topics:

    $ cf eva-subscribe webhook-ops healthwatch --all
    
  4. Publish a sample message to a healthwatch topic. This requires that you have the notifications.write scope.

    cf eva-sample-publish healthwatch system.healthy
    

    All targets subscribed to the system.healthy topic receive a sample alert.

  5. Verify a webhook message was posted to the webhook-ops URL. In this example, that is http://example.com/healthwatch-events/foundation-prod.

PCF Event Alert Plugin Commands

Use the following PCF Event Alert plugin commands to perform various tasks related to event alerts.

Command Action
eva-api Displays the endpoint and version of the Event Alerts API server.
eva-create-target Creates a target.
eva-delete-target Deletes a target.
eva-publishers Displays a list of publishers.
eva-sample-publish Publishes a sample event to all subscribers of the specified topic. To issue this command, you must have the notifications.write UAA scope. For information about these UAA privileges, see Scopes.
eva-smoke Runs a smoke test against event alerts.
eva-subscribe Subscribes to topics for a publisher.
eva-subscriptions Displays a list of current subscriptions.
eva-targets Displays a list of targets.
eva-topics Displays a list of topics.
eva-unsubscribe Unsubscribes from all topics from a specific publisher.
eva-update-target Updates a target name.

You can also use commands to get help directly in the terminal window while working with the PCF Event Alerts plugin.

  • To list all event alert commands:

    cf plugins | grep event-alerts
    
  • To get help with a specific command:

    cf help COMMAND