Monitoring and Troubleshooting Pivotal Event Alerts

Page last updated:

This topic describes how to monitor and troubleshoot Pivotal Event Alerts.

Monitor Event Alerts

Event Alerts is deployed as an app in Pivotal Platform. By default, this app is called event-alerts and is deployed in the event-alerts space within the system org.

To monitor the performance of Event Alerts or to retrieve troubleshooting information:

  • Use the Cloud Foundry Command Line Interface (cf CLI) to obtain logs for the event-alerts app. For more information, see Application Logging in Cloud Foundry.
  • Use the Pivotal App Metrics service. For more information, see Pivotal App Metrics.
  • Use the Pivotal Metrics Registrar service. To use this service, you must install the plugin and register the event-alerts app as a JSON metrics source. For instructions, see Install the Plugin and Register a Structured Log Format in Using Metric Registrar.
  • Use the Pivotal Metrics Forwarder service. The event-alerts app detects if you have an available instance of Pivotal Metrics Forwarder and transmits custom app metrics through Loggregator to the service. For more information, see Pivotal Metrics Forwarder.

    Note: When Pivotal Metrics Forwarder is not installed, Event Alerts produces an error. This indicates that no custom metrics are being published to the Firehose. It does not indicate a problem with the system.

    Note: If you use Event Alerts v1.2.9 and earlier on Pivotal Application Service v2.5 and later, App Metrics and Metric Forwarder fail to emit metrics.

Scale Event Alerts

You might need to scale Event Alerts to improve performance.

To scale Event Alerts:

  1. Use the cf CLI to log in to your Pivotal Platform deployment as an admin. For example:
    $ cf login
    API endpoint: https://api.sys.cf-example.com
    Email> admin
    Password>
    
  2. When prompted, select the system org and the event-alerts space.
  3. Scale up the number of instances of the event-alerts app. Two instances should adequately handle most alerting workloads from Pivotal Healthwatch. For example:
    $ cf scale event-alerts -i 2
    If you are running very large Pivotal Platform deployments with frequent threshold events, you might need to increase the instance count above two.

Reduce Notifications

If Event Alerts is receiving frequent event notifications, you can do one or both of the following:

  • Unsubscribe from alerts that have minimal action.
  • Increase threshold values to supply better indicators for action.

For more information about unsubscribing from alerts and increasing threshold values, see Using Event Alerts.

Troubleshooting Errors

This section provides information about how to troubleshoot specific errors.

Deploy Event Alerts Errand Fails

This problem happens when you do not have SUPER privileges and binary logging is enabled. If you do not resolve this issue, Event Alerts is not deployed.

Symptom

When you Apply Changes, you see in the Pivotal Ops Manager installation pane that the Deploy Event Alerts errand fails with the following error:

You do not have the SUPER privilege and
binary logging is enabled (you *might* want to use the less safe
log_bin_trust_function_creators variable)

Explanation

For information about the above error message, see Stored Program Binary Logging in the open source MySQL documentation.

Solution

To solve this issue:

  1. Drop the Event Alerts database and re-create it. For information about dropping a database, see DROP DATABASE Syntax in the open source MySQL documentation.
  2. Do one of the following:
    • In the database, disable binary logging. For information about disabling binary logging, see The Binary Log in the open source MySQL documentation.
    • In the database, set log_bin_trust_function_creators to 1. For information about log_bin_trust_function_creators, see log_bin_trust_function_creators in the open source MySQL documentation.
  3. In Ops Manager, Apply Changes with the Deploy Event Alerts errand checked.

Event Alerts Fails to Release IP Address Bindings on MySQL Upgrade

Symptom

When upgrading from MySQL for Pivotal Cloud Foundry v2.4 to v2.5, all service bindings must be changed from IP address based bindings to hostname based bindings. Event Alerts sometimes fails to release IP address bindings after following the procedure in Deprecated Service Bindings Found (Upgrade Error).

Explanation

The cause of this issue is unknown.

Solution

Re-create the service key by running:

Warning: You must use the username migrations_user in the command below. Without this, a random name is generated that causes Event Alerts to fail.

cf create-service-key pcf-event-alerts-db migrations_service_key -c '{ "username": "migrations_user" }'