Monitoring and Troubleshooting PCF Event Alerts

Page last updated:

This topic describes how to monitor and troubleshoot Pivotal Cloud Foundry (PCF) Event Alerts.

Monitor PCF Event Alerts

PCF Event Alerts is deployed as an app in PCF. By default, this app is called event-alerts and is deployed in the event-alerts space within the system org.

To monitor the performance of PCF Event Alerts or to retrieve troubleshooting information, you can do one or more of the following:

  • Use the Cloud Foundry Command Line Interface (cf CLI) to obtain logs for the event-alerts app. For more information, see Application Logging in Cloud Foundry.
  • Use the PCF Metrics service. For more information, see PCF Metrics.
  • Use the Metrics Forwarder for PCF service. The event-alerts app detects if you have an available instance of Metrics Forwarder for PCF and transmits custom app metrics through Loggregator to the service. For more information, see Metrics Forwarder for PCF.

    Note: When Metrics Forwarder is not installed, Event Alerts produces an error. This indicates that no custom metrics are being published to the Firehose. It does not indicate a problem with the system.

Scale PCF Event Alerts

You might need to scale PCF Event Alerts to improve performance.

To scale PCF Event Alerts, perform the following steps:

  1. Use the cf CLI to log in to your PCF deployment as an admin. For example:
    $ cf login
    API endpoint: https://api.sys.cf-example.com
    Email> admin
    Password>
    
  2. When prompted, select the system org and the event-alerts space.
  3. Scale up the number of instances of the event-alerts app. Two instances should adequately handle most alerting workloads from PCF Healthwatch. For example:
    $ cf scale event-alerts -i 2
    If you are running very large PCF deployments with frequent threshold events, you might need to increase the instance count above two.

Reduce Notifications

If PCF Event Alerts is receiving frequent event notifications, you can do one or both of the following:

  • Unsubscribe from alerts that have minimal action.
  • Increase threshold values to supply better indicators for action.

For more information about unsubscribing from alerts and increasing threshold values, see Using PCF Event Alerts.

Troubleshooting Errors

This section provides information on how to troubleshooting specific errors or error messages.

Deploy PCF Event Alerts Errand Fails

This problem happens when you do not have SUPER privileges and binary logging is enabled. If you do not resolve this issue, PCF Event Alerts is not deployed.

Symptom

When you Apply Changes, you see in the Ops Manager installation pane that the Deploy PCF Event Alerts errand fails with the following error:

You do not have the SUPER privilege and
binary logging is enabled (you *might* want to use the less safe
log_bin_trust_function_creators variable)

Explanation

For information about the above error message, see Stored Program Binary Logging in the open source MySQL documentation.

Solution

To solve this issue, do the following:

  1. Drop the PCF Event Alerts database and recreate it. For information about dropping a database, see DROP DATABASE Syntax in the open source MySQL documentation.
  2. Do one of the following:
    • In the database, disable binary logging. For information about disabling binary logging, see The Binary Log in the open source MySQL documentation.
    • In the database, set log_bin_trust_function_creators to 1. For information about log_bin_trust_function_creators, see log_bin_trust_function_creators in the open source MySQL documentation.
  3. In Ops Manager, Apply Changes with the Deploy PCF Event Alerts errand checked.

Event Alerts Fails to Release IP Address Bindings on MySQL Upgrade

Symptom

When upgrading from MySQL for PCF v2.4 to v2.5, all service bindings must be changed from IP address based bindings to hostname based bindings. Event Alerts sometimes fails to release IP address bindings after following the procedure in Deprecated Service Bindings Found (Upgrade Error).

Explanation

The cause of this issue is unknown.

Solution

Recreate the service key by running:

Warning: You must use the username migrations_user in the command below. Without this, a random name is generated that causes Event Alerts to fail.

cf create-service-key pcf-event-alerts-db migrations_service_key -c '{ "username": "migrations_user" }'