Monitoring and Troubleshooting Event Alerts

Page last updated:

This topic describes how to monitor and troubleshoot Event Alerts.

Monitor Event Alerts

Event Alerts is deployed as an app in VMware Tanzu Application Service for VMs (TAS for VMs). By default, this app is called event-alerts and is deployed in the event-alerts space within the system org.

To monitor the performance of Event Alerts or to retrieve troubleshooting information:

  • Use the Cloud Foundry Command Line Interface (cf CLI) to obtain logs for the event-alerts app. For more information, see Application Logging in Cloud Foundry.
  • Use the App Metrics service. For more information, see App Metrics.
  • Use the Metrics Registrar service. To use this service, you must install the plugin and register the event-alerts app as a JSON metrics source. For instructions, see Install the Plugin and Register a Structured Log Format in Using Metric Registrar.
  • Use the Metrics Forwarder service. The event-alerts app detects if you have an available instance of Metrics Forwarder and transmits custom app metrics through Loggregator to the service. For more information, see Metrics Forwarder.

    Note: When Metrics Forwarder is not installed, Event Alerts produces an error. This indicates that no custom metrics are being published to the Firehose. It does not indicate a problem with the system.

    Note: If you use Event Alerts v1.2.9 and earlier on VMware Tanzu Application Service for VMs v2.5 and later, App Metrics and Metric Forwarder fail to emit metrics.

Scale Event Alerts

You might need to scale Event Alerts to improve performance.

To scale Event Alerts:

  1. Use the cf CLI to log in to your TAS for VMs deployment as an admin. For example:
    $ cf login
    API endpoint: https://api.sys.cf-example.com
    Email> admin
    Password>
    
  2. When prompted, select the system org and the event-alerts space.
  3. Scale up the number of instances of the event-alerts app. Two instances should adequately handle most alerting workloads from Healthwatch. For example:
    $ cf scale event-alerts -i 2
    If you are running very large TAS for VMs deployments with frequent threshold events, you might need to increase the instance count above two.

Reduce Notifications

If Event Alerts is receiving frequent event notifications, you can do one or both of the following:

  • Unsubscribe from alerts that have minimal action.
  • Increase threshold values to supply better indicators for action.

For more information about unsubscribing from alerts and increasing threshold values, see Using Event Alerts.

Troubleshooting Errors

This section provides information about how to troubleshoot specific errors.

Note: This issue has been resolved in Event Alerts v1.2.12.

This issue occurs when you have VMware Tanzu™ SQL with MySQL for VMs (Tanzu SQL for VMs) v2.10 installed and TLS Options is set to any value other than Not Configured in the Security pane of the Tanzu SQL for VMs tile. If you do not resolve this issue, Event Alerts is not deployed.

Symptom

After you deploy Tanzu SQL for VMs v2.10, you see in the Ops Manager Installation Dashboard that the Deploy Event Alerts errand fails with an error similar to the following example:

ERROR:
Unable to obtain connection from database (jdbc://example.com) for user 'migrations_user': Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State  : 08S01
Error Code : 0
Message    : Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.

Explanation

In Tanzu SQL for VMs v2.10, MySQL requires TLS v1.2 for all TLS connections even when the Accept only TLS 1.2 connections checkbox in the Security pane of the Tanzu SQL for VMs tile is disabled. When the Deploy Event Alerts Errand runs, the errand does not connect to the database using TLS v1.2 when attempting to migrate the database. This causes the errand to fail.

Solution

To solve this issue:

  1. Export the GUID of the event-alerts space by running:

    export SPACE_GUID=$(cf space event-alerts --guid)
    
  2. To find the GUID of your Tanzu SQL for VMs service instance, run:

    cf curl "/v3/service_instances?names=pcf-event-alerts-db&space_guids=${SPACE_GUID}"
    
  3. Record the GUID from the resources[0].guid property in the JSON output of the command in the step above.

  4. To retrieve a list of the VMs in your Tanzu SQL for VMs service instance, run:

    bosh -d service-instance_GUID vms
    

    Where GUID is the GUID of your Tanzu SQL for VMs service instance that you recorded in the step above.

  5. For each mysql VM:

    1. Run:

      bosh -d service-instance_GUID ssh MYSQL-INSTANCE-NAME
      

      Where:

      • GUID is the GUID of your Tanzu SQL for VMs service instance.
      • MYSQL-INSTANCE-NAME is one of the MySQL instance names from the list of VMs that you retrieved in the step above.
    2. Switch to the root user profile in your terminal by running:

      sudo -i
      
    3. Open the /var/vcap/jobs/mysql/config/my.cnf file in a text editor.

    4. Delete the line containing tls_version = TLSv1.2.

    5. Restart the VM by running:

      monit restart mysql
      
  6. After you have restarted each MySQL VM, rerun the Deploy Event Alerts Errand.

Deploy Event Alerts Errand Fails Due to Binary Logging

This problem happens when you do not have SUPER privileges and binary logging is enabled. If you do not resolve this issue, Event Alerts is not deployed.

Symptom

When you Apply Changes, you see in the installation pane that the Deploy Event Alerts errand fails with the following error:

You do not have the SUPER privilege and
binary logging is enabled (you *might* want to use the less safe
log_bin_trust_function_creators variable)

Explanation

For information about the above error message, see Stored Program Binary Logging in the open source MySQL documentation.

Solution

To solve this issue:

  1. Drop the Event Alerts database and re-create it. For information about dropping a database, see DROP DATABASE Syntax in the open source MySQL documentation.
  2. Do one of the following:
    • In the database, disable binary logging. For information about disabling binary logging, see The Binary Log in the open source MySQL documentation.
    • In the database, set log_bin_trust_function_creators to 1. For information about log_bin_trust_function_creators, see log_bin_trust_function_creators in the open source MySQL documentation.
  3. In Ops Manager, Apply Changes with the Deploy Event Alerts errand checked.

Event Alerts Fails to Release IP Address Bindings on MySQL Upgrade

Symptom

When upgrading from MySQL for Pivotal Cloud Foundry v2.4 to v2.5, all service bindings must be changed from IP address based bindings to hostname based bindings. Event Alerts sometimes fails to release IP address bindings after following the procedure in Deprecated Service Bindings Found (Upgrade Error).

Explanation

The cause of this issue is unknown.

Solution

Re-create the service key by running:

Warning: You must use the username migrations_user in the command below. Without this, a random name is generated that causes Event Alerts to fail.

cf create-service-key pcf-event-alerts-db migrations_service_key -c '{ "username": "migrations_user" }'