Altoros Heartbeat Troubleshooting

This topic describes known issues and troubleshooting strategies.

Altoros Heartbeat Troubleshooting

Below is a list of common issues you may face when troubleshooting Heartbeat, as well as some ways to resolve them.

  1. An error occurs when trying to open Grafana: Probably, there is an issue with MySQL. Please, check the status of MySQL VMs and the corresponding logs. If the MySQL cluster is down, run the following command:

    bosh -d altoros-heartbeat-... run-errand bootstrap

  2. No new metrics appear in the dashboards: You have likely run out of disk space. Run the following command:

    bosh -d altoros-heartbeat-... ssh heartbeat -c 'df -h /var/vcap/store' | grep stdout
    If there is no free space left, you must scale Persistent Disk Type for Heartbeat PCF Monitoring backend job VMs (Resource Config page). You will also receive an alert if any free space is available on Heartbeat’s back ends.

  3. You experience a delay in graphs visualization after using Heartbeat for a while: Run the following command:

    bosh -d altoros-heartbeat-... ssh heartbeat -c 'ss -o state close-wait | wc -l' | grep stdout
    If it returns a big value as an output (e.g. exceeding 100), run the following command:
    bosh -d altoros-heartbeat-... ssh heartbeat -c 'sudo /var/vcap/bosh/bin/monit restart go-carbon'
    You will also get an alert identifying the number of the CLOSE_WAIT connections on Heartbeat’s back ends.

MySQL Troubleshooting

If the MySQL cluster used by your Heartbeat installation is broken, see –°luster Behavior and Bootstrapping documentation of the MySQL BOSH release for guidance on how to eliminate the problem, or contact Altoros for help.

Create a pull request or raise an issue on the source for this page in GitHub