Altoros Heartbeat Troubleshooting
This topic describes known issues and troubleshooting strategies.
Below is a list of common issues you may face when troubleshooting Heartbeat, as well as some ways to resolve them.
An error occurs when trying to open Grafana: Probably, there is an issue with MySQL. Please, check the status of MySQL VMs and the corresponding logs. If the MySQL cluster is down, run the following command:
bosh -d altoros-heartbeat-... run-errand bootstrap
No new metrics appear in the dashboards: You have likely run out of disk space. Run the following command:
bosh -d altoros-heartbeat-... ssh heartbeat -c 'df -h /var/vcap/store' | grep stdoutIf there is no free space left, you must scale Persistent Disk Type for Heartbeat PCF Monitoring backend job VMs (Resource Config page). You will also receive an alert if any free space is available on Heartbeat’s back ends.
You experience a delay in graphs visualization after using Heartbeat for a while: Run the following command:
bosh -d altoros-heartbeat-... ssh heartbeat -c 'ss -o state close-wait | wc -l' | grep stdoutIf it returns a big value as an output (e.g. exceeding 100), run the following command:
bosh -d altoros-heartbeat-... ssh heartbeat -c 'sudo /var/vcap/bosh/bin/monit restart go-carbon'You will also get an alert identifying the number of the CLOSE_WAIT connections on Heartbeat’s back ends.
If the MySQL cluster used by your Heartbeat installation is broken, see Сluster Behavior and Bootstrapping documentation of the MySQL BOSH release for guidance on how to eliminate the problem, or contact Altoros for help.