Troubleshooting PCF Metrics

This topic describes how to resolve common issues experienced while operating or using Pivotal Cloud Foundry (PCF) Metrics.

Insufficient Resources

Missing Specific Logs

High CPU on PostgreSQL VM

Too Many Clients Error

Failed to Fetch Apps

Redis Temporary Datastore Stops Accepting Metrics

Received No Results Back from MySQL - Failing

Failed to Connect to MySQL

Never Received App Logs

Metrics and Events Not Available

Logs and Histograms Not Available

No Logs or Metrics in the UI

In some cases, the PCF Metrics UI might not display metrics and logs after successfully deploying.

Follow the steps in this section to help locate the app or component causing the problem.

Step 1: Check your Load Balancer Configuration

If you use a load balancer, the event-stream mechanism used by the Metrics UI might be blocked. See the table below to resolve this error.

If you do not use a load balancer, or this issue does not apply to your deployment, proceed to the next step.

Step 2: Check the PCF Metrics Apps

1. From Ops Manager, click the PAS Tile.

   1. Click the Credentials tab.
   2. Under the UAA job, next to Admin Credentials, click Link to Credential.
   3. Record the username and password for use in the next step.

2. Log in to the Cloud Foundry Command Line Interface (cf CLI) using the credentials from the previous step.

   $ cf login -a https://api.YOUR-SYSTEM-DOMAIN -u admin -p PASSWORD

3. When prompted, select the system org and the metrics-v1-6 space.

4. Ensure that the output displays the following apps, each in a started state:

   • metrics-ingestor
   • metrics-queue
   • logs-queue
   • metrics
   • metrics-ui
   • metrics-alerting

5. Check the logs of each app for errors using the following command:

   $ cf logs APP-NAME --recent

   If you do not see any output, or if you did not find any errors, proceed to the next step.

Step 3: Check the Metrics Ingestor

1. To get a higher level of detail from the metrics-ingestor application, set the LOG_LEVEL env variable:

   $ cf set-env metrics-ingestor LOG_LEVEL DEBUG

2. To apply this setting, restage the application:

   $ cf restage metrics-ingestor

3. Run the following command to stream logs:

   $ cf logs metrics-ingestor

4. Watch the logs emitted by the metrics-ingestor app for errors. See the list below for common errors:

   • Aggregation Stored Procedures key is not in redis: Redis may have been restarted or is in a bad state. Stop the metrics-ingestor app. Restart redis. Start the metrics-ingestor app again.
   • Cannot connect to the firehose: PCF Metrics creates a UAA user to authenticate the connection to the Firehose. This user must have the doppler.firehose authority.
   • Could not find service with name: metrics-forwarder: The Metrics Forwarder Tile is not installed. Metrics will not display custom metrics without the Metrics Forwarder Tile but will otherwise function normally.
   • WebSocket Disconnects: If you see WebSocket disconnects logs in the Ingestor app, consider adding additional Ingestor instances. The Firehose might be dropping the Ingestor connection to avoid back pressure.
   • Redis errors: Investigate redis logs, for instructions see Advanced Troubleshooting with the BOSH CLI. Many possible solutions start with restarting redis.
     1. If the app is started and you do not find any errors, proceed to the next step.

Step 4: Check the log-queue

1. To get a higher level of detail from the logs-queue application, set the LOG_LEVEL env variable:

   $ cf set-env logs-queue LOG_LEVEL DEBUG

2. To apply this setting, restage the application:

   $ cf restage logs-queue

3. Run the following command to stream logs:

   $ cf logs logs-queue

4. Watch the logs emitted by the logs-queue app for errors.

   • A common error is that the app cannot connect to PostgreSQL due to the application security group (ASG) being deleted. This ASG allows the logs-queue application to create a network connection to the PostgreSQL VM. You can run cf security-group metrics-api to see if the ASG exists. If the ASG is not present, see Creating Application Security Groups to recreate it.
   • Could not find service with name: metrics-forwarder: The Metrics Forwarder Tile is not installed. Metrics will not display custom metrics without the Metrics Forwarder Tile but will otherwise function normally.
     1. If the app is started and you do not find any errors, proceed to the next step.

Step 5: Check MySQL

1. From Ops Manager, select the PCF Metrics tile.

2. Under the Status tab, record the IP of a MySQL Server node.

3. Use bosh ssh to access the VM from the previous step. For instructions, see Advanced Troubleshooting with the BOSH CLI.

4. Log in to mysql by running mysql -u USERNAME -p PASSWORD

   Note: If you do not know the username and password, you can run cf env metrics-queue with the system org and the metrics-v1-6 space targeted.

5. Verify that the database was bootstrapped correctly:

   1. Run show databases and check for a metrics database.
      1. If there is no metrics database, the Push PCF Metrics Components Errand errand of the BOSH release might not have run or succeeded. Ensure the errand is selected in the tile configuration and update the tile.

6. Run use metrics to select the metrics database:

   mysql> use metrics;

7. Run show tables and ensure you see the following tables:

   mysql> show tables;
   +-----------------------------+
   | Tables_in_metrics           |
   +-----------------------------+
   | app_event                   |
   | app_metric                  |
   | app_metric_rollup           |
   | schema_version              |
   | app_metric_identifier       |
   +-----------------------------+
   

8. Enter the following query several times to verify that the value returned does not decrease over time:

   mysql> select count(*) from metrics.app_metric_identifier where timestamp > ((UNIX_TIMESTAMP() - 60) * POW(10, 3));

   This command displays the rate at which metrics flow in over the last minute.
   1. If the command returns 0 or a consistently decreasing value, the problem is likely further up in ingestion; proceed to the next step.

Step 6: Check the Metrics Queue

1. To get a higher level of detail from the metrics-queue application, set the LOG_LEVEL env variable:

   $ cf set-env metrics-queue LOG_LEVEL DEBUG

2. To apply this setting, restage the application:

   $ cf restage metrics-queue

3. Run the following command to stream logs:

   $ cf logs metrics-queue

4. Watch the logs emitted by the metrics-queue app for errors.

   • A common error is that the app cannot connect to MySQL due to the application security group (ASG) being deleted. This ASG allows the logs-queue application to create a network connection to the MySQL VM. You can run cf security-group metrics-api to see if the ASG exists. If the ASG is not present, see Creating Application Security Groups to recreate it.
   • Could not find service with name: metrics-forwarder: The Metrics Forwarder Tile is not installed. Metrics will not display custom metrics without the Metrics Forwarder Tile but will otherwise function normally.
     1. If the app is started and you do not find any errors, proceed to the next step.

MySQL Failure

In some cases, a MySQL server might fail to restart. The following two sections describe the known conditions that cause this failure as well as steps for diagnosing and resolving them.

Cause 1: Monit Timed Out

Diagnose

Follow these steps to see if a monit time-out caused the MySQL node restart to fail:

1. Use bosh ssh to access the failing node, using the IP address in the Ops Manager Director tile Status tab. For instructions, see Advanced Troubleshooting with the BOSH CLI.
2. Run monit summary and check the status of the galera-init job.
3. If the status of the galera-init job is Execution Failed, open the following file: /var/vcap/sys/log/pxc-mysql/galera-init.log.
   1. If the last line of the log indicates that MySQL started without issue, such as in the example below, monit likely timed out while waiting for the job to report healthy. Follow the steps below to resolve the issue.

      {"timestamp":"1536851105.372446537","source":"/var/vcap/packages/galera-init/bin/galera-init","message":"/var/vcap/packages/galera-init/bin/galera-init.galera-init started","log_level":1,"data":{}}
      

Resolve

Run the following commands to return the galera-init job to a healthy state:

1. Run monit unmonitor galera-init.
2. Run monit monitor galera-init.
3. Run monit summary and confirm that the output lists galera-init as running.

Cause 2: Bin Logs Filled up the Disk

Diagnose

1. Use bosh ssh to access the failing node. For instructions, see Advanced Troubleshooting with the BOSH CLI.
2. Open the following log file: /var/vcap/sys/log/pxc-mysql/mysql.err.log.
3. If you see log messages that indicate insufficient disk space, the persistent disk is likely storing too many bin logs. Confirm insufficient disk space by doing the following:
   1. Run df -h.
      1. Ensure that you see the /var/vcap/store folder is at or over 90% usage.
   2. Navigate to /var/vcap/store/pxc-mysql and run ls -al.
      1. Ensure that you see many files named with the format mysql-bin.########.

In MySQL for PCF, the server node does not make use of these logs and you can remove all except the most recent bin log. Follow the steps below to resolve the issue.

Resolve

1. Log in to mysql by running mysql -u USERNAME -p PASSWORD

   Note: If you do not know the username and password, you can run cf env metrics-queue with the system org and the metrics-v1-6 space targeted.

2. Run use metrics;.
3. Run the following command:

   mysql> PURGE BINARY LOGS BEFORE 'YYYY-MM-DD HH:MM:SS'; 

Edit Your MySQL Server Configuration

1. From Ops Manager, click the PCF Metrics Tile.
2. Navigate to the Resource Config section of the PCF Metrics Tile.
3. Increase the Persistent Disk size of MySQL Server to at least twice the current size.

Forward PCF Metrics Logs to a Syslog Endpoint

When using PCF Metrics 1.6 or higher on PCF 2.4 and beyond, you can configure the tile to forward its logs to a syslog endpoint for troubleshooting. Please see the “Forward PCF Metrics Logs to a Syslog Endpoint” section under the Configure the PCF Metrics Tile docs for more details.

Service metrics-forwarder does not exist

Metrics API Unavailable