Scaling PCF Log Search

This topic describes when and how to scale Pivotal Cloud Foundry (PCF) Log Search.

Prerequisite: The procedures in this topic assume that you are logged into Kibana. For more information, see Log in to Kibana.

Determine if You Need to Scale

Follow the steps below to view a histogram for your recent logs and identify if all logs are being indexed.

  1. Enter * in the Kibana search bar to search for all logs.

  2. Examine the histogram for a gap in recent logs as shown in the image below. A gap means that Log Search cannot keep up with indexing the incoming data. To address this issue, you can scale up your cluster to handle the load, reduce resource usage, or both. Falling behind on log indexing

Scale Up Your Cluster

Follow these steps to scale up your PCF Log Search cluster to handle more incoming data.

  1. On the Log Search tile in Ops Manager, click the Status tab. Log Search Status tab

  2. Examine the CPU and PERS DISK usage for the Elasticsearch Data nodes job. If the load is high, especially if CPU > 50% or PERS DISK > 70%, follow the steps below:

    1. In Kibana, enter a search for *.
    2. Identify the log volume per second by changing the interval to Second and examining the height of the bars. volume
    3. Navigate to the PCF Log Search tile and click Resource Config.
    4. Edit the values for Elasticsearch Data nodes and Log parser in the Resource Config section of the PCF Log Search tile. Use the table below as a guide. As a general rule, use more Log parser nodes than Elasticsearch Data nodes.
      Log volume Elasticsearch Data nodes Log parser nodes
      200 / second 2 x medium.mem (~8GB RAM) with 50GB persistent disks 2 x small (1 CPU)
      2,000 / second ~10 x xlarge.mem (~32GB RAM) with 200GB persistent disks ~15 x medium (2 CPU)
      10,000 / second ~30 x xlarge.mem (~32GB RAM) with 500GB persistent disks ~40 x medium (2 CPU)

Reduce Resource Usage

PCF produces a large volume of logs. As a result, Log Search can consume significant resources. Use the following strategies to reduce the resources consumed by Log Search.

Reduce the Log Retention Period

Shorter retention periods dramatically reduce the amount of disk resources required by Log Search. Follow these steps to reduce the log retention period for Log Search:

  1. Navigate to the Settings section of the PCF Log Search tile.
  2. Enter a lower value for Log retention period.
  3. Click Save and Apply Changes.

Index Less Data

If you attached Log Search to the Elastic Runtime firehose, application logs can make up a large portion of the total logs indexed. Follow these steps to index less data:

  1. In Kibana, navigate to the Visualize tab.
  2. Select Pie Chart, and then From a new search.
  3. Search for tags:LogMessage.
  4. In the buckets section, select Split Chart:

    1. For Aggregation, choose Terms.
    2. For Field, choose cf.app_id.
    3. (Optional) Increase or decrease the Size value to see results for more or fewer apps.
  5. Click the play button to update the visualization.

  6. Examine the pie chart to identify apps that produce a disproportionately high number of log entries.

  7. Contact the developers of those apps and ask them to reduce the logging verbosity.