Installing PCF Log Search
This document describes how to install and configure Pivotal Cloud Foundry (PCF) Log Search.
Install from Ops Manager
To install PCF Log Search, follow the standard procedure below for installing Ops Manager tiles:
Download the product file from the Log Search section of Pivotal Network.
Upload the product file to your Ops Manager installation.
Click Add next to the uploaded product description in the Available Products view to add this product to your staging area.
Configure Assign AZs and Networks.
Note: Do not balance jobs between multiple AZs if your environment has relatively high network latency between AZs ( > 5ms ). You must have consistent low latency between AZs to balance jobs between them or you risk partial data loss. Contact your IAAS provider if you are unsure about the network latency between your AZs.
Configure Settings:
- Specify the log retention period in number of days. The default is
7
, the minimum is1
, and the maximum is15
. - Specify the NATS Credentials for the BOSH Director, which you can find under the Credentials tab on the BOSH Director tile.
- Review the max queue length field. By default, the queue node has 2 GB of RAM, and the max queue length is set to
1000000
. As a general rule, increase this value by 1,000,000 for every 2 GB of RAM in the queue node. You can view the queue node RAM in the Resource Config section under the VM Type column for the Queue job.
- Specify the log retention period in number of days. The default is
Click Apply Changes.
Configure Pivotal Application Service and Data Service Logs
You can configure Pivotal Application Service (PAS) and other PCF Data Service tiles to send their component logs to the Log Search router through syslog over TCP.
Note: PCF Log Search does not support syslog over UDP.
In the Log Search tile, click Status and record the IP address of the Router VM.
In the Pivotal Application Service tile, click System Logging. Complete the following fields:
- External Syslog Aggregator Hostname: Enter the IP address from step 1.
- External Syslog Aggregator Port: Enter
514
. - External Syslog Network Protocol: Choose
TCP
.
In the RabbitMQ for PCF tile, update the syslog forwarder settings with the IP address from step 1, protocol
TCP
, and port514
.For any other tiles you want to analyze with Log Search, such as MySQL, Redis, and Spring Cloud Services, update the syslog forwarder settings using the IP address from step 1, port
514
, and protocolTCP
.Click Apply Changes.
Consume App Logs from the PAS Firehose
By default, Log Search attaches to the PAS Firehose and consumes platform metrics, specifically the ValueMetric
, CounterEvent
and Error
types. You can also configure Log Search to consume app logs and metrics, specifically the ContainerMetric
, LogMessage
, and HttpStartStop
types.
Find the Rate of Incoming Messages
You can discover the rate of messages coming from the PAS Firehose by using the nozzle plugin for the cf CLI.
Run the following command to get a rough estimate of the volume of LogMessages
coming from the Firehose. The command outputs the amount of LogMessages
per second every 10 seconds.
$ cf nozzle -filter LogMessage | pv -l -i10 -r >/dev/null [50.5 /s]
Configure Log Search to Receive App Logs
WARNING: Some applications send a large amount of application logs over the Firehose, so you will need to scale up your cluster to handle the additional load. See the Scaling topic for more information.
In the Log Search tile in Ops Manager, click Experimental - Firehose and check the additional Firehose event types you want to consume.
Click Apply Changes.