Streaming App Logs with Fluentd
Page last updated:
Fluentd is an open source log collector that allows you to implement unified logging layers. With Fluentd, you can stream app logs to different backends or services like Elasticsearch, HDFS and Amazon S3. This topic explains how to integrate Fluentd with Cloud Foundry apps.
Step 1: Create a Cloud Foundry Syslog Drain for Fluentd
In Cloud Foundry, create a syslog drain user-provided service instance as described in Using Third-Party Log Management Services.
Choose one or more apps whose logs you want to drain to Fluentd through the service.
Bind each app to the service instance, and restart the app.
Note the GUID for each app, the IP address of the Loggregator host, and the port number for the service.
Locate the port number in the syslog URL. For example:
Step 2: Set up Fluentd for Cloud Foundry
This section assumes you have an active Fluentd instance running. If you do not have an active Fluentd instance, refer to the Fluentd Documentation/Install steps for more details. If you use cf to deploy your fluentd instances, you will have to use tcp routing.
Fluentd comes with native support for syslog protocol. To set up Fluentd for Cloud Foundry, configure the syslog input of Fluentd as follows.
In your main Fluentd configuration file, add the following
<source> @type syslog port 8080 bind 0.0.0.0 tag cf.app message_length_limit 99990 frame_type octet_count <transport tcp> </transport> <parse> message_format rfc5424 </parse> </source>
Restart the Fluentd service.
Note: The Fluentd syslog input plugin supports
tcp options. Make sure to use the same transport that Cloud Foundry is using.
Fluentd will start listening for Syslog message on port 8080 and tagging the messages with
cf.app, which can be used later for data routing. For more details about the full setup for the service, refer to the Config File article.
If your goal is to use an Elasticsearch or Amazon S3 backend, read the following guide: http://www.fluentd.org/guides/recipes/elasticsearch-and-s3