Streaming App Logs to Log Management Services
Page last updated:
Warning: Pivotal Cloud Foundry (PCF) v2.5 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy. To stay up to date with the latest software and security updates, upgrade to a supported version.
This topic describes how to drain logs from Pivotal Application Service (PAS) to a third-party log management service.
PAS aggregates logs for all instances of your apps as well as for requests made to your apps through internal components of PAS. For example, when the PAS Gorouter forwards a request to an app, the Gorouter records that event in the log stream for that app. To access the log stream for an app in the terminal, run:
cf logs YOUR-APP-NAME
YOUR-APP-NAME is the name of the app whose log stream you want to access.
If you want to persist more than the limited amount of logging information that PAS can buffer, drain these logs to a log management service.
For more information about the systems responsible for log aggregation and streaming in PAS, see App Logging in PAS.
Your PCF marketplace may offer one or more log management services. To use one of these services, create an instance of the service and bind it to your app by running:
cf create-service SERVICE PLAN SERVICE-INSTANCE cf bind-service YOUR-APP YOUR-LOG-STORE
SERVICEis the name of the log management service you want to create an instance of.
PLANis the name of a plan that meets your needs. Service providers use plans to offer varying levels of resources or features for the same service.
SERVICE-INSTANCEis the name you provide for your service instance. You use this name to refer to your service instance with other commands. Service instance names can include alpha-numeric characters, hyphens, and underscores, and you can rename the service instance at any time.
YOUR-APPis the app to which you want to bind the service.
YOUR-LOG-STOREis the log store where you store your app logs.
For more information about service instance lifecycle management, see Managing Service Instances with the cf CLI.
Note: Not all marketplace services support syslog drains. Some services implement an integration with PCF that enables automated streaming of app syslogs. If you are interested in building services for Cloud Foundry and making them available to end users, see the Custom Services documentation.
If a compatible log management service is not available in your PCF marketplace, you can use user-provided service instances to stream app logs to a service of your choice. For more information, see the Stream App Logs to a Service section of the User-Provided Service Instances topic.
You can install and use the CF Drain CLI Plugin to create and manage user-provided syslog drains from the cf CLI.
You may need to prepare your log management service to receive app logs from PAS. For specific instructions for several popular services, see Service-Specific Instructions for Streaming App Logs. If you cannot find instructions for your service, follow the generic instructions below.
To set up a communication channel between the log management service and your PAS deployment:
Obtain the external IP addresses that your Cloud Foundry administrator assigns to outbound traffic.
Provide these IP addresses to the log management service. The specific steps to configure a third-party log management service depend on the service.
Whitelist these IP addresses to ensure unrestricted log routing to your log management service.
Record the syslog URL provided by the third-party service. Third-party services typically provide a syslog URL to use as an endpoint for incoming log data. You use this syslog URL in Step 2: Create a User-Provided Service Instance.
PAS uses the syslog URL to route messages to the service. The syslog URL has a scheme of
https, and can include a port number. For example:
Note: PAS does not support using
https with self-signed certificates. If you are running your own syslog server and want to use
https, you must have an SSL certificate signed by a well-known certificate authority.
You can create a syslog drain service and bind apps to it using either generic cf CLI commands, or drain-specific commands enabled by the CF Drain plugin for the cf CLI.
If the CF Drain CLI Plugin is not installed on your local workstation, see Installing Plugin in the CF Drain CLI repository on GitHub.
Decide whether to bind the drain to a single app or all apps in a space, and run the corresponding command:
cf drain APP-NAME SYSLOG-DRAIN-URL
APP-NAMEis name of the app to stream logs from
SYSLOG-DRAIN-URLis the syslog URL from Step 1: Configure the Log Management Service
All apps in a space:
cf drain-space --drain-name DRAIN-NAME --drain-url SYSLOG-DRAIN-URL --username USERNAME
DRAIN-NAMEis the name of the app to stream logs from.
SYSLOG-DRAIN-URL: The syslog URL from Step 1: Configure the Log Management Service.
USERNAME: Username to use when pushing the app. If you do not specify a username, you must have admin permissions because the plugin will create a user.
After a short delay, logs begin to flow automatically.
Note: To bind a drain to all apps in a space with a single command, you must use the CF Drain CLI Plugin as described in the previous section.
To create the service instance, run:
cf create-user-provided-service DRAIN-NAME -l SYSLOG-URL
To bind an app to the service instance, either:
cf pushwith a manifest. The services block in the manifest must specify the service instance that you want to bind.
cf bind-service YOUR-APP-NAME DRAIN-NAME
DRAIN-NAMEis a name to use for your syslog drain service instance. *
SYSLOG-DRAIN-URLis the syslog URL from Step 1: Configure the Log Management Service.
After a short delay, logs begin to flow automatically.
For more information, see Managing Service Instances with the CLI.
To verify that logs are draining correctly to a third-party log management service:
Take actions that produce log messages, such as making requests of your app.
Compare the logs displayed in the CLI against those displayed by the log management service.
For example, if your app serves web pages, you can send HTTP requests to the app. In PAS, these generate Gorouter log messages, which you can view in the CLI. Your third-party log management service should display corresponding messages.
Note: For security reasons, PAS apps do not respond to
ping. You cannot use
ping to generate log entries.
The CF Drain CLI plugin extends the cf CLI by adding simple commands for user-provided syslog drains. Also, you can use the plugin to bind all apps in a space to a syslog drain. This option includes app, space, and org names in the drain. It also binds any new apps pushed to the space.
To install the CF Drain CLI plugin, see Installing Plugin in the CF Drain CLI repository on GitHub.
The plugin adds commands for creating, deleting, and listing syslog drains, and for binding apps to the drains. For more information, see Usage in the CF Drain CLI repository on GitHub.