Configuring Logging in PAS

Page last updated:

This topic describes the types of logs that Pivotal Cloud Foundry (PCF) generates. It also explains how to forward system logs to an external aggregator service, how to scale Loggregator component VMs to keep up with app log volume, and how to manage app traffic logging.

System Logs, App Logs, App Traffic Logs

Pivotal Cloud Foundry (PCF) generates two types of logs, system logs from PCF components and app logs from hosted apps, as differentiated in the table below:

Log Type Originate from Follow format Stream from Can to stream out to (configurable) Visible to
System Logs Platform components Syslog standard rsyslog agent Component syslog drain Operators
App Logs Hosted apps Format is up to the developer Firehose1 External data platform, optionally via nozzles Developers and Operators
(Optional) With Syslog Adapter
Converted to syslog standard Syslog Adapter External syslog drain

1The Loggregator Firehose also streams component metrics.

App traffic logs are system logs. When app containers communicate, or attempt to communicate, their host cells generate app traffic logs. App traffic logs are system logs, not app logs. These logs come from host cells, not apps, and they carry no information from within the app. App traffic logs only show app communication behavior, as detected from outside by the host cell.

Log Cache

Log Cache is a Loggregator feature that lets you filter and query app logs through a CLI plugin or API endpoints. Cached app logs are available on demand; you do not need to stream them continuously.

Example Uses of the Log Cache CLI Plugin

To access cached logs with the Log Cache CLI plugin, you must first download and install the plugin.

Download the Log Cache CLI plugin.

One you have installed the plugin, the basic command to access cached app logs is:

$ cf tail [OPTIONS] [SOURCE ID/APP]

Here are some ways you can use Log Cache to filter app logs:

  • --start-time: Displays the start of the cache or the start of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with --end-time to view logs within a time period.
  • --end-time: Displays the end of the cache or the end of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with --start-time to view logs within a time period.
  • --json: Output logs in JSON.
  • --follow: Append exported logs to stdout.

For more information on using the Log Cache CLI, see Log Cache CLI: Usage.

Example Uses of the Log Cache API

The Log Cache API is hosted on PCF, and references your system domain to return responses. The root URL for API calls is https://log-cache.[YOUR-SYSTEM-DOMAIN].

The basic call to access and filter cached app logs is:

GET https://log-cache.[YOUR-SYSTEM-DOMAIN]/v1/read/[SOURCE-ID]

Append the following parameters to the GET to customize app logs:

  • start_time: Displays the start of the cache or the start of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with end_time to view logs within a time period.
  • end_time: Displays the end of the cache or the end of a time period you specify. Results display as a UNIX timestamp, in nanoseconds. Pair with start_time to view logs within a time period.
  • envelope_types: Filters by Envelope Type. The available filters are: LOG, COUNTER, GAUGE, TIMER, and EVENT. Set an envelope type filter to emit logs of only that type. Specify this parameter multiple times to include more types.
  • limit: Sets a maximum number of envelopes to request. The max limit is 1000. This value defaults to 100.

More API parameters are available to customize retrieved app logs. For more information, see Log Cache: RESTful API Gateway.

Enable Syslog Forwarding

You can configure system logging in PAS to forward log messages from PAS component VMs to an external service. Pivotal recommends forwarding logs to an external service for use in troubleshooting.

Note: The following instructions explain how to configure system logging for PAS component VMs. To forward logs from PCF tiles to an external service, you must also configure system logging in each tile. See the documentation for the given tiles for information about configuring system logging.

To configure system logging in PAS, do the following:

  1. In the PAS Settings tab, select the System Logging pane. The following image shows the System Logging pane. There is a heading at the top of the System Logging page that begins with the text, Optionally configure rsyslog. There are several fields on the System Logging page. The first three fields are Address, Port, and Transport Protocol. There is a Save button at the bottom of the page.

  2. For Address, enter the IP address of the syslog server.

  3. For Port, enter the port of the syslog server. The default port for a syslog server is 514.

    Note: The host must be reachable from the PAS network and accept UDP or TCP connections. Ensure the syslog server listens on external interfaces.

  4. For Transport Protocol, select a transport protocol for log forwarding.

  5. For Encrypt syslog using TLS?, select Yes to use TLS encryption when forwarding logs to a remote server.

    1. For Permitted Peer, enter either the name or SHA1 fingerprint of the remote peer.
    2. For TLS CA Certificate, enter the TLS CA certificate for the remote server.
  6. For Number of syslog drain addresses to retrieve, enter the number of apps with syslog drain addresses to retrieve from Cloud Controller per request.

  7. Disable the Include container metrics in Syslog Drains checkbox to prevent the CF Drain CLI plugin from including app container metrics in syslog drains. This feature is enabled by default.

  8. Enable Enable agent-based syslog egress for app logs to use agent-based syslog. Agent-based syslog provides an alternative to scalable-syslog architecture. For more information about agent-based syslog, see Loggregator Architecture.

    Note: Selecting this checkbox disables the Syslog Adapter and Syslog Scheduler VMs to prevent log duplication.

  9. Enable the Enable Cloud Controller security event logging checkbox to include security events in the log stream. This feature logs all API requests, including the endpoint, user, source IP address, and request result, in the Common Event Format (CEF).

  10. Enable the Use TCP for file forwarding local transport checkbox to transmit logs over TCP. This prevents log truncation, but may cause performance issues.

  11. Disable the Don’t Forward Debug Logs checkbox to forward DEBUG syslog messages to an external service. This checkbox is enabled by default.

    Note: Some PAS components generate a high volume of DEBUG syslog messages. Enabling the Don’t Forward Debug Logs checkbox prevents PAS components from forwarding the DEBUG syslog messages to external services. However, PAS still writes the messages to the local disk.

  12. For Custom rsyslog Configuration, enter a custom syslog rule. For more information about adding custom syslog rules, see Customizing Syslog Rules.

  13. Click Save.

To configure Ops Manager for system logging, see the Settings section in the Using the Ops Manager Interface topic.

Include Container Metrics in Syslog Drains

Developers can monitor container metrics over the syslog protocol using the CF Drain CLI plugin. With the CF Drain CLI plugin, you can use the Cloud Foundry Command Line Interface (cf CLI) tool to set the app container to deliver container metrics to a syslog drain. Developers can then monitor the app container based on those metrics.

In PAS, the Include container metrics in Syslog Drains checkbox is enabled by default in the System Logging pane. If you have security concerns about streaming container metrics from your app, you can disable this checkbox.


For more information, see Including Container Metrics in Syslog Drains in Application Logging in Cloud Foundry.

Scale Loggregator

Apps constantly generate app logs and PCF platform components constantly generate component metrics. The Loggregator system combines these data streams and handles them as follows. See Loggregator Architecture for more information.

  • The Loggregator agent running on each component or application VM collects and sends this data out to Doppler components.
  • Doppler components temporarily buffer the data before periodically forwarding it to the Traffic Controller. When the log and metrics data input to a Doppler exceeds its buffer size for a given interval, data can be lost.
  • The Traffic Controller serves the aggregated data stream through the Firehose WebSocket endpoint.

Follow the instructions below to scale the Loggregator system. For guidance on monitoring and capacity planning, see Monitoring Pivotal Cloud Foundry.

Add Component VM Instances

  1. From the PAS tile, select Resource Config.

  2. Increase the number in the Instances column of the component you want to scale. You can add instances for the following Loggregator components:

    • Loggregator Traffic Controller

      Note: The Reverse Log Proxy (RLP) BOSH job is colocated on the Traffic Controller VM. If you want to scale Loggregator to handle more logs for syslog drains, you can add instances of the Traffic Controller.

      Note: The BOSH System Metrics Forwarder job is colocated on the Traffic Controller VM. If you want to scale Loggregator to handle more BOSH component metrics, you can add instances of the Traffic Controller.

    • Syslog Adapter
    • Doppler Server Loggregator vms
  3. Click Save.

  4. Click Review Pending Changes, then Apply Changes.

App Traffic Logging

App traffic logging generates logs when app containers communicate with each other directly, or attempt to communicate, as allowed by container-to-container networking (C2C) policies and Application Security Groups (ASGs).

App traffic logging lets network security teams audit C2C traffic, by seeing allowed and denied packets, without needing access to the Cloud Controller or the apps themselves.

Enable App Traffic Logging

To enable app traffic logging:

  1. From Ops Manager, navigate to the Pivotal Application Service tile > Networking pane.

  2. Under Log traffic for all accepted/denied application packets, select Enable (will increase log volume) or Disable to enable or disable app traffic logging.

Log app traffic enable

App Logging Behavior

App traffic logging generates log messages as follows:

  • TCP traffic - Logs the first packet of every new TCP connection.
  • UDP traffic - Logs UDP packets sent and received, up to a maximum per-second rate for each container. Set this rate limit in the UDP logging interval field (default: 100).
  • Packets denied - Logs packets blocked by either a container-specific networking policy or by Application Security Group (ASG) rules applied across the space, org, or deployment. Logs packet denials up to a maximum per-second rate for each container, set in the Denied logging interval field (default: 1).

App Traffic Log Format and Contents

App traffic logs are formatted as described in the cf-networking-release Traffic logging documentation, following the iptables-logger format but without line breaks. For example, the first part of an app traffic log line looks like: {"timestamp": "1500924070.182554722", "source": "cfnetworking.iptables", "message": "cfnetworking.iptables.ingress-allowed", "log_level": 1, "data": { "destination": { "container_id": "d5978989-1401-49ff-46cd-33e5","app_guid": "bc6f229d-5e4a-4c41-a63f-e8795496c283",.

Each log message includes the following:

  • Timestamp
  • The GUID for the source or destination app that sent or was designated to receive the packet
  • The protocol of the communication, TCP or UDP
  • GUIDs for the container, space, and org running the source or destination app
  • IP addresses and ports for both source and destination apps
  • A message field recording whether the packet was allowed or denied, with one of the following four possibilities:
    • ASG allowed packet to exit source app container
    • C2C policy allowed packet to enter destination app container
    • ASG prevented packet from exiting source app container
    • C2C policy prevented packet from entering destination app container
  • Additional information described in the cf-networking-release.

Denied Packet Causes

You can determine whether a denied-packet log resulted from a container networking policy or an ASG rule as follows:

  • Container networking policy: Log message string includes ingress-denied and packet direction is ingress.

  • ASG rule: Log message string includes egress-denied and packet direction is egress.

Global vs. ASG-Level App Traffic Logging

PCF supports two mechanisms for enabling app traffic logging. Setting Log traffic to Enable in Ops Manager enables app traffic logging globally for all ASGs and container policies. Setting the log property of an ASG to true enables app traffic logging at the individual ASG level.

Because these two mechanisms operate independently, PCF generates duplicate logs when app traffic logging is enabled globally and an ASG’s log property is set to true. To avoid duplicate logs, Pivotal recommends setting the log property to false for all ASGs, or leaving it out entirely, when app traffic logging is enabled globally.

To focus on specific ASGs for log analysis, Pivotal recommends enabling app traffic logs globally and using a logging platform to filter traffic logs by ASG, rather than setting log at the individual ASG level.