LATEST VERSION: 2.1 - CHANGELOG
Pivotal Cloud Foundry v2.0

Customizing Platform Log Forwarding

Page last updated:

You can configure Pivotal Application Service (PAS) to forward logs to remote endpoints using the Syslog protocol defined in RFC 5424. For more information, see Enable Syslog Forwarding in Configuring Logging in PAS.

PAS annotates forwarded messages with structured data. This structured data identifies the originating BOSH Director, deployment, instance group, availability zone, and instance ID. All logs forwarded from BOSH jobs have their PRI set to 14, representing Facility: user-level messages and Warning: warning conditions, as defined in RFC 5424 Section 6.2.1. This PRI value may not reflect the originally intended PRI of the log.

Logs forwarded from other sources, such as kernel logs, retain their original PRI value.

The following table describes the log line Structured Data:

Structured Data Description
ENTERPRISE_NUMBER Cloud Foundry’s private enterprise number, 47450, as defined in RFC 5424 Section 7.2.2.
DIRECTOR The name of the BOSH Director managing the deployment.
DEPLOYMENT The name of the BOSH deployment.
INSTANCE_GROUP The name of the BOSH instance_group.
AVAILABILITY_ZONE The name of the BOSH availability zone.
ID The BOSH GUID.

Log lines use the format below:

<$PRI>$VERSION $TIMESTAMP $HOST $APP_NAME $PROC_ID $MSG_ID   
    [instance@ENTERPRISE_NUMBER director="$DIRECTOR" deployment="$DEPLOYMENT" 
    group="$INSTANCE_GROUP" az="$AVAILABILITY_ZONE" id="$ID"] $MESSAGE

Example log messages:

<14>1 2017-01-25T13:25:03.18377Z 192.0.2.10 etcd - - [instance@47450 
    director="test-env" deployment="cf" group="diego_database" az="us-west1-a" 
    id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] [INFO] the leader is 
    [https://diego-database-0.etcd.service.cf.internal:4001]
<14>1 2017-01-25T13:25:03.184491Z 192.0.2.10 bbs - - [instance@47450 
    director="test-env" deployment="cf" group="diego_database" az="us-west1-a" 
    id="83bd66e5-3fdf-44b7-bdd6-508deae7c786"] 
    {"timestamp":"1485350702.539694548","source":"bbs","message":
    "bbs.request.start-actual-lrp.starting","log_level":1,"data":
    {"actual_lrp_instance_key":{"instance_guid":
    "271f71c7-4119-4490-619f-4f44694717c0","cell_id":
    "diego_cell-2-41f21178-d619-4976-901c-325bc2d0d11d"},"actual_lrp_key":
    {"process_guid":"1545ce89-01e6-4b8f-9cb1-5654a3ecae10-137e7eb4-12de-457d-
     8e3e-1258e5a74687","index":5,"domain":"cf-apps"},"method":"POST","net_info":
    {"address":"192.0.2.12","ports":[{"container_port":8080,"host_port":61532},
    {"container_port":2222,"host_port":61533}]},"request":
    "/v1/actual_lrps/start","session":"418.1"}}

Modify Which Logs PAS Forwards

When you enable log forwarding, PAS forwards all log lines written to the /var/vcap/sys/log directories on all Cloud Foundry virtual machines (VMs) to your configured External Syslog Aggregator endpoint by default.

You can configure PAS to forward a subset of logs instead of forwarding all logs as follows.

  1. In the PAS tile, select System Logging.
  2. In the Custom rsyslog Configuration textbox, enter a custom syslog rule. See the example custom syslog rules below.
  3. Click Save.

For more information about enabling and configuring syslog forwarding, see Configuring Logging in PAS.

The custom rsyslog rules shown below are written in RainerScript. The custom rules are inserted before the rule that forwards logs. The stop command, stop, prevents logs from reaching the forwarding rule. This filters out these logs.

Logs filtered out before forwarding remain on the local disk, where the BOSH job originally wrote them. These logs remain on the local disk only until BOSH Director recreates the VMs. You can access these logs from Ops Manager or through SSH.

Note: PAS requires a valid custom rule to forward logs. If your custom rule contains syntax errors, PAS forwards no logs.

Forward Only Logs From a Certain Job

This rule filters out logs unless they originate from the uaa job:

if ($app-name != "uaa") then stop

Exclude Logs With Certain Content

This rule filters out logs that contain “DEBUG” in the body.

if ($msg contains "DEBUG") then stop

Note: The above example contains “DEBUG” in the message body. Not all logs intended for debugging purposes contain the string “DEBUG” in the message body.

Create a pull request or raise an issue on the source for this page in GitHub