Parsing Rules for Altoros Log Search for PCF

This topic describes default configuration of parsing rules for Altoros Log Search for Pivotal Cloud Foundry (PCF).

Indices

By default, Altoros Log Search for PCF stores parsed logs in indices named logs-%{[@metadata][index]}-%{+YYYY.MM.dd}. %{[@metadata][index]} is calculated as the following:

  • platform for Pivotal Cloud Foundry Elastic Runtime logs
  • app-%{[cf][org]}-%{[cf][space]} for Pivotal Cloud Foundry Elastic Runtime application logs.

So the following indices are created as the result:

  • logs-platform-%{+YYYY.MM.dd} for Pivotal Cloud Foundry Elastic Runtime platform logs.
  • logs-app-%{[cf][org]}-%{[cf][space]}-%{+YYYY.MM.dd} for Pivotal Cloud Foundry Elastic Runtime application logs.

Also it can be useful to read how to get list of all indices in Elasticsearch.

Fields

Altoros Log Search provides Logstash parsing rules which are used to parse incoming log events and create sets of fields from parsed data. Some fields are common for application and platform logs, some are event-specific. Apart from that, there are system fields added by Logstash. Read below sections to get detailed information on how fields the logs are split when using Altoros Log Search.

Common Fields

These fields are common for application and platform logs. They store the following information from a log event:

  • Log input (@input, @index_type)
  • Log shipping (@shipper.* fields)
  • Log source (@source.* fields)
  • Log destination in Elasticsearch (@metadata.index, @type)
  • Log message payload (@message, @level)
FieldValue examplesComment
@inputsyslog, relp, …
@index_typeapp, platformEither app or platform. Default is platform.
@metadata.indexplatform, app-myorg-myspace, …Constructed as app-${org}-${space} for application logs. Note that ${space} and ${org} are ommitted in index name if corresponding info is missing in log event.
@shipper.priority6, 14, …
@shipper.namedoppler_syslog, vcap.nats_relp, …
@source.host192.168.111.63, …
@source.deploymentcf, …For application logs this value is shipped within a log event.
@source.jobapi_z1, …
@source.job_index52ba268e-5578-4e79-afa2-2ddefd70badg, …Bosh ID of the job (guid) — value of spec.id extracted from Bosh for the job
@source.index0, 1, …Bosh instance index - value of spec.index extracted from Bosh for the job
@source.vmapi_z1/0For those entries where @source.index is passed, calculated as @source.job/@source.index
@source.componentrep, nats, bbs, uaa, …
@source.typeAPP, RTR, STG, system, cf, …For application logs the field is set with CloudFoundry log source types. Additionally, for log events that don’t specify a source type we use, a dictionary based on an event type: LogMessage -> LOG, Error -> ERR, ContainerMetric -> CONTAINER, ValueMetric -> METRIC, CounterEvent -> COUNT, HttpStartStop -> HTTP. For platform logs the value is either system or cf.
@typeLogMessage, Error, ValueMetric, system, cf, haproxy, uaa, vcap, …The field is used to define documents type in Elasticsearch. This field is set with values distinguishing logs of different types.
@messageThis is a sample log message text
@levelINFO, ERROR, WARN, …
@raw<13>2016-09-26T18:20:25.134194+00:00 192.168.111.63 vcap.rep [job=cell_z1 index=0] My log messageThis field stores an unparsed log event (as it came).
@timestampSeptember 26th 2016, 21:04:17.928The field is set with value of a log event timestamp (time when the log was collected by CloudFoundry logging agent).
tagssyslog_standard, app, logmessage, logmessage-app, …This field stores tags set during parsing. A specific tag is set in each parsing snippet which helps to track parsing (name of tag = name of snippet). Fail tags are set in case of parsing failures.

Application Meta Fields

These fields are specific to application logs only. They store Pivotal Cloud Foundry Elastic Runtime metadata about an application that emmitted the log or relates to the log event (e.g. metrics).

FieldValues
@cf.org_id2d5f8dc7-dcf4-443b-9491-a54d27db785f, …
@cf.orgmyorg, …
@cf.space_idc9290e71-780b-43ee-8074-f37ee33b2ff7, …
@cf.spacemyspace, …
@cf.app_idee61d1b6-f08f-4f93-b93f-2a9b0ae82dfc, …
@cf.appmyapp, …
@cf.app_instance0, 1, 2, …

Event Specific Fields

  • Application logs are shipped as JSON events. Set of JSON fields varies for different event types. All common fields from JSON are mapped accordingly to common fields and Elastic Runtime meta fields listed above. Other JSON fields (those extra fields specific to a particular event type) are stored as <@type>.<json field name>. Example: logmessage.message_type. Additionally, format of a log line (message shipped in a log event) may vary for different events. Parsed fields from a log line are stored as <@source.type>.<field name>. Example: rtr.path.

  • Platform logs are shipped in events of plain text format. The format is parsed and common fields are set from the parsed data. A format of a log line (message shipped in a log event) may vary for different event types. For consistency we store fields parsed from the log line as <@source.component>.<field name>. Example: uaa.pid.

Elasticsearch Meta Fields

Each parsed log event also has: a set of Elasticsearch meta fields (prefixed with _ underscore).

Create a pull request or raise an issue on the source for this page in GitHub