Monitoring PKS with Sinks

Page last updated:

Warning: Pivotal Container Service (PKS) v1.3 is no longer supported because it has reached the End of General Support (EOGS) phase as defined by the Support Lifecycle Policy.
To stay up to date with the latest software and security updates, upgrade to a supported version.

This topic describes how to monitor Pivotal Container Service (PKS) workloads using sink resources.

Prerequisites

Using sink resources for monitoring requires that you have set up a log processing solution capable of log ingress over TCP as described in RFC 5424.

In addition, you must configure sinks in PKS to send logs to that destination. For information on how to create and manage sinks in PKS, see Creating Sink Resources.

Note: Sinks created in PKS only support TCP connections. UDP connections are not currently supported.

About Sink Log Entries

Sinks and ClusterSinks include both pod logs as well as events from the Kubernetes API.

These logs and events are combined in a shared format to provide operators with a robust set of filtering and monitoring options.

Sink data has the following characteristics. All entries:

• Are timestamped.
• Contain the host ID of the BOSH-defined VM.
• Are annotated with a set of structured data, which includes the namespace, the object name or pod ID, and the container name.

Sink Log Entry Format

All sink log entries use the following format:

APP-NAME/NAMESPACE/POD-ID/CONTAINER-NAME

Where:

• APP-NAME is pod.log or k8s.event.
• NAMESPACE is the namespace associated with the pod log or Kubernetes event.
• POD-ID is the ID of the pod associated with the pod log or Kubernetes event.
• CONTAINER-NAME is the workload associated with the pod log or Kubernetes event.

Pod Logs

Pod logs entries are distinguished by the string pod.log in the APP-NAME field.

Pod Log Example

The following is a sample pod log entry:

36 <14>1 2018-11-26T18:51:41.647825+00:00 vm-3ebfe45d-492d-4bfd-59c4-c45d91688c65 
pod.log/rocky-raccoon/logspewer-6b58b6689d-dhddj - - [kubernetes@47450 
app="logspewer" pod-template-hash="2614622458" namespace_name="rocky-raccoon" 
object_name="logspewer-6b58b6689d-dhddj" container_name="logspewer"] 
2018/11/26 18:51:41 Log Message 589910

Where:

• vm-3ebfe45d-492d-4bfd-59c4-c45d91688c65 is the host ID of the BOSH VM.
• pod.log is the APP-NAME.
• rocky-raccoon is the NAMESPACE.
• logspewer-6b58b6689d-dhddj is the POD-ID.

Kubernetes API Events

Kubernetes API Event entries are distinguished by the string k8s.event in the APP-NAME field.

Kubernetes API Event Example

The following is an example Kubernetes API event log entry:

Nov 14 16:01:49 vm-b409c60e-2517-47ac-7c5b-2cd302287c3a 
k8s.event/rocky-raccoon/logspewer-6b58b6689d-j9n: 
Successfully assigned rocky-raccoon/logspewer-6b58b6689d-j9nq7 
to vm-38dfd896-bb21-43e4-67b0-9d2f339adaf1

Where:

• vm-b409c60e-2517-47ac-7c5b-2cd302287c3a the host ID of the BOSH VM.
• k8s.event is the APP-NAME.
• rocky-raccoon is the NAMESPACE.
• logspewer-6b58b6689d-j9n is the POD-ID.

Notable Kubernetes API Events

The following section lists Kubernetes API Events that can help assess any Kubernetes scheduling problems in PKS.

To monitor for these events, look for log entries that contain the Identifying String indicated below for each event.

Failure to Retrieve Containers from Registry

Malfunctioning Containers

Successful Scheduling of Containers

Failure to Schedule Containers

Related Links

For more information on sinks in PKS, see the following topics:

• For information about creating sinks in PKS, see Creating Sink Resources.
• For information about sink architecture, see Sink Architecture in PKS.