Monitoring PKS with Sinks

Page last updated:

This topic describes how to monitor Pivotal Container Service (PKS) workloads using sink resources.

Prerequisites

Using sink resources for monitoring requires that you have set up a log processing solution capable of log ingress over TCP as described in RFC 5424.

In addition, you must configure sinks in PKS to send logs to that destination. For information on how to create and manage sinks in PKS, see Creating Sink Resources.

Note: Sinks created in PKS only support TCP connections. UDP connections are not currently supported.

About Sink Log Entries

Sinks and ClusterSinks include both pod logs as well as events from the Kubernetes API.

These logs and events are combined in a shared format to provide operators with a robust set of filtering and monitoring options.

Sink data has the following characteristics. All entries:

  • Are timestamped.
  • Contain the host ID of the BOSH-defined VM.
  • Are annotated with a set of structured data, which includes the namespace, the object name or pod ID, and the container name.

Sink Log Entry Format

All sink log entries use the following format:

APP-NAME/NAMESPACE/POD-ID/CONTAINER-NAME

Where:

  • APP-NAME is pod.log or k8s.event.
  • NAMESPACE is the namespace associated with the pod log or Kubernetes event.
  • POD-ID is the ID of the pod associated with the pod log or Kubernetes event.
  • CONTAINER-NAME is the workload associated with the pod log or Kubernetes event.

Pod Logs

Pod logs entries are distinguished by the string pod.log in the APP-NAME field.

Pod Log Example

The following is a sample pod log entry:

36 <14>1 2018-11-26T18:51:41.647825+00:00 vm-3ebfe45d-492d-4bfd-59c4-c45d91688c65 
pod.log/rocky-raccoon/logspewer-6b58b6689d-dhddj - - [kubernetes@47450 
app="logspewer" pod-template-hash="2614622458" namespace_name="rocky-raccoon" 
object_name="logspewer-6b58b6689d-dhddj" container_name="logspewer"] 
2018/11/26 18:51:41 Log Message 589910

Where:

  • vm-3ebfe45d-492d-4bfd-59c4-c45d91688c65 is the host ID of the BOSH VM.
  • pod.log is the APP-NAME.
  • rocky-raccoon is the NAMESPACE.
  • logspewer-6b58b6689d-dhddj is the POD-ID.

Kubernetes API Events

Kubernetes API Event entries are distinguished by the string k8s.event in the APP-NAME field.

Kubernetes API Event Example

The following is an example Kubernetes API event log entry:

Nov 14 16:01:49 vm-b409c60e-2517-47ac-7c5b-2cd302287c3a 
k8s.event/rocky-raccoon/logspewer-6b58b6689d-j9n: 
Successfully assigned rocky-raccoon/logspewer-6b58b6689d-j9nq7 
to vm-38dfd896-bb21-43e4-67b0-9d2f339adaf1

Where:

  • vm-b409c60e-2517-47ac-7c5b-2cd302287c3a the host ID of the BOSH VM.
  • k8s.event is the APP-NAME.
  • rocky-raccoon is the NAMESPACE.
  • logspewer-6b58b6689d-j9n is the POD-ID.

Notable Kubernetes API Events

The following section lists Kubernetes API Events that can help assess any Kubernetes scheduling problems in PKS.

To monitor for these events, look for log entries that contain the Identifying String indicated below for each event.

Failure to Retrieve Containers from Registry


ImagePullBackOff

Description Image pull back offs occur when the Kubernetes API cannot reach a registry to retrieve a container or the container does not exist in the registry. The scheduler might be trying to access a registry that is not available on the network. For example, Docker Hub is blocked by a firewall. Other reasons might include the registry is experiencing an outage or a specified container has been deleted or was never uploaded.
Identifying String Error:ErrImagePull
Example Sink Log Entry Jan 25 10:18:58 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test-669d4d66b9-zd9h4/: Error: ErrImagePull

Malfunctioning Containers


CrashLoopBackOff

Description Crash loop back offs imply that the container is not functioning as intended. There are several potential causes of crash loop back offs which depend on the related workload. To investigate further, examine the logs for that workload.
Identifying String Back-off restarting failed container
Example Sink Log Entry Jan 25 09:26:44 vm-bfdfedef-4a6a-4c36-49fc-8b290ad42623 k8s.event/monitoring/cost-analyzer-prometheus-se: Back-off restarting failed container

Successful Scheduling of Containers


ContainerCreated

Description Operators can monitor the creation and successful start of containers to keep track of platform usage at a high level. Cluster users can track this event to monitor the usage of their cluster.
Identifying String Started container
Example Sink Log Entries Jan 25 09:14:55 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d/: Created pod: logspewer-6b58b6689d-sr96t
Jan 25 09:14:55 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Successfully assigned rocky-raccoon/ logspewer-6b58b6689d-sr96t to vm-efe48928-be8e-4db5-772c-426ee7aa52f2
Jan 25 09:14:55 vm-efe48928-be8e-4db5-772c-426ee7aa52f2 k8s.event/rocky-raccoon/logspewer-6b58b6689d-mkd: Killing container with id docker://logspewer:Need to kill Pod
Jan 25 09:14:56 vm-efe48928-be8e-4db5-772c-426ee7aa52f2 k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Container image "oratos/logspewer:v0.1" already present on machine
Jan 25 09:14:56 vm-efe48928-be8e-4db5-772c-426ee7aa52f2 k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Created container
Jan 25 09:14:56 vm-efe48928-be8e-4db5-772c-426ee7aa52f2 k8s.event/rocky-raccoon/logspewer-6b58b6689d-sr9: Started container

Failure to Schedule Containers


FailedScheduling

Description This event occurs when a container cannot be scheduled. For instance, this may occur due to lack of node resources..
Identifying String Insufficient RESOURCE
where RESOURCE is a specific type of resource. For example, cpu.
Example Sink Log Entries Jan 25 10:51:48 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test2-5c87bf4b65-7fdtd/: 0/1 nodes are available: 1 Insufficient cpu.

Related Links

For more information on sinks in PKS, see the following topics:

Please send any feedback you have to pks-feedback@pivotal.io.