Monitoring PKS with Sinks
Page last updated:
Warning: Pivotal Container Service (PKS)
v1.3 is no longer supported because it has reached the End
of General Support (EOGS) phase as defined by the
Support Lifecycle Policy.
To stay up to date with the latest software and security updates, upgrade to a supported version.
This topic describes how to monitor Pivotal Container Service (PKS) workloads using sink resources.
Prerequisites
Using sink resources for monitoring requires that you have set up a log processing solution capable of log ingress over TCP as described in RFC 5424.
In addition, you must configure sinks in PKS to send logs to that destination. For information on how to create and manage sinks in PKS, see Creating Sink Resources.
Note: Sinks created in PKS only support TCP connections. UDP connections are not currently supported.
About Sink Log Entries
Sinks and ClusterSinks include both pod logs as well as events from the Kubernetes API.
These logs and events are combined in a shared format to provide operators with a robust set of filtering and monitoring options.
Sink data has the following characteristics. All entries:
- Are timestamped.
- Contain the host ID of the BOSH-defined VM.
- Are annotated with a set of structured data, which includes the namespace, the object name or pod ID, and the container name.
Sink Log Entry Format
All sink log entries use the following format:
APP-NAME/NAMESPACE/POD-ID/CONTAINER-NAME
Where:
APP-NAME
ispod.log
ork8s.event
.NAMESPACE
is the namespace associated with the pod log or Kubernetes event.POD-ID
is the ID of the pod associated with the pod log or Kubernetes event.CONTAINER-NAME
is the workload associated with the pod log or Kubernetes event.
Pod Logs
Pod logs entries are distinguished by the string pod.log
in the APP-NAME
field.
Pod Log Example
The following is a sample pod log entry:
36 <14>1 2018-11-26T18:51:41.647825+00:00 vm-3ebfe45d-492d-4bfd-59c4-c45d91688c65 pod.log/rocky-raccoon/logspewer-6b58b6689d-dhddj - - [kubernetes@47450 app="logspewer" pod-template-hash="2614622458" namespace_name="rocky-raccoon" object_name="logspewer-6b58b6689d-dhddj" container_name="logspewer"] 2018/11/26 18:51:41 Log Message 589910
Where:
vm-3ebfe45d-492d-4bfd-59c4-c45d91688c65
is the host ID of the BOSH VM.pod.log
is theAPP-NAME
.rocky-raccoon
is theNAMESPACE
.logspewer-6b58b6689d-dhddj
is thePOD-ID
.
Kubernetes API Events
Kubernetes API Event entries are distinguished by the string k8s.event
in the APP-NAME
field.
Kubernetes API Event Example
The following is an example Kubernetes API event log entry:
Nov 14 16:01:49 vm-b409c60e-2517-47ac-7c5b-2cd302287c3a k8s.event/rocky-raccoon/logspewer-6b58b6689d-j9n: Successfully assigned rocky-raccoon/logspewer-6b58b6689d-j9nq7 to vm-38dfd896-bb21-43e4-67b0-9d2f339adaf1
Where:
vm-b409c60e-2517-47ac-7c5b-2cd302287c3a
the host ID of the BOSH VM.k8s.event
is theAPP-NAME
.rocky-raccoon
is theNAMESPACE
.logspewer-6b58b6689d-j9n
is thePOD-ID
.
Notable Kubernetes API Events
The following section lists Kubernetes API Events that can help assess any Kubernetes scheduling problems in PKS.
To monitor for these events, look for log entries that contain the Identifying String indicated below for each event.
Failure to Retrieve Containers from Registry
ImagePullBackOff | |
---|---|
Description | Image pull back offs occur when the Kubernetes API cannot reach a registry to retrieve a container or the container does not exist in the registry. The scheduler might be trying to access a registry that is not available on the network. For example, Docker Hub is blocked by a firewall. Other reasons might include the registry is experiencing an outage or a specified container has been deleted or was never uploaded. |
Identifying String | Error:ErrImagePull |
Example Sink Log Entry |
Jan 25 10:18:58 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test-669d4d66b9-zd9h4/: Error: ErrImagePull
|
Malfunctioning Containers
CrashLoopBackOff | |
---|---|
Description | Crash loop back offs imply that the container is not functioning as intended. There are several potential causes of crash loop back offs which depend on the related workload. To investigate further, examine the logs for that workload. |
Identifying String | Back-off restarting failed container |
Example Sink Log Entry |
Jan 25 09:26:44 vm-bfdfedef-4a6a-4c36-49fc-8b290ad42623 k8s.event/monitoring/cost-analyzer-prometheus-se: Back-off restarting failed container
|
Successful Scheduling of Containers
ContainerCreated | |
---|---|
Description | Operators can monitor the creation and successful start of containers to keep track of platform usage at a high level. Cluster users can track this event to monitor the usage of their cluster. |
Identifying String | Started container |
Example Sink Log Entries |
Jan 25 09:14:55 35.239.18.250 k8s.event/rocky-raccoon/logspewer-6b58b6689d/: Created pod: logspewer-6b58b6689d-sr96t
|
Failure to Schedule Containers
FailedScheduling | |
---|---|
Description | This event occurs when a container cannot be scheduled. For instance, this may occur due to lack of node resources.. |
Identifying String | Insufficient RESOURCE where RESOURCE is a specific type of resource. For example, cpu . |
Example Sink Log Entries |
Jan 25 10:51:48 gke-bf-test-default-pool-aa8027bc-rnf6 k8s.event/default/test2-5c87bf4b65-7fdtd/: 0/1 nodes are available: 1 Insufficient cpu.
|
Related Links
For more information on sinks in PKS, see the following topics:
- For information about creating sinks in PKS, see Creating Sink Resources.
- For information about sink architecture, see Sink Architecture in PKS.
Please send any feedback you have to pks-feedback@pivotal.io.