Creating Sink Resources
Page last updated:
This topic describes how to create a sink resource for a Kubernetes cluster provisioned with Enterprise Pivotal Container Service (Enterprise PKS) or for a namespace within a cluster.
Overview
Sink resources enable Enterprise PKS users to collect logs and metrics. You can create the following types of sink resources:
Sink Resource | Sink Type | Description |
---|---|---|
ClusterLogSink |
Log sink | Forwards logs from a cluster to a log destination. Logs are transported using one of the following:
|
LogSink |
Log sink | Forwards logs from a namespace within a cluster to a log destination. Logs are transported using one of the following:
|
ClusterMetricSink |
Metric sink | Collects and writes metrics from a cluster to specified outputs using input and output plugins. |
MetricSink |
Metric sink | Collects and writes metrics from a namespace within a cluster to specified outputs using input and output plugins. |
Prerequisites
Before creating a sink resource, do the following:
- If you want to create a
ClusterLogSink
orLogSink
resource, ensure that the Enable Log Sink Resources checkbox is selected in the Enterprise PKS tile. - If you want to create a
ClusterMetricSink
orMetricSink
resource, ensure that the Enable Metric Sink Resources checkbox is selected in the Enterprise PKS tile.
For more information, see the Enterprise PKS installation topic for your IaaS:
- Installing Enterprise PKS on vSphere
- Installing Enterprise PKS on vSphere with NSX-T Integration
- Installing Enterprise PKS on GCP
- Installing Enterprise PKS on AWS
- Installing Enterprise PKS on Azure
CLI Requirements
To create and manage sink resources, you must install the Kubernetes CLI, kubectl
.
Alternately, to manage ClusterLogSink
resources with the PKS CLI, you must use PKS CLI v1.3 or later.
Note: The pks create-sink
, pks sinks
, and pks delete-sink
commands
are deprecated in the PKS CLI. Enterprise PKS will remove these commands in a future release. You can use the Kubernetes CLI for creating and managing ClusterLogSink
resources instead.
For more information, see Deprecation of Sink Commands in the PKS CLI.
For installation instructions, see Installing the Kubernetes CLI and Installing the PKS CLI.
Create Sinks
You can create log and metric sinks for clusters and namespaces.
- If you want to create a
ClusterLogSink
, see ClusterLogSink Resources below. - If you want to create a
LogSink
, see LogSink Resources below. - If you want to create a
ClusterMetricSink
, see ClusterMetricSink Resources below. - If you want to create a
MetricSink
, see MetricSink Resources below.
Note: Log sinks created in Enterprise PKS do not support UDP connections.
ClusterLogSink Resources
ClusterLogSink
resources enable you to capture logs from your clusters.
To create and manage ClusterLogSink
resources, you can:
- Create ClusterLogSink Resource with the PKS CLI
- Create a Syslog ClusterLogSink Resource with YAML and kubectl
- Create a Webhook ClusterLogSink Resource with YAML and kubectl
Note: Enterprise PKS requires a secure connection for log forwarding. To forward logs using an unsecured connection, see Unsecured ClusterLogSink and LogSink Log Forwarding below.
Create ClusterLogSink Resource with the PKS CLI
To create and apply a log sink to a cluster, run the following command:
pks create-sink CLUSTER-NAME \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT
Where:
CLUSTER-NAME
is the name of your cluster.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.
For example:
$ pks create-sink my-cluster syslog-tls://example.com:12345
If you do not specify a name, the command creates a log sink resource in the cluster that shares the same name as the cluster.
To provide a name for the log sink resources in your cluster, run the following command.
pks create-sink CLUSTER-NAME --name YOUR-SINK \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT
Where:
CLUSTER-NAME
is the name of your cluster.YOUR-SINK
is the name of the log sink you want to create.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.
For example:
$ pks create-sink my-cluster --name second-sink syslog-tls://example.org:54321
Specifying a name is useful if you need to manage multiple log sink resources in your cluster.
Create a Syslog ClusterLogSink Resource with YAML and kubectl
A syslog ClusterLogSink
resource delivers logs using the TCP-based syslog protocol.
To define a syslog ClusterLogSink
resource with YAML and kubectl
, perform the following steps:
Create a YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1 kind: ClusterLogSink metadata: name: YOUR-SINK-NAME spec: type: syslog host: YOUR-LOG-DESTINATION port: YOUR-LOG-DESTINATION-PORT enable_tls: true
Where:
YOUR-SINK
is a name you choose for your sink.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.Note:
enable_tls
must betrue
.
Save the YAML file with an appropriate file name. For example,
my-cluster-sink.yml
.Apply the
ClusterLogSink
resource to your cluster by running the following command:kubectl apply -f MY-SINK.yml
Where
MY-SINK.yml
is the name of your YAML file. For example:$ kubectl apply -f my-cluster-sink.yml
Create a Webhook ClusterLogSink Resource with YAML and kubectl
A webhook ClusterLogSink
resource batches logs into 1 second units, wraps the resulting payload in JSON, and uses the POST method to deliver the logs to the address of your log management service.
To define a webhook ClusterLogSink
resource with YAML and kubectl
, perform the following steps:
Create a YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1 kind: ClusterLogSink metadata: name: YOUR-SINK-NAME spec: type: webhook url: YOUR-LOG-DESTINATION
Where:
YOUR-SINK-NAME
is a name you choose for your sink.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.
Save the YAML file with an appropriate filename. For example,
my-cluster-sink.yml
.Apply the
ClusterLogSink
resource to your cluster by running the following command:kubectl apply -f MY-SINK.yml
Where
MY-SINK.yml
is the name of your YAML file. For example:$ kubectl apply -f my-cluster-sink.yml
LogSink Resources
A LogSink
resource filters logs by namespace within a cluster.
You can use only kubectl
to create and manage log sinks for namespaces.
For instructions, see the following:
- Create a Syslog LogSink Resource with YAML and kubectl
- Create a Webhook LogSink Resource with YAML and kubectl
Create a Syslog LogSink Resource with YAML and kubectl
To define a syslog LogSink
resource with YAML and kubectl
, perform the following steps:
Create a YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1 kind: LogSink metadata: name: YOUR-SINK namespace: YOUR-NAMESPACE spec: type: syslog host: YOUR-LOG-DESTINATION port: YOUR-LOG-DESTINATION-PORT enable_tls: true
Where:
YOUR-SINK
is a name you choose for your log sink.YOUR-NAMESPACE
is the name of your namespace.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.YOUR-LOG-DESTINATION-PORT
is the port number of your log management service.Note:
enable_tls
must betrue
.
Save the YAML file with an appropriate file name. For example,
my-namespace-sink.yml
.Apply the
LogSink
resource to your cluster by running the following command:kubectl apply -f MY-SINK.yml
Where
MY-SINK.yml
is the name of your YAML file. For example:$ kubectl apply -f my-namespace-sink.yml
Create a Webhook LogSink Resource with YAML and kubectl
To define a webhook LogSink
resource with YAML and kubectl
, perform the following steps:
Create a YAML file that specifies your log destination in the following format:
apiVersion: pksapi.io/v1beta1 kind: LogSink metadata: name: YOUR-SINK-NAME namespace: YOUR-NAMESPACE spec: type: webhook url: YOUR-LOG-DESTINATION
Where:
YOUR-SINK-NAME
is a name you choose for your sink.YOUR-NAMESPACE
is the name of your namespace.YOUR-LOG-DESTINATION
is the URL or IP address of your log management service.
Save the YAML file with an appropriate filename. For example,
my-namespace-sink.yml
.Apply the
LogSink
resource to your cluster by running the following command:kubectl apply -f MY-SINK.yml
Where
MY-SINK.yml
is the name of your YAML file. For example:$ kubectl apply -f my-namespace-sink.yml
Unsecured ClusterLogSink and LogSink Log Forwarding
By default, Enterprise PKS requires a secure connection for ClusterLogSink
and LogSink
log forwarding.
To forward ClusterLogSink
and LogSink
logs using an unsecured connection, you must first disable log forwarding validation.
To disable sink forwarding validation, run the following command:
kubectl delete validatingwebhookconfigurations validator.pksapi.io
Warning: Disabling secure log forwarding is not recommended.
ClusterMetricSink Resources
A ClusterMetricSink
resource collects metrics from a cluster using the Kubernetes Input Plugin and writes them to one or more outputs
that you specify in the configuration of your ClusterMetricSink
.
For a list of supported metrics and output plugins,
see Metrics
and Output Plugins in the telegraf GitHub repository.
You can use only kubectl
to create and manage metric sinks for clusters.
Create a ClusterMetricSink Resource with YAML and kubectl
To define a ClusterMetricSink
resource with YAML and kubectl
, perform the following steps:
Create a YAML file in the following format:
apiVersion: pksapi.io/v1beta1 kind: ClusterMetricSink metadata: name: YOUR-SINK spec: inputs: outputs: - type: YOUR-OUTPUT-PLUGIN
Where:
YOUR-SINK
is a name you choose for your sink.YOUR-OUTPUT-PLUGIN
is the name of the output plugin you want to use for your metrics.Note: You can leave the
inputs
field blank. This field is configured to include metrics from the Kubelet by default.
For example:
apiVersion: pksapi.io/v1beta1 kind: ClusterMetricSink metadata: name: http spec: inputs: outputs: - type: http url: https:example.com method: POST data_format: json
Save the YAML file with an appropriate filename. For example,
my-cluster-metric-sink.yml
.Apply the
ClusterMetricSink
resource to your cluster by running the following command:kubectl apply -f MY-SINK.yml
Where
MY-SINK.yml
is the name of your YAML file. For example:$ kubectl apply -f my-cluster-metric-sink.yml
Create a ClusterMetricSink Resource to Export Node Exporter Metrics
To define a ClusterMetricSink
resource to export Node Exporter metrics, perform the following steps:
- Enable Node Exporter on cluster workers by selecting the
Enable node exporter on workers
checkbox in theIn-Cluster Monitoring
section in PKS. Create a YAML file in the following format:
apiVersion: pksapi.io/v1beta1 kind: ClusterMetricSink metadata: name: YOUR-SINK spec: inputs: - monitor_kubernetes_pods: true type: prometheus outputs: - type: YOUR-OUTPUT-PLUGIN
Where:
YOUR-SINK
is a name you choose for your sink.YOUR-OUTPUT-PLUGIN
is the name of the output plugin you want to use for your metrics.Note: You can leave the
inputs
field blank. This field is configured to include metrics from the kubelet by default.
For example:
apiVersion: pksapi.io/v1beta1 kind: ClusterMetricSink metadata: name: http spec: inputs: - monitor_kubernetes_pods: true type: prometheus outputs: - type: http url: https:example.com method: POST data_format: json
MetricSink Resources
A MetricSink
resource collects metrics from a namespace within a cluster.
You can use only kubectl
to create and manage metric sinks for namespaces.
Create a MetricSink Resource with YAML and kubectl
To define a MetricSink
resource with YAML and kubectl
, perform the following steps:
Create a YAML file in the following format:
apiVersion: pksapi.io/v1beta1 kind: MetricSink metadata: name: YOUR-SINK namespace: YOUR-NAMESPACE spec: inputs: outputs: - type: YOUR-OUTPUT-PLUGIN
Where:
YOUR-SINK
is a name you choose for your sink.YOUR-NAMESPACE
is the name of your namespace.YOUR-OUTPUT-PLUGIN
is the name of the output plugin you want to use for your metrics.Note: You can leave the
inputs
field blank. This field is configured to include allprometheus.io/scrape
annotations set totrue
by default.
For example:
apiVersion: pksapi.io/v1beta1 kind: MetricSink metadata: name: http spec: inputs: outputs: - type: http url: https:example.com method: POST data_format: json
Save the YAML file with an appropriate filename. For example,
my-metric-sink.yml
.Apply the
MetricSink
resource to your cluster by running the following command:kubectl apply -f MY-SINK.yml
Where
MY-SINK.yml
is the name of your YAML file. For example:$ kubectl apply -f my-metric-sink.yml
List Sinks
To list sinks for clusters and namespaces, use the commands in the following sections.
ClusterLogSink Resources
To list cluster log sinks, run the following command:
pks sinks CLUSTER-NAME
Where CLUSTER-NAME
is the name of your cluster.
Alternately, you can use kubectl
:
kubectl get clusterlogsinks
LogSink Resources
To list namespace log sinks, run the following command:
kubectl get logsinks -n YOUR-NAMESPACE
Where YOUR-NAMESPACE
is the name of your namespace.
ClusterMetricSink Resources
To list cluster metric sinks, run the following command:
kubectl get clustermetricsinks
MetricSink Resources
To list namespace metric sinks, run the following command:
kubectl get metricsinks -n YOUR-NAMESPACE
Where YOUR-NAMESPACE
is the name of your namespace.
Delete Sinks
To delete sinks for clusters and namespaces, use the commands in the following sections.
ClusterLogSink Resources
To delete a cluster log sink, run the following PKS CLI command:
pks delete-sink CLUSTER-NAME --name YOUR-SINK
Where:
CLUSTER-NAME
is the name of your cluster.YOUR-SINK
is the name of your sink.
Alternately, you can use kubectl
:
kubectl delete clusterlogsink YOUR-SINK
Where YOUR-SINK
is the name of your sink.
LogSink Resources
To delete a namespace log sink using kubectl
, run the following command:
kubectl delete logsink YOUR-SINK -n YOUR-NAMESPACE
Where:
YOUR-SINK
is the name of your log sink.YOUR-NAMESPACE
is the name of your namespace.
ClusterMetricSink Resources
To delete a cluster metric sink, use the following command:
kubectl delete clustermetricsink YOUR-SINK
Where YOUR-SINK
is the name of your sink.
MetricSink Resources
To delete a namespace metric sink, use the following command:
kubectl delete metricsink YOUR-SINK -n YOUR-NAMESPACE
Where:
YOUR-SINK
is the name of your metric sink.YOUR-NAMESPACE
is the name of your namespace.
Related Links
For more information about sinks in Enterprise PKS, see the following topics:
- For information about using sinks for monitoring, see Monitoring Enterprise PKS with Log Sinks.
- For information about sink architecture, see Sink Architecture in Enterprise PKS.
Please send any feedback you have to pks-feedback@pivotal.io.