Creating Sink Resources

Page last updated:

This topic describes how to create a sink resource for a Kubernetes cluster provisioned with Enterprise Pivotal Container Service (Enterprise PKS) or for a namespace within a cluster.

Overview

Sink resources enable Enterprise PKS users to collect logs and metrics. You can create the following types of sink resources:

Prerequisites

Before creating a sink resource, do the following:

• If you want to create a ClusterLogSink or LogSink resource, ensure that the Enable Log Sink Resources checkbox is selected in the Enterprise PKS tile.
• If you want to create a ClusterMetricSink or MetricSink resource, ensure that the Enable Metric Sink Resources checkbox is selected in the Enterprise PKS tile.

For more information, see the Enterprise PKS installation topic for your IaaS:

• Installing Enterprise PKS on vSphere
• Installing Enterprise PKS on vSphere with NSX-T Integration
• Installing Enterprise PKS on GCP
• Installing Enterprise PKS on AWS
• Installing Enterprise PKS on Azure

CLI Requirements

To create and manage sink resources, you must install the Kubernetes CLI, kubectl.

Alternately, to manage ClusterLogSink resources with the PKS CLI, you must use PKS CLI v1.3 or later.

Note: The pks create-sink, pks sinks, and pks delete-sink commands are deprecated in the PKS CLI. Enterprise PKS will remove these commands in a future release. You can use the Kubernetes CLI for creating and managing ClusterLogSink resources instead. For more information, see Deprecation of Sink Commands in the PKS CLI.

For installation instructions, see Installing the Kubernetes CLI and Installing the PKS CLI.

Create Sinks

You can create log and metric sinks for clusters and namespaces.

• If you want to create a ClusterLogSink, see ClusterLogSink Resources below.
• If you want to create a LogSink, see LogSink Resources below.
• If you want to create a ClusterMetricSink, see ClusterMetricSink Resources below.
• If you want to create a MetricSink, see MetricSink Resources below.

Note: Log sinks created in Enterprise PKS do not support UDP connections.

ClusterLogSink Resources

ClusterLogSink resources enable you to capture logs from your clusters.

To create and manage ClusterLogSink resources, you can:

• Create ClusterLogSink Resource with the PKS CLI
• Create a Syslog ClusterLogSink Resource with YAML and kubectl
• Create a Webhook ClusterLogSink Resource with YAML and kubectl

Note: Enterprise PKS requires a secure connection for log forwarding. To forward logs using an unsecured connection, see Unsecured ClusterLogSink and LogSink Log Forwarding below.

Create ClusterLogSink Resource with the PKS CLI

To create and apply a log sink to a cluster, run the following command:

pks create-sink CLUSTER-NAME \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT

Where:

• CLUSTER-NAME is the name of your cluster.
• YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
• YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

For example:

$ pks create-sink my-cluster syslog-tls://example.com:12345

If you do not specify a name, the command creates a log sink resource in the cluster that shares the same name as the cluster.

To provide a name for the log sink resources in your cluster, run the following command.

pks create-sink CLUSTER-NAME --name YOUR-SINK \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT

Where:

• CLUSTER-NAME is the name of your cluster.
• YOUR-SINK is the name of the log sink you want to create.
• YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
• YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

For example:

$ pks create-sink my-cluster --name second-sink syslog-tls://example.org:54321

Specifying a name is useful if you need to manage multiple log sink resources in your cluster.

Create a Syslog ClusterLogSink Resource with YAML and kubectl

A syslog ClusterLogSink resource delivers logs using the TCP-based syslog protocol.

To define a syslog ClusterLogSink resource with YAML and kubectl, perform the following steps:

1. Create a YAML file that specifies your log destination in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: ClusterLogSink
   metadata:
      name: YOUR-SINK-NAME
   spec:
      type: syslog
      host: YOUR-LOG-DESTINATION
      port: YOUR-LOG-DESTINATION-PORT
      enable_tls: true
   

   Where:

   • YOUR-SINK is a name you choose for your sink.
   • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
   • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

     Note: enable_tls must be true.

2. Save the YAML file with an appropriate file name. For example, my-cluster-sink.yml.

3. Apply the ClusterLogSink resource to your cluster by running the following command:

   kubectl apply -f MY-SINK.yml
   

   Where MY-SINK.yml is the name of your YAML file. For example:

   $ kubectl apply -f my-cluster-sink.yml
   

Create a Webhook ClusterLogSink Resource with YAML and kubectl

A webhook ClusterLogSink resource batches logs into 1 second units, wraps the resulting payload in JSON, and uses the POST method to deliver the logs to the address of your log management service.

To define a webhook ClusterLogSink resource with YAML and kubectl, perform the following steps:

1. Create a YAML file that specifies your log destination in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: ClusterLogSink
   metadata:
     name: YOUR-SINK-NAME
   spec:
     type: webhook
     url: YOUR-LOG-DESTINATION
   

   Where:

   • YOUR-SINK-NAME is a name you choose for your sink.
   • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.

2. Save the YAML file with an appropriate filename. For example, my-cluster-sink.yml.

3. Apply the ClusterLogSink resource to your cluster by running the following command:

   kubectl apply -f MY-SINK.yml
   

   Where MY-SINK.yml is the name of your YAML file. For example:

   $ kubectl apply -f my-cluster-sink.yml
   

<<<<<<< HEAD

LogSink Resources

A LogSink resource filters logs by namespace within a cluster.

You can use only kubectl to create and manage log sinks for namespaces. For instructions, see the following:

• Create a Syslog LogSink Resource with YAML and kubectl
• Create a Webhook LogSink Resource with YAML and kubectl

Create a Syslog LogSink Resource with YAML and kubectl

To define a syslog LogSink resource with YAML and kubectl, perform the following steps:

1. Create a YAML file that specifies your log destination in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: LogSink
   metadata:
     name: YOUR-SINK
     namespace: YOUR-NAMESPACE
   spec:
     type: syslog
     host: YOUR-LOG-DESTINATION
     port: YOUR-LOG-DESTINATION-PORT
     enable_tls: true
   

   Where:

   • YOUR-SINK is a name you choose for your log sink.
   • YOUR-NAMESPACE is the name of your namespace.
   • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
   • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

     Note: enable_tls must be true.

2. Save the YAML file with an appropriate file name. For example, my-namespace-sink.yml.

3. Apply the LogSink resource to your cluster by running the following command:

   kubectl apply -f MY-SINK.yml
   

   Where MY-SINK.yml is the name of your YAML file. For example:

   $ kubectl apply -f my-namespace-sink.yml
   

Create a Webhook LogSink Resource with YAML and kubectl

To define a webhook LogSink resource with YAML and kubectl, perform the following steps:

1. Create a YAML file that specifies your log destination in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: LogSink
   metadata:
     name: YOUR-SINK-NAME
     namespace: YOUR-NAMESPACE
   spec:
     type: webhook
     url: YOUR-LOG-DESTINATION
   

   Where:

   • YOUR-SINK-NAME is a name you choose for your sink.
   • YOUR-NAMESPACE is the name of your namespace.
   • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.

2. Save the YAML file with an appropriate filename. For example, my-namespace-sink.yml.

3. Apply the LogSink resource to your cluster by running the following command:

   kubectl apply -f MY-SINK.yml
   

   Where MY-SINK.yml is the name of your YAML file. For example:

   $ kubectl apply -f my-namespace-sink.yml
   

Unsecured ClusterLogSink and LogSink Log Forwarding

By default, Enterprise PKS requires a secure connection for log forwarding when using ClusterLogSink and LogSink resources of type syslog or webhook.

For debugging purposes on a local machine, you may want to temporarily forward logs using an unsecured connection. To do this, you must:

1. Disable sink forwarding validation by running the following command:

   kubectl delete validatingwebhookconfigurations validator.pksapi.io
   

2. Set enable_tls to false in your log destination YAML file.

Warning: Disabling secure log forwarding is not recommended.

ClusterMetricSink and MetricSink Resources

A ClusterMetricSink resource collects metrics from a cluster using the Kubernetes Input Plugin and writes them to one or more outputs that you specify in the configuration of your ClusterMetricSink. For a list of supported metrics and output plugins, see Metrics and Output Plugins in the telegraf GitHub repository.

You can use only kubectl to create and manage metric sinks for clusters.

Create a ClusterMetricSink Resource with YAML and kubectl

To define a ClusterMetricSink resource with YAML and kubectl, perform the following steps:

1. Create a YAML file in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: ClusterMetricSink
   metadata:
     name: YOUR-SINK
   spec:
     inputs:
     outputs:
     - type: YOUR-OUTPUT-PLUGIN
   

   Where:

   • YOUR-SINK is a name you choose for your sink.
   • YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

     Note: You can leave the inputs field blank. This field is configured to include metrics from the Kubelet by default.

   For example:

   apiVersion: pksapi.io/v1beta1
   kind: ClusterMetricSink
   metadata:
     name: http
   spec:
     inputs:
     outputs:
     - type: http
       url: https:example.com
       method: POST
       data_format: json
   

2. Save the YAML file with an appropriate filename. For example, my-cluster-metric-sink.yml.

3. Apply the ClusterMetricSink resource to your cluster by running the following command:

   kubectl apply -f MY-SINK.yml
   

   Where MY-SINK.yml is the name of your YAML file. For example:

   $ kubectl apply -f my-cluster-metric-sink.yml
   

Create a ClusterMetricSink Resource to Export Node Exporter Metrics

To define a ClusterMetricSink resource to export Node Exporter metrics, perform the following steps:

1. Enable Node Exporter on cluster workers by selecting the Enable node exporter on workers checkbox in the In-Cluster Monitoring section in PKS.

2. Create a YAML file in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: ClusterMetricSink
   metadata:
     name: YOUR-SINK
   spec:
     inputs:
     - monitor_kubernetes_pods: true
       type: prometheus
     outputs:
     - type: YOUR-OUTPUT-PLUGIN
   

   Where:

   • YOUR-SINK is a name you choose for your sink.
   • YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

     Note: You can leave the inputs field blank. This field is configured to include metrics from the kubelet by default.

   For example:

   apiVersion: pksapi.io/v1beta1
   kind: ClusterMetricSink
   metadata:
     name: http
   spec:
     inputs:
     - monitor_kubernetes_pods: true
       type: prometheus
     outputs:
     - type: http
       url: https:example.com
       method: POST
       data_format: json
   

MetricSink Resources

A MetricSink resource collects metrics from a namespace within a cluster.

You can use only kubectl to create and manage metric sinks for namespaces.

Create a MetricSink Resource with YAML and kubectl

To define a MetricSink resource with YAML and kubectl, perform the following steps:

1. Create a YAML file in the following format:

   apiVersion: pksapi.io/v1beta1
   kind: MetricSink
   metadata:
     name: YOUR-SINK
     namespace: YOUR-NAMESPACE
   spec:
     inputs:
     outputs:
     - type: YOUR-OUTPUT-PLUGIN
   

   Where:

   • YOUR-SINK is a name you choose for your sink.
   • YOUR-NAMESPACE is the name of your namespace.
   • YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

     Note: You can leave the inputs field blank. This field is configured to include all prometheus.io/scrape annotations set to true by default.

   For example:

   apiVersion: pksapi.io/v1beta1
   kind: MetricSink
   metadata:
     name: http
   spec:
     inputs:
     outputs:
     - type: http
       url: https:example.com
       method: POST
       data_format: json
   

2. Save the YAML file with an appropriate filename. For example, my-metric-sink.yml.

3. Apply the MetricSink resource to your cluster by running the following command:

   kubectl apply -f MY-SINK.yml
   

   Where MY-SINK.yml is the name of your YAML file. For example:

   $ kubectl apply -f my-metric-sink.yml
   

List Sinks

To list sinks for clusters and namespaces, use the commands in the following sections.

ClusterLogSink Resources

To list cluster log sinks, run the following command:

pks sinks CLUSTER-NAME

Where CLUSTER-NAME is the name of your cluster.

Alternately, you can use kubectl:

kubectl get clusterlogsinks

LogSink Resources

To list namespace log sinks, run the following command:

kubectl get logsinks -n YOUR-NAMESPACE

Where YOUR-NAMESPACE is the name of your namespace.

ClusterMetricSink Resources

To list cluster metric sinks, run the following command:

kubectl get clustermetricsinks

MetricSink Resources

To list namespace metric sinks, run the following command:

kubectl get metricsinks -n YOUR-NAMESPACE

Where YOUR-NAMESPACE is the name of your namespace.

Delete Sinks

To delete sinks for clusters and namespaces, use the commands in the following sections.

ClusterLogSink Resources

To delete a cluster log sink, run the following PKS CLI command:

pks delete-sink CLUSTER-NAME --name YOUR-SINK

Where:

• CLUSTER-NAME is the name of your cluster.
• YOUR-SINK is the name of your sink.

Alternately, you can use kubectl:

kubectl delete clusterlogsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

LogSink Resources

To delete a namespace log sink using kubectl, run the following command:

kubectl delete logsink YOUR-SINK -n YOUR-NAMESPACE

Where:

• YOUR-SINK is the name of your log sink.
• YOUR-NAMESPACE is the name of your namespace.

ClusterMetricSink Resources

To delete a cluster metric sink, use the following command:

kubectl delete clustermetricsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

MetricSink Resources

To delete a namespace metric sink, use the following command:

kubectl delete metricsink YOUR-SINK -n YOUR-NAMESPACE

Where:

• YOUR-SINK is the name of your metric sink.
• YOUR-NAMESPACE is the name of your namespace.

Related Links

For more information about sinks in Enterprise PKS, see the following topics:

• For information about using sinks for monitoring, see Monitoring Clusters with Log Sinks.
• For information about sink architecture, see Sink Architecture in Enterprise PKS.