Creating Sink Resources

This topic describes how to create a sink resource for a Kubernetes cluster provisioned with Enterprise Pivotal Container Service (Enterprise PKS) or for a namespace within a cluster.

Overview

Sink resources enable Enterprise PKS users to collect logs and metrics. You can create the following types of sink resources:

Sink Resource Sink Type Description
ClusterLogSink Log sink

Securely forwards logs from a cluster, including logs from all namespaces within the cluster, to a log destination. Logs are transported using one of the following:

  • The Syslog Protocol defined in RFC 5424
  • WebHook
LogSink Log sink Securely forwards logs from a namespace within a cluster to a log destination. Logs are transported following the Syslog Protocol defined in RFC 5424.
ClusterMetricSink Metric sink Collects and writes metrics from a cluster to specified outputs using input and output plugins.
MetricSink Metric sink Collects and writes metrics from a namespace within a cluster to specified outputs using input and output plugins.

Prerequisites

Before creating a sink resource, do the following:

  • If you want to create a ClusterLogSink or LogSink resource, ensure that the Enable Log Sink Resources checkbox is selected in the Enterprise PKS tile.
  • If you want to create a ClusterMetricSink or MetricSink resource, ensure that the Enable Metric Sink Resources checkbox is selected in the Enterprise PKS tile.

For more information, see the Enterprise PKS installation topic for your IaaS:

CLI Requirements

To create and manage sink resources, you must install the Kubernetes CLI, kubectl.

Alternately, to manage ClusterLogSink resources with the PKS CLI, you must use PKS CLI v1.3 or later.

Note: The pks create-sink, pks sinks, and pks delete-sink commands are deprecated in the PKS CLI. Enterprise PKS will remove these commands in a future release. You can use the Kubernetes CLI for creating and managing ClusterLogSink resources instead. For more information, see Deprecation of Sink Commands in the PKS CLI.

For installation instructions, see Installing the Kubernetes CLI and Installing the PKS CLI.

Create Sinks

You can create log and metric sinks for clusters and namespaces.

Note: Log sinks created in Enterprise PKS do not support UDP connections.

ClusterLogSink Resources

ClusterLogSink resources enable you to capture logs from your clusters.

To create and manage ClusterLogSink resources, you can:

Note: Enterprise PKS requires a secure connection for log forwarding. To forward logs using an unsecured connection, see Unsecured ClusterLogSink and LogSink Log Forwarding below.

Create ClusterLogSink Resource with the PKS CLI

To create and apply a log sink to a cluster, run the following command:

pks create-sink CLUSTER-NAME \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT

Where:

  • CLUSTER-NAME is the name of your cluster.
  • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

For example:

$ pks create-sink my-cluster syslog-tls://example.com:12345

If you do not specify a name, the command creates a log sink resource in the cluster that shares the same name as the cluster.

To provide a name for the log sink resources in your cluster, run the following command. 

pks create-sink CLUSTER-NAME --name YOUR-SINK \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT

Where:

  • CLUSTER-NAME is the name of your cluster.
  • YOUR-SINK is the name of the log sink you want to create.
  • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

For example:

$ pks create-sink my-cluster --name second-sink syslog-tls://example.org:54321

Specifying a name is useful if you need to manage multiple log sink resources in your cluster.

Create a Syslog ClusterLogSink Resource with YAML and kubectl

A syslog ClusterLogSink resource delivers logs using the TCP-based syslog protocol.

To define a syslog ClusterLogSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: pksapi.io/v1beta1
kind: ClusterLogSink
metadata:
   name: YOUR-SINK-NAME
spec:
   type: syslog
   host: YOUR-LOG-DESTINATION
   port: YOUR-LOG-DESTINATION-PORT
   enable_tls: true

    Where:

    • YOUR-SINK is a name you choose for your sink.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
    • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

      Note: enable_tls must be true.

  2. Save the YAML file with an appropriate file name. For example, my-cluster-sink.yml.

  3. Apply the ClusterLogSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml

    Where MY-SINK.yml is the name of your YAML file. For example: 

    $ kubectl apply -f my-cluster-sink.yml

Create a Webhook ClusterLogSink Resource with YAML and kubectl

A webhook ClusterLogSink resource batches logs into 1 second units, wraps the resulting payload in JSON, and uses the POST method to deliver the logs to the address of your log management service.

To define a webhook ClusterLogSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: pksapi.io/v1beta1
kind: ClusterLogSink
metadata:
  name: YOUR-SINK-NAME
spec:
  type: webhook
  url: YOUR-LOG-DESTINATION

    Where:

    • YOUR-SINK-NAME is a name you choose for your sink.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.

  2. Save the YAML file with an appropriate filename. For example, my-cluster-sink.yml.

  3. Apply the ClusterLogSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml

    Where MY-SINK.yml is the name of your YAML file. For example: 

    $ kubectl apply -f my-cluster-sink.yml

LogSink Resources

A LogSink resource filters logs by namespace within a cluster.

You can use only kubectl to create and manage log sinks for namespaces.

Create a LogSink Resource with YAML and kubectl

To define a LogSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: pksapi.io/v1beta1
kind: LogSink
metadata:
  name: YOUR-SINK
  namespace: YOUR-NAMESPACE
spec:
  type: syslog
  host: YOUR-LOG-DESTINATION
  port: YOUR-LOG-DESTINATION-PORT
  enable_tls: true

    Where:

    • YOUR-SINK is a name you choose for your log sink.
    • YOUR-NAMESPACE is the name of your namespace.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
    • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

      Note: enable_tls must be true.

  2. Save the YAML file with an appropriate file name. For example, my-namespace-sink.yml.

  3. Apply the LogSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml

    Where MY-SINK.yml is the name of your YAML file. For example: 

    $ kubectl apply -f my-namespace-sink.yml

Unsecured ClusterLogSink and LogSink Log Forwarding

By default, Enterprise PKS requires a secure connection for ClusterLogSink and LogSink log forwarding. To forward ClusterLogSink and LogSink logs using an unsecured connection, you must first disable log forwarding validation.

  1. To disable sink forwarding validation, run the following command:

    kubectl delete validatingwebhookconfigurations validator.pksapi.io

    Warning: Disabling secure log forwarding is not recommended.

ClusterMetricSink Resources

A ClusterMetricSink resource collects metrics from a cluster using the Kubernetes Input Plugin and writes them to one or more outputs that you specify in the configuration of your ClusterMetricSink. For a list of supported metrics and output plugins, see Metrics and Output Plugins in the telegraf GitHub repository.

You can use only kubectl to create and manage metric sinks for clusters.

Create a ClusterMetricSink Resource with YAML and kubectl

To define a ClusterMetricSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file in the following format:

    apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
  name: YOUR-SINK
spec:
  inputs:
  outputs:
  - type: YOUR-OUTPUT-PLUGIN

    Where:

    • YOUR-SINK is a name you choose for your sink.
    • YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

      Note: You can leave the inputs field blank. This field is configured to include metrics from the Kubelet by default.

    For example:

    apiVersion: pksapi.io/v1beta1
kind: ClusterMetricSink
metadata:
  name: http
spec:
  inputs:
  outputs:
  - type: http
    url: https:example.com
    method: POST
    data_format: json

  2. Save the YAML file with an appropriate filename. For example, my-cluster-metric-sink.yml.

  3. Apply the ClusterMetricSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml

    Where MY-SINK.yml is the name of your YAML file. For example: 

    $ kubectl apply -f my-cluster-metric-sink.yml

MetricSink Resources

A MetricSink resource collects metrics from a namespace within a cluster.

You can use only kubectl to create and manage metric sinks for namespaces.

Create a MetricSink Resource with YAML and kubectl

To define a MetricSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file in the following format:

    apiVersion: pksapi.io/v1beta1
kind: MetricSink
metadata:
  name: YOUR-SINK
  namespace: YOUR-NAMESPACE
spec:
  inputs:
  outputs:
  - type: YOUR-OUTPUT-PLUGIN

    Where:

    • YOUR-SINK is a name you choose for your sink.
    • YOUR-NAMESPACE is the name of your namespace.
    • YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

      Note: You can leave the inputs field blank. This field is configured to include all prometheus.io/scrape annotations set to true by default.

    For example:

    apiVersion: pksapi.io/v1beta1
kind: MetricSink
metadata:
  name: http
spec:
  inputs:
  outputs:
  - type: http
    url: https:example.com
    method: POST
    data_format: json

  2. Save the YAML file with an appropriate filename. For example, my-metric-sink.yml.

  3. Apply the MetricSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml

    Where MY-SINK.yml is the name of your YAML file. For example: 

    $ kubectl apply -f my-metric-sink.yml

List Sinks

To list sinks for clusters and namespaces, use the commands in the following sections.

ClusterLogSink Resources

To list cluster log sinks, run the following command:

pks sinks CLUSTER-NAME

Where CLUSTER-NAME is the name of your cluster.

Alternately, you can use kubectl:

kubectl get clusterlogsinks

LogSink Resources

To list namespace log sinks, run the following command:

kubectl get logsinks -n YOUR-NAMESPACE

Where YOUR-NAMESPACE is the name of your namespace.

ClusterMetricSink Resources

To list cluster metric sinks, run the following command:

kubectl get clustermetricsinks

MetricSink Resources

To list namespace metric sinks, run the following command:

kubectl get metricsinks -n YOUR-NAMESPACE

Where YOUR-NAMESPACE is the name of your namespace.

Delete Sinks

To delete sinks for clusters and namespaces, use the commands in the following sections.

ClusterLogSink Resources

To delete a cluster log sink, run the following PKS CLI command:

pks delete-sink CLUSTER-NAME --name YOUR-SINK

Where:

  • CLUSTER-NAME is the name of your cluster.
  • YOUR-SINK is the name of your sink.

Alternately, you can use kubectl:

kubectl delete clusterlogsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

LogSink Resources

To delete a namespace log sink using kubectl, run the following command:

kubectl delete logsink YOUR-SINK -n YOUR-NAMESPACE

Where:

  • YOUR-SINK is the name of your log sink.
  • YOUR-NAMESPACE is the name of your namespace.

ClusterMetricSink Resources

To delete a cluster metric sink, use the following command:

kubectl delete clustermetricsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

MetricSink Resources

To delete a namespace metric sink, use the following command:

kubectl delete metricsink YOUR-SINK -n YOUR-NAMESPACE

Where:

  • YOUR-SINK is the name of your metric sink.
  • YOUR-NAMESPACE is the name of your namespace.

Related Links

For more information about sinks in Enterprise PKS, see the following topics:

Please send any feedback you have to pks-feedback@pivotal.io.