Creating Sink Resources

Page last updated:

This topic describes how to create a sink resource for a Kubernetes cluster provisioned with Enterprise Pivotal Container Service (Enterprise PKS) or for a namespace within a cluster.

Overview

Sink resources enable Enterprise PKS users to collect logs and metrics. You can create the following types of sink resources:

Sink Resource Sink Type Description
ClusterSink Log sink

Sends logs from a cluster to a log destination. Logs are transported using one of the following:

  • The Syslog Protocol defined in RFC 5424
  • WebHook
Sink Log sink

Sends logs from a namespace within a cluster to a log destination. Logs are transported using one of the following:

  • The Syslog Protocol defined in RFC 5424
  • WebHook
ClusterMetricSink Metric sink Collects and writes metrics from a cluster to specified outputs using input and output plugins.

Prerequisites

Before creating a sink resource, do the following:

  • If you want to create a ClusterSink or Sink resource, ensure that the Enable Log Sink Resources checkbox is selected in the Enterprise PKS tile.
  • If you want to create a ClusterMetricSink resource, ensure that the Enable Metric Sink Resources checkbox is selected in the Enterprise PKS tile.

For more information, see the Logging section of the Enterprise PKS installation topic for your IaaS:

CLI Requirements

To create and manage sink resources, you must install the Kubernetes CLI, kubectl.

Alternately, to manage ClusterSink resources with the PKS CLI, you must use PKS CLI v1.3 or later.

For installation instructions, see Installing the Kubernetes CLI and Installing the PKS CLI.

Create Sinks

You can create log sinks for clusters and namespaces and metric sinks for clusters.

Note: Log sinks created in Enterprise PKS do not support UDP connections.

ClusterSink Resources

ClusterSink resources enable you to capture logs from your clusters.

To create and manage ClusterSink resources, you can:

Create ClusterSink Resource with the PKS CLI

To create and apply a log sink to a cluster, run the following command:

pks create-sink CLUSTER-NAME \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT

Where:

  • CLUSTER-NAME is the name of your cluster.
  • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

For example:

$ pks create-sink my-cluster syslog-tls://example.com:12345

If you do not specify a name, the command creates a sink resource in the cluster that shares the same name as the cluster.

To provide a name for the sink resources in your cluster, run the following command.

pks create-sink CLUSTER-NAME --name YOUR-SINK \
syslog-tls://YOUR-LOG-DESTINATION:YOUR-LOG-DESTINATION-PORT

Where:

  • CLUSTER-NAME is the name of your cluster.
  • YOUR-SINK is the name of the sink you want to create.
  • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.

For example:

$ pks create-sink my-cluster --name second-sink syslog-tls://example.org:54321

Specifying a name is useful if you need to manage multiple sink resources in your cluster.

Create a Syslog ClusterSink Resource with YAML and kubectl

A syslog ClusterSink resource delivers logs using the TCP-based syslog protocol.

To define a syslog ClusterSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: apps.pivotal.io/v1beta1
    kind: ClusterSink
    metadata:
       name: YOUR-SINK-NAME
    spec:
       type: syslog
       host: YOUR-LOG-DESTINATION
       port: YOUR-LOG-DESTINATION-PORT
       enable_tls: true
    

    Where:

    • YOUR-SINK is a name you choose for your sink.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
    • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.
  2. Save the YAML file with an appropriate file name. For example, my-cluster-sink.yml.

  3. Apply the ClusterSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml
    

    Where MY-SINK.yml is the name of your YAML file. For example:

    $ kubectl apply -f my-cluster-sink.yml
    

Create a Webhook ClusterSink Resource with YAML and kubectl

A webhook ClusterSink resource batches logs into 1 second units, wraps the resulting payload in JSON, and uses the POST method to deliver the logs to the address of your log management service.

To define a webhook ClusterSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: apps.pivotal.io/v1beta1
    kind: ClusterSink
    metadata:
      name: YOUR-SINK-NAME
    spec:
      type: webhook
      url: YOUR-LOG-DESTINATION
    

    Where:

    • YOUR-SINK-NAME is a name you choose for your sink.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  2. Save the YAML file with an appropriate file name. For example, my-cluster-sink.yml.

  3. Apply the ClusterSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml
    

    Where MY-SINK.yml is the name of your YAML file. For example:

    $ kubectl apply -f my-cluster-sink.yml
    

Sink Resources

A Sink resource filters logs by namespace within a cluster.

You can use only kubectl to create and manage namespace sinks. For instructions, see the following:

Create a Syslog Sink Resource with YAML and kubectl

To define a syslog Sink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: apps.pivotal.io/v1beta1
    kind: Sink
    metadata:
      name: YOUR-SINK
      namespace: YOUR-NAMESPACE
    spec:
      type: syslog
      host: YOUR-LOG-DESTINATION
      port: YOUR-LOG-DESTINATION-PORT
      enable_tls: true
    

    Where:

    • YOUR-SINK is a name you choose for your sink.
    • YOUR-NAMESPACE is the name of your namespace.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
    • YOUR-LOG-DESTINATION-PORT is the port number of your log management service.
  2. Save the YAML file with an appropriate file name. For example, my-namespace-sink.yml.

  3. Apply the Sink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml
    

    Where MY-SINK.yml is the name of your YAML file. For example:

    $ kubectl apply -f my-namespace-sink.yml
    

Create a Webhook Sink Resource with YAML and kubectl

To define a webhook Sink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file that specifies your log destination in the following format:

    apiVersion: apps.pivotal.io/v1beta1
    kind: Sink
    metadata:
      name: YOUR-SINK-NAME
      namespace: YOUR-NAMESPACE
    spec:
      type: webhook
      url: YOUR-LOG-DESTINATION
    

    Where:

    • YOUR-SINK-NAME is a name you choose for your sink.
    • YOUR-NAMESPACE is the name of your namespace.
    • YOUR-LOG-DESTINATION is the URL or IP address of your log management service.
  2. Save the YAML file with an appropriate file name. For example, my-namespace-sink.yml.

  3. Apply the Sink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml
    

    Where MY-SINK.yml is the name of your YAML file. For example:

    $ kubectl apply -f my-namespace-sink.yml
    

ClusterMetricSink Resources

A ClusterMetricSink resource collects metrics from a cluster using the Kubernetes Input Plugin and writes them to one or more outputs that you specify in the configuration of your ClusterMetricSink. For a list of supported metrics and output plugins, see Metrics and Output Plugins in the telegraf GitHub repository.

Currently, you can use only kubectl to create and manage metric sinks for clusters.

Create a ClusterMetricSink Resource with YAML and kubectl

To define a ClusterMetricSink resource with YAML and kubectl, perform the following steps:

  1. Create a YAML file in the following format:

    apiVersion: apps.pivotal.io/v1beta1
    kind: ClusterMetricSink
    metadata:
      name: YOUR-SINK
    spec:
      inputs:
      outputs:
      - type: YOUR-OUTPUT-PLUGIN
    

    Where:

    • YOUR-SINK is a name you choose for your sink.
    • YOUR-OUTPUT-PLUGIN is the name of the output plugin you want to use for your metrics.

      Note: You must leave the inputs field blank.

    For example:

    apiVersion: apps.pivotal.io/v1beta1
    kind: ClusterMetricSink
    metadata:
      name: http
    spec:
      inputs:
      outputs:
      - type: http
        url: https:example.com
        method: POST
        data_format: json
    
  2. Save the YAML file with an appropriate file name. For example, my-cluster-metric-sink.yml.

  3. Apply the ClusterMetricSink resource to your cluster by running the following command:

    kubectl apply -f MY-SINK.yml
    

    Where MY-SINK.yml is the name of your YAML file. For example:

    $ kubectl apply -f my-cluster-metric-sink.yml
    

List Sinks

To list sinks for clusters and namespaces, use the commands in the following sections.

ClusterSink Resources

To list log sinks deployed to your cluster, run the following command:

pks sinks CLUSTER-NAME

Where CLUSTER-NAME corresponds to the name of your cluster.

Alternately, you can use kubectl:

kubectl get clustersinks

Sink Resources

To list namespace log sinks, run the following command:

kubectl get sinks

ClusterMetricSink Resources

To list metric sinks deployed to your cluster, run the following command:

kubectl get clustermetricsinks

Delete Sinks

To delete sinks for clusters and namespaces, use the commands in the following sections.

ClusterSink Resources

To delete a cluster log sink, specify the name of the sink you want to delete:

pks delete-sink CLUSTER-NAME --name YOUR-SINK

Where:

  • CLUSTER-NAME is the name of your cluster.
  • YOUR-SINK is the name of your sink.

Alternately, you can use kubectl:

kubectl delete clustersink YOUR-SINK

Where YOUR-SINK is the name of your sink.

Sink Resources

To delete a namespace log sink, run the following command:

kubectl delete sink YOUR-SINK

Where YOUR-SINK is the name of your sink.

ClusterMetricSink Resources

To delete a metric sink deployed to your cluster, use the following command:

kubectl delete clustermetricsink YOUR-SINK

Where YOUR-SINK is the name of your sink.

For more information about sinks in Enterprise PKS, see the following topics:


Please send any feedback you have to pks-feedback@pivotal.io.