Monitoring Master/etcd Node VMs

Page last updated:

This topic describes how platform operators can monitor and retrieve metrics from master/etcd node VMs in Enterprise Pivotal Container Service (Enterprise PKS) deployments.

Overview

etcd is an open source distributed key value store. In Enterprise PKS, etcd is co-located on Kubernetes master node VMs. The master node VMs use etcd for service discovery and configuration sharing within the cluster. You can monitor Kubernetes cluster health by using metrics from etcd.

For more information about etcd, see the etcd documentation in GitHub.

For more information about configuring master/etcd nodes in the Enterprise PKS tile, see the Plans section of Installing Enterprise PKS for your IaaS:

Collect Metrics Using etcd

You can configure the Telgraf agent to collect metrics from etcd and send the metrics to a third-party monitoring service. The Telgraf agent can read metrics from the following:

  • a /metric endpoint exposed by the etcd VM
  • the Node Exporter

Node Exporter provides additional infrastructure metrics on the localhost of the master node. For more information, see node_exporter in GitHub.

To monitor etcd with a third-party monitoring service, do the following:

  1. Create a configuration file for your output plugin. See Create a Configuration File
  2. Configure Telegraf in the Enterprise PKS tile. See Configure Telegraf in the Tile.

For more information about monitoring etcd, see Monitoring etcd in GitHub.

Create a Configuration File

To connect a third-party monitoring service to Enterprise PKS, you must create a configuration file for the service. The configuration file is written in a TOML format and consists of key-value pairs. After you create your configuration file, you can enter the file into the Enterprise PKS tile to connect the service.

To create a configuration file your monitoring service, do the following:

  1. Locate the required format for your monitoring service in the README.md file for your service in telegraf in GitHub. The etcd documentation recommends using the open source Prometheus monitoring service.

    For more information about using Prometheus, see Overview in the Prometheus documentation.

  2. Create your configuration file using the required format of your monitoring service. For example, if you want to create a configuration file for an HTTP output plugin, create a file similar to the following:

    [[outputs.http]]
       url="https://example.com"
       method="POST"
       data_format="json"
    [[processors.override]]
      [processors.override.tags]
        director = "bosh-director-1"
    

    Note: You can add tags to your configuration file to label etcd metrics. For example, the above code snippet adds a bosh-director-1 tag to the etcd metrics. If you have multiple BOSH Directors, Pivotal recommends adding tags to filter your metrics in your monitoring service.

Configure Telegraf in the Tile

To connect a monitoring service to etcd, do the following:

  1. From the Ops Manager Installation Dashboard, navigate to Enterprise PKS > Settings > Monitoring.

    Monitoring pane

  2. (Optional) If you want to send metrics from Node Exporter, select Enable Node-Exporter on master. This exposes a wide variety of node metrics. For more information about these metrics, see the Node Exporter repository in GitHub.

  3. For Setup Telegraf Outputs, enter the contents of the configuration file you created in Create a Configuration File above. If you do not want to output any metrics, leave the default value [[outputs.discard]].

  4. Click Save

  5. Click Apply Changes.

Troubleshoot etcd

Pivotal recommends working with Pivotal or VMware Support to troubleshoot master/etcd node VMs. The monitoring and metrics data you retrieve from the master/etcd node VMs can help the Support team diagnose and troubleshoot errors.


Please send any feedback you have to pks-feedback@pivotal.io.