Monitoring Windows Worker Clusters and Nodes

Page last updated:

This topic describes how to install and configure components and integrations to capture Tanzu Kubernetes Grid Integrated Edition (TKGI) Windows worker Kubernetes cluster and worker node metrics.

Overview

Prometheus and Grafana are open source tools that you can use to monitor your Kubernetes Windows clusters and worker nodes, and to visualize and alert on events occurring in those clusters and worker nodes.

To monitor Windows clusters and worker node metrics using Prometheus and Grafana:

1. Install Prometheus and Grafana
2. Install the Windows Node Exporter
3. Administer Prometheus and Grafana
   

Prerequisites

Before installing Prometheus ensure you have installed kubectl. For more information about installing kubectl, see Install and Set Up kubectl in the Kubernetes documentation.

For additional Prometheus and Grafana prerequisites, see Prerequisites in the kube-prometheus GitHub repository.

Install Prometheus and Grafana

To install Prometheus and Grafana:

1. Download the Prometheus Source Code
   
2. Generate Your Prometheus Dashboard Configuration
   
3. Generate Monitoring Rules
4. Deploy Prometheus and Grafana
5. Verify Grafana is Running

Download the Prometheus Source Code

To download the Prometheus source code:

1. Clone or download the Prometheus operator source code from the prometheus-operator/kube-prometheus GitHub repository.

2. Clone or download the Prometheus monitoring mixin for kubernetes source code from the kubernetes-monitoring/kubernetes-mixin GitHub repository.

Generate Your Prometheus Dashboard Configuration

To generate a Prometheus dashboard configuration:

1. Open a command line.
2. Change directory to the kubernetes-mixin/dashboards source code directory.
3. Edit the dashboards.libsonnet configuration file.

4. To add the Windows dashboard to the configuration, add the following to the dashboards.libsonnet file:

   (import 'windows.libsonnet')
   

5. Save the file.

6. Change directory to the kubernetes-mixin source code root directory.

7. To generate the Prometheus dashboard, run:

   make dashboards_out
   

8. Review the kubernetes-mixin/dashboards_out directory and confirm the following Windows dashboard definition files were created:

   k8s-resources-windows-cluster.json
   k8s-resources-windows-namespace.json
   k8s-resources-windows-pod.json
   k8s-windows-cluster-rsrc-use.json
   k8s-windows-node-rsrc-use.json
   

Generate Monitoring Rules

To generate Prometheus monitoring rules:

1. Open a command line.

2. Copy the rule definition file from the kubernetes_mixin/rules directory to the kube-prometheus/jsonnet/kube-prometheus/rules directory:

   cp KUBERNETES-MIXIN-PATH/rules/windows.libsonnet PROMETHEUS-OP-PATH/jsonnet/kube-prometheus/rules/windows.libsonnet
   

   Where:

   • KUBERNETES-MIXIN-PATH is the kubernetes_mixin source directory.
     
   • PROMETHEUS-OP-PATH is the Prometheus operator path.
     

3. Change directory to the kube-prometheus/jsonnet/kube-prometheus/rules source code directory.

4. Edit the rules.libsonnet configuration file.

5. To add the Windows rule to the configuration, add the following to the rules.libsonnet file:

   (import 'windows.libsonnet')
   

6. Save the file.

7. Change directory to the kube-prometheus Prometheus source code root directory.

8. To generate all rules, run:

   make manifests
   

Deploy Prometheus and Grafana

To deploy Prometheus and Grafana:

1. Get kubeconfig for your Windows cluster. For more information, see Retrieving Cluster Credentials and Configuration.
   
2. Open a command line.
3. Change directory to the kube-prometheus Prometheus source code root directory.

4. To deploy Prometheus and Grafana, run:

   kubectl create -f manifests/setup
   until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
   kubectl create -f manifests/
   

5. To forward the Prometheus and Grafana ports to localhost, run:

   kubectl --namespace monitoring port-forward svc/prometheus-k8s PROMETHEUS-PORT
   kubectl --namespace monitoring port-forward svc/grafana GRAFANA-PORT
   

   Where:

   • PROMETHEUS-PORT is a localhost port for Prometheus, for example 9090.
   • GRAFANA-PORT is a localhost port for Grafana, for example 3000.

For more information about installing Prometheus and Grafana, see Quickstart in the prometheus-operator/kube-prometheus GitHub repository.

Verify Grafana is Running

To verify Grafana is running:

1. To open the Grafana Dashboard, open a browser to:

   http://localhost:GRAFANA-PORT
   

   Where GRAFANA-PORT is the localhost port for Grafana, for example 3000.

2. To log in to the Grafana Dashboard authenticate with user admin, password admin.

3. To view Kubelet metrics from all of your windows workers, navigate to Dashboard > Default > Kubernetes/Kubelet.

Install the Windows Node Exporter

To monitor Windows worker node metrics, you need endpoints, the Windows Node Exporter service, and the Windows Node Exporter service monitor.

To install and configure the Windows Node Exporter:

1. Deploy the Windows Node Exporter
2. Configure the Windows Node Exporter
3. Set Up the Grafana Windows Node Dashboard
   

Deploy the Windows Node Exporter

The Windows Node Exporter must be installed on the Windows worker nodes that you want to monitor.

To determine the names and IP addresses for all your Windows worker nodes:

1. Open a command line.

2. Run the following:

   kubectl get nodes -L spec.ip
   

3. Note the names and IP addresses of the Windows worker nodes.

To deploy the Windows Node Exporter to your Windows worker nodes, repeat the steps below for each Windows worker node you want to monitor:

1. Start an SSH session to the Windows worker.
   

2. To download the windows_exporter-0.13.0-amd64.msi to the worker from v0.13.0 in the prometheus-community/windows_exporter repository on GitHub:

   Powershell Invoke-WebRequest https://github.com/prometheus-community/windows_exporter/releases/download/v0.13.0/windows_exporter-0.13.0-amd64.msi ^
       -OutFile DOWNLOAD-PATH 
   

   Where DOWNLOAD-PATH is the full path to the output location and filename for the retrieved file. For example, c:\windows_exporter-0.13.0-amd64.msi.

3. To start the windows_exporter msi as a Windows service, open a command line:

   msiexec /i DOWNLOAD-PATH ENABLED_COLLECTORS="cpu,cs,logical_disk,net,os,system,container,memory"
   

   Where DOWNLOAD-PATH is the full path to the output location and filename for the retrieved file. For example, c:\windows_exporter-0.13.0-amd64.msi.

Configure the Windows Node Exporter

To support monitoring by Prometheus and Grafana, the Windows Node Exporter service must be configured to access your Windows worker node metrics.

To configure the Windows Node Exporter:

1. Create a file named windows-exporter.yml.

2. Populate the file with the following:

   apiVersion: v1
   kind: Endpoints
   metadata:
     labels:
       k8s-app: APP-NAME
     name: ENDPOINT-NAME
     namespace: kube-system
   subsets:
   - addresses:
     - ip: NODE-IP
       targetRef:
         kind: Node
         name: NODE-NAME
     ports:
     - name: http-metrics
       port: 9182
       protocol: TCP
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       k8s-app: APP-NAME
     name: SERVICE-NAME
     namespace: kube-system
   spec:
     clusterIP: None
     ports:
     - name: http-metrics
       port: 9182
       protocol: TCP
       targetPort: 9182
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: APP-NAME
     name: SERVICEMONITOR-NAME
     namespace: monitoring
   spec:
     endpoints:
     - interval: 30s
       port: http-metrics
     jobLabel: k8s-app
     namespaceSelector:
       matchNames:
       - kube-system
     selector:
       matchLabels:
         k8s-app: APP-NAME
   

   Where:

   • APP-NAME is the name for the Kubernetes app.
     
   • ENDPOINT-NAME is the name for the endpoint.
     
   • SERVICE-NAME is the name for the service.
     
   • SERVICEMONITOR-NAME is the name for the service monitor.
     
   • NODE-IP is the IP address of the Windows worker node to monitor.
     
   • NODE-NAME is the name of the Windows worker node to monitor.
     

   For example:

   apiVersion: v1
   kind: Endpoints
   metadata:
     labels:
       k8s-app: wmi-exporter
     name: wmi-exporter
     namespace: kube-system
   subsets:
   - addresses:
     - ip: 192.168.1.1
       targetRef:
         kind: Node
         name: nodeone
     - ip: 192.168.1.1
       targetRef:
         kind: Node
         name: nodetwo
     ports:
     - name: http-metrics
       port: 9182
       protocol: TCP
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       k8s-app: wmi-exporter
     name: wmi-exporter
     namespace: kube-system
   spec:
     clusterIP: None
     ports:
     - name: http-metrics
       port: 9182
       protocol: TCP
       targetPort: 9182
     sessionAffinity: None
     type: ClusterIP
   ---
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       k8s-app: wmi-exporter
     name: wmi-exporter
     namespace: mornitoring
   spec:
     endpoints:
     - interval: 30s
       port: http-metrics
     jobLabel: k8s-app
     namespaceSelector:
       matchNames:
       - kube-system
     selector:
       matchLabels:
         k8s-app: wmi-exporter
   

3. Create an additional subsets.addresses item for each Windows worker node you want to monitor:

   subsets:
   - addresses:
     - ip: NODE-ONE-IP
       targetRef:
         kind: Node
         name: NODE-ONE-NAME
     - ip: NODE-TWO-IP
       targetRef:
         kind: Node
         name: NODE-TWO-NAME
     - ip: NODE-THREE-IP
       targetRef:
         kind: Node
         name: NODE-THREE-NAME
   

   Where:

   • NODE-ONE-IP is the IP address of the first Windows worker node to monitor.
   • NODE-ONE-NAME is the name of the first Windows worker node to monitor.
   • NODE-TWO-IP is the IP address of the second Windows worker node to monitor.
   • NODE-TWO-NAME is the name of the second Windows worker node to monitor.
   • NODE-THREE-IP is the IP address of the third Windows worker node to monitor.
   • NODE-THREE-NAME is the name of the third Windows worker node to monitor.

4. Save your changes to the windows-exporter.yml file.

5. To create the Windows node exporter objects, run:

   kubectl create -f windows-exporter.yml
   

6. Wait a few seconds for the service to be ready.

7. To verify your new targets in Prometheus, open a browser to:

   http://localhost:PROMETHEUS-PORT/targets
   

   Where PROMETHEUS-PORT is the localhost port for Prometheus, for example 9090.

Set Up the Grafana Windows Node Dashboard

To import the dashboard files generated above to Grafana:

1. Open a browser to your Grafana dashboard.
2. To open the Import tab, click + on the Grafana side menu.
3. Click Upload .json file.
   

4. Upload the five dashboard files from the kubernetes-mixin/dashboards_out directory one at a time:

   • k8s-resources-windows-cluster.json
   • k8s-resources-windows-namespace.json
   • k8s-resources-windows-pod.json
   • k8s-windows-cluster-rsrc-use.json
   • k8s-windows-node-rsrc-use.json

5. To verify the dashboard, open a browser to your Grafana dashboard.

To use a different Grafana dashboard, see Dashboards at the Grafana Labs site.

Administer Prometheus and Grafana

After completing the installation steps above:

• Change your Grafana account authentication to not use the default admin/admin user/password pair.
  

• Review the Grafana and Prometheus documentation for any additional steps you need to complete:

  • For information about managing Grafana, see Administration in the Grafana documentation.
    
  • For information about managing Prometheus, see Customizing Kube-Prometheus in the kube-prometheus documentation.