Logging Windows Worker Workloads

Page last updated:

This topic describes how to install and configure components and integrations to capture VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Windows Worker Kubernetes cluster and worker node logs.

Prerequisites

Before starting the tasks in this topic:

  • Your environment should be a Windows 2019 data center with vRLI.
  • You must have an accessible container image registry.
  • Docker must be installed on the local machine you will be working from. For more information, see Prepare the Working Environment below.

Overview

This procedure describes how to send logs to vRLI from Windows workers on TKGI-provisioned Windows clusters using Fluent Bit.

In addition to deploying Fluent Bit, you must also deploy the syslog and Kubernetes filter plugins. The Kubernetes filter plugin is a natively supported Fluent Bit filter which improves logging by adding Kubernetes metadata, such as namespace and Pod name to your logs.

To send Windows worker logs to vRLI using Fluent Bit:

  1. Install Fluent Bit
  2. Validate the Fluent Bit Deployment Using a Sample App

Install Fluent Bit

To install Fluent Bit:

  1. Build a Windows Fluent Bit Docker Image
  2. Build a Windows Fluent Bit Syslog Plugin Docker Image
  3. Configure Fluent Bit
  4. Deploy Fluent Bit on the Windows Cluster

Build a Windows Fluent Bit Docker Image

To build a Fluent Bit Windows Docker image:

  1. Log in to the Windows 2019 machine where you will do your work.
  2. Download the Fluent Bit source code from the fluent/fluent-bit repository on GitHub.
  3. Configure Docker for creating Windows containers. For more information, see Prepare the Working Environment below.
  4. To build a Fluent Bit container image, run:

    docker.exe build  -f Dockerfile.windows -t fluent-bit .
    

    Note: Docker downloads the Microsoft Visual C++ Redistributable Update while building the Fluent Bit container and installs vc_redist.x64.exe in the new container. If this process fails, see Troubleshooting below.

  5. Push the Fluent Bit container image to your registry.

Build a Windows Fluent Bit Syslog Plugin Docker Image

Fluent Bit does not officially support syslog output. To send TKGI Windows worker logs to vRLI, you must use a syslog plugin.

To build a Fluent Bit syslog plugin image:

  1. Download the syslog plugin from the benmoss/fluent-bit-out-syslog GitHub repository.
  2. To remove a conflict between this syslog plugin and the default Fluent Bit syslog output:

    1. Locate the main.go file you downloaded to .../windows/cmd/main.go.
    2. Open the main.go file in a text editor.
    3. Change theout_syslog plugin setting to syslogvrli.

      For example:

      //export FLBPluginRegister
      func FLBPluginRegister(def unsafe.Pointer) int {
          return output.FLBPluginRegister(
              def,
              "syslogvrli",
              "syslog output plugin that follows RFC 5424",
          )
      }
      
    4. Save the file.

  3. Modify the Dockerfile.windows file:

    1. Remove the following lines from the file:

      ARG BASE_IMAGE=mcr.microsoft.com/windows/servercore:ltsc2019
      FROM $BASE_IMAGE as builder
      
    2. Replace those lines with the following:

      FROM REGISTRY-IMAGE
      

      Where REGISTRY-IMAGE is the registry image location of the container you built in Build a Windows Fluent Bit Docker Image above.

      For example:

      FROM example.com/k8s-services/windows-images/fluent-bit:latest
      
    3. Save the file.

  4. Configure Docker for creating Windows containers. For more information, see Prepare the Working Environment below.

  5. To build the container, use:

    docker.exe build  -f Dockerfile.windows -t FLUENT-BIT-IMAGE
    

    Where FLUENT-BIT-IMAGE is the name for the Fluent Bit image to store in your registry.

    For example:

    docker.exe build  -f Dockerfile.windows -t fluent-bit-syslog
    
  6. Push the image to your registry.

Configure Fluent Bit

The Fluent Bit Service Account, Cluster Role, and Cluster Role Binding objects ensure that the Fluent Bit Kubernetes filter can access and read metadata from the Kubernetes API server kubernetes.default.svc.cluster.local:443.

Your Fluent Bit configuration is a YAML file that defines the Fluent Bit Service Account, Cluster Role, Cluster Role Binding and other objects.

To create a Fluent Bit deployment configuration file:

  1. Create a file named fluent-bit.yml.
  2. Populate the file with the following:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: fluent-bit-win
      namespace: pks-system
    --- 
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: fluent-bit-read
    rules:
    - apiGroups: [""]
      resources:
      - namespaces
      - pods
      verbs: ["get", "list", "watch"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: fluent-bit-read
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: fluent-bit-read
    subjects:
    - kind: ServiceAccount
      name: fluent-bit-win
      namespace: pks-system
    ---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: fluent-bit-win
      labels:
        app: fluent-bit
      namespace: pks-system
    data:
      fluent-bit.conf: |
        [SERVICE]
            Flush         5
            Log_Level     debug
            Daemon        off
            Parsers_File  parsers.conf
    
        [INPUT]
            Name              tail
            Tag               kube.*
            Path              C:\var\log\containers\*.log
            Parser            docker
            DB                /var/log/flb_kube1.db
            Skip_Long_Lines   On
            Refresh_Interval  60
    
        [FILTER]
            Name                kubernetes
            Match               kube.*
            Kube_URL            https://kubernetes.default.svc.cluster.local:443
            Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
            Merge_Log           On
            DNS_Retries         10
            Kube_Tag_Prefix     kube.c.var.log.containers.
            K8S-Logging.Parser  On
    
        [OUTPUT]
            Name           PLUG-IN-ID                              #Use the plug-in id you changed before you use the image.
            Match          *
            InstanceName  plaintext-cluster-sink
            Addr          OUTPUT-ADDRESS:OUTPUT-PORT               #If you enable TLS, please use port 1514.
            Cluster       true
            #TLSConfig     {"insecure_skip_verify":true}           #If you enable TLS, place TLS-related configurations here.
      parsers.conf: |
        [PARSER]
            Name   json
            Format json
            Time_Key time
            Time_Format %d/%b/%Y:%H:%M:%S %z
    
        [PARSER]
            Name        docker
            Format      json
            Time_Key    time
            Time_Format %Y-%m-%dT%H:%M:%S.%L
            Time_Keep   On
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      labels:
        app: fluent-bit
      name: fluent-bit-windows
      namespace: pks-system
    spec:
      selector:
        matchLabels:
          app: fluent-bit
      template:
        metadata:
          labels:
            app: fluent-bit
        spec:
          nodeSelector:
            beta.kubernetes.io/os: windows
          tolerations:
          - key: "windows"
            operator: "Equal"
            value: "2019"
            effect: "NoSchedule"
          containers:
          - image: FLUENT-BIT-IMAGE:latest                                          #Change image to point to the image in your registry.
            imagePullPolicy: IfNotPresent
            name: fluent-bit
            command: [fluent-bit]
            args:
            - --plugin
            - /syslog-plugin/out_syslog.so
            - --config
            - /fluent-bit/etc/fluent-bit.conf
            workingDir: /fluent-bit/etc
            volumeMounts:
            - mountPath: /fluent-bit/etc
              name: fluent-bit-config
            - mountPath: /var/log
              name: varlog
              readOnly: false
            - mountPath: /ProgramData/docker/containers
              name: dockercontainers
              readOnly: true
          volumes:
          - name: varlog
            hostPath:
              path: /var/log
          - name: dockercontainers
            hostPath:
              path: /ProgramData/docker/containers
          - configMap:
              defaultMode: 420
              name: fluent-bit-win                                        
            name: fluent-bit-config
          serviceAccountName: fluent-bit-win
      updateStrategy:
        type: RollingUpdate
    

    Where:

    • PLUG-IN-ID is the plug in ID for your plug in. For example, syslogvrli.
    • OUTPUT-ADDRESS is the IP address of your vRealize Log Insight installation.
    • OUTPUT-PORT is the port to use to communicate with your vRealize Log Insight installation. Use port 514 for most installations. Use port 1514 if you enable TLS.
    • FLUENT-BIT-IMAGE is the name of the Fluent Bit image in your registry. For example, fluent-bit-syslog.
  3. Save the file.

Deploy Fluent Bit on the Windows Cluster

To send container logs to vRLI, deploy Fluent Bit and related objects using your deployment configuration file.

To deploy Fluent Bit:

  1. Deploy Fluent Bit using kubectl:

    kubectl create -f CONFIG-FILE
    

    Where CONFIG-FILE is the filename of your Fluent Bit deployment configuration file.

    For example:

    kubectl create -f fluent-bit.yml
    

Validate the Fluent Bit Deployment Using a Sample App

  1. Configure a Sample App
  2. Deploy the Sample App
  3. Validate the Fluent Bit Deployment

Configure a Sample App

To confirm that logging is working correctly, use a sample app that outputs log entries frequently. The logspewer sample app defined below outputs a log every 10 seconds when running.

To configure a logspewer sample app for testing:

  1. Create a file named sample.yml.
  2. Populate the file with the following:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: logspewer
      namespace: pks-system
      labels:
        app: logspewer
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: logspewer
      template:
        metadata:
          labels:
            app: logspewer
        spec:
          containers:
          - name: logspewer
            image: pivotalgreenhouse/logspewer:latest
            env:
            - name: INTERVAL_IN_SECONDS
              value: "10"
          nodeSelector:
            kubernetes.io/os: windows
          tolerations:
          - key: "windows"
            operator: "Equal"
            value: "2019"
            effect: "NoSchedule"
    
  3. Save the file.

Deploy the Sample App

Deploy the test app using kubectl:

  1. To deploy the test app:

    kubectl create -f CONFIG-FILE
    

    Where CONFIG-FILE is the filename of your sample app deployment configuration file.

    For example:

    kubectl create -f sample.yml
    

Validate the Fluent Bit Deployment

Validate your Fluent Bit configuration and confirm that Fluent Bit is functioning using the test app you created.

To confirm the sample app’s logs are being written to vRLI:

  1. Open vRLI.
  2. Open the vRLI > Interactive Analytics tab.
  3. Search the list for “logspewer”. The “logspewer” items are the log entries generated by the sample app.

Prepare the Working Environment

To prepare your working environment:

  1. Install docker on the local machine you will be working from. If you are working from a TKGI-provisioned Windows worker node docker is already installed.
  2. To configure Docker to create Windows containers:

    1. To open the Docker Desktop menu, click the Docker icon in the Windows system tray notification area.
    2. If the Switch to Windows containers… option is present on the Docker Desktop menu, click it. The Docker Desktop menu should now display the Switch to Linux containers… option.

    For more information, see Switch between Windows and Linux containers in Docker Desktop for Windows user manual in the Docker documentation.

Troubleshooting

‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container

Symptom

The error message Copy-Item : Cannot find path 'C:\Windows\System32\msvcp140.dll' is displayed while Docker installs the Microsoft Visual C++ Redistributable Update.

Description

When Docker builds the Fluent Bit container, it installs the Microsoft Visual C++ Redistributable Update to the container. To do this, it downloads the Redistributable Update as vc_redist.x64.exe, installs the update in the new Docker container, and copies three DLL files to the /fluent-bit/bin/ directory.

If a Copy-Item : Cannot find path error is returned for either msvcp140.dll, vccorlib140.dll, or vcruntime140.dll the installation of vc_redist.x64.exe has failed.

Workaround

To manually install the Microsoft Visual C++ Redistributable Update to the Fluent Bit container:

  1. Download the Microsoft Visual C++ Redistributable Update, vc_redist.x64.exe, from Microsoft.
  2. Install vc_redist.x64.exe on your local Windows 2019 machine.
  3. Copy the following files from C:\Windows\System32\ to the directory containing dockerfile.windows: msvcp140.dll, vccorlib140.dll, and vcruntime140.dll.
  4. Modify your Dockerfile.windows file:

    1. Remove the following lines from the file:

      RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
          Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; `
          Copy-Item -Path /Windows/System32/msvcp140.dll -Destination /fluent-bit/bin/; `
          Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; `
          Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;
      
    2. Replace those lines with the following:

      RUN Write-Host ('Installing Visual C++ Redistributable Package'); `
          Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; 
      
      COPY msvcp140.dll  /fluent-bit/bin/; 
      COPY vccorlib140.dll /fluent-bit/bin/; 
      COPY vcruntime140.dll /fluent-bit/bin/;
      
    3. Save the file.

  5. Re-run the docker build command as shown in Build a Windows Fluent Bit Docker Image above.


Please send any feedback you have to pks-feedback@pivotal.io.