Logging Windows Worker Workloads
Page last updated:
This topic describes how to install and configure components and integrations to capture VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) Windows Worker Kubernetes cluster and worker node logs.
Prerequisites
Before starting the tasks in this topic:
- Your environment should be a Windows 2019 data center with vRLI.
- You must have an accessible container image registry.
- Docker must be installed on the local machine you will be working from. For more information, see Prepare the Working Environment below.
Overview
This procedure describes how to send logs to vRLI from Windows workers on TKGI-provisioned Windows clusters using Fluent Bit.
In addition to deploying Fluent Bit, you must also deploy the syslog and Kubernetes filter plugins. The Kubernetes filter plugin is a natively supported Fluent Bit filter which improves logging by adding Kubernetes metadata, such as namespace and Pod name to your logs.
To send Windows worker logs to vRLI using Fluent Bit:
Install Fluent Bit
To install Fluent Bit:
- Build a Windows Fluent Bit Docker Image
- Build a Windows Fluent Bit Syslog Plugin Docker Image
- Configure Fluent Bit
- Deploy Fluent Bit on the Windows Cluster
Build a Windows Fluent Bit Docker Image
To build a Fluent Bit Windows Docker image:
- Log in to the Windows 2019 machine where you will do your work.
- Download the Fluent Bit source code from the fluent/fluent-bit repository on GitHub.
- Configure Docker for creating Windows containers.
For more information, see Prepare the Working Environment below.
To build a Fluent Bit container image, run:
docker.exe build -f Dockerfile.windows -t fluent-bit .
Note: Docker downloads the Microsoft Visual C++ Redistributable Update while building the Fluent Bit container and installs
vc_redist.x64.exe
in the new container. If this process fails, see Troubleshooting below.Push the Fluent Bit container image to your registry.
Build a Windows Fluent Bit Syslog Plugin Docker Image
Fluent Bit does not officially support syslog output. To send TKGI Windows worker logs to vRLI, you must use a syslog plugin.
To build a Fluent Bit syslog plugin image:
- Download the syslog plugin from the benmoss/fluent-bit-out-syslog GitHub repository.
To remove a conflict between this syslog plugin and the default Fluent Bit syslog output:
- Locate the
main.go
file you downloaded to.../windows/cmd/main.go
. - Open the
main.go
file in a text editor. Change the
out_syslog
plugin setting tosyslogvrli
.
For example://export FLBPluginRegister func FLBPluginRegister(def unsafe.Pointer) int { return output.FLBPluginRegister( def, "syslogvrli", "syslog output plugin that follows RFC 5424", ) }
Save the file.
- Locate the
Modify the
Dockerfile.windows
file:Remove the following lines from the file:
ARG BASE_IMAGE=mcr.microsoft.com/windows/servercore:ltsc2019 FROM $BASE_IMAGE as builder
Replace those lines with the following:
FROM REGISTRY-IMAGE
Where
REGISTRY-IMAGE
is the registry image location of the container you built in Build a Windows Fluent Bit Docker Image above.
For example:FROM example.com/k8s-services/windows-images/fluent-bit:latest
Save the file.
Configure Docker for creating Windows containers. For more information, see Prepare the Working Environment below.
To build the container, use:
docker.exe build -f Dockerfile.windows -t FLUENT-BIT-IMAGE
Where
FLUENT-BIT-IMAGE
is the name for the Fluent Bit image to store in your registry.
For example:docker.exe build -f Dockerfile.windows -t fluent-bit-syslog
Push the image to your registry.
Configure Fluent Bit
The Fluent Bit Service Account, Cluster Role, and Cluster Role Binding objects ensure that the
Fluent Bit Kubernetes filter can access and read metadata from the
Kubernetes API server kubernetes.default.svc.cluster.local:443
.
Your Fluent Bit configuration is a YAML file that defines the Fluent Bit Service Account, Cluster Role, Cluster Role Binding and other objects.
To create a Fluent Bit deployment configuration file:
- Create a file named
fluent-bit.yml
. Populate the file with the following:
apiVersion: v1 kind: ServiceAccount metadata: name: fluent-bit-win namespace: pks-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: fluent-bit-read rules: - apiGroups: [""] resources: - namespaces - pods verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: fluent-bit-read roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluent-bit-read subjects: - kind: ServiceAccount name: fluent-bit-win namespace: pks-system --- apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-win labels: app: fluent-bit namespace: pks-system data: fluent-bit.conf: | [SERVICE] Flush 5 Log_Level debug Daemon off Parsers_File parsers.conf [INPUT] Name tail Tag kube.* Path C:\var\log\containers\*.log Parser docker DB /var/log/flb_kube1.db Skip_Long_Lines On Refresh_Interval 60 [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc.cluster.local:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Merge_Log On DNS_Retries 10 Kube_Tag_Prefix kube.c.var.log.containers. K8S-Logging.Parser On [OUTPUT] Name PLUG-IN-ID #Use the plug-in id you changed before you use the image. Match * InstanceName plaintext-cluster-sink Addr OUTPUT-ADDRESS:OUTPUT-PORT #If you enable TLS, please use port 1514. Cluster true #TLSConfig {"insecure_skip_verify":true} #If you enable TLS, place TLS-related configurations here. parsers.conf: | [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: fluent-bit name: fluent-bit-windows namespace: pks-system spec: selector: matchLabels: app: fluent-bit template: metadata: labels: app: fluent-bit spec: nodeSelector: beta.kubernetes.io/os: windows tolerations: - key: "windows" operator: "Equal" value: "2019" effect: "NoSchedule" containers: - image: FLUENT-BIT-IMAGE:latest #Change image to point to the image in your registry. imagePullPolicy: IfNotPresent name: fluent-bit command: [fluent-bit] args: - --plugin - /syslog-plugin/out_syslog.so - --config - /fluent-bit/etc/fluent-bit.conf workingDir: /fluent-bit/etc volumeMounts: - mountPath: /fluent-bit/etc name: fluent-bit-config - mountPath: /var/log name: varlog readOnly: false - mountPath: /ProgramData/docker/containers name: dockercontainers readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: dockercontainers hostPath: path: /ProgramData/docker/containers - configMap: defaultMode: 420 name: fluent-bit-win name: fluent-bit-config serviceAccountName: fluent-bit-win updateStrategy: type: RollingUpdate
Where:
PLUG-IN-ID
is the plug in ID for your plug in. For example,syslogvrli
.OUTPUT-ADDRESS
is the IP address of your vRealize Log Insight installation.OUTPUT-PORT
is the port to use to communicate with your vRealize Log Insight installation. Use port514
for most installations. Use port1514
if you enable TLS.FLUENT-BIT-IMAGE
is the name of the Fluent Bit image in your registry. For example,fluent-bit-syslog
.
Save the file.
Deploy Fluent Bit on the Windows Cluster
To send container logs to vRLI, deploy Fluent Bit and related objects using your deployment configuration file.
To deploy Fluent Bit:
Deploy Fluent Bit using
kubectl
:kubectl create -f CONFIG-FILE
Where
CONFIG-FILE
is the filename of your Fluent Bit deployment configuration file.
For example:kubectl create -f fluent-bit.yml
Validate the Fluent Bit Deployment Using a Sample App
Configure a Sample App
To confirm that logging is working correctly, use a sample app that outputs log entries frequently.
The logspewer
sample app defined below outputs a log every 10 seconds when running.
To configure a logspewer
sample app for testing:
- Create a file named
sample.yml
. Populate the file with the following:
apiVersion: apps/v1 kind: Deployment metadata: name: logspewer namespace: pks-system labels: app: logspewer spec: replicas: 1 selector: matchLabels: app: logspewer template: metadata: labels: app: logspewer spec: containers: - name: logspewer image: pivotalgreenhouse/logspewer:latest env: - name: INTERVAL_IN_SECONDS value: "10" nodeSelector: kubernetes.io/os: windows tolerations: - key: "windows" operator: "Equal" value: "2019" effect: "NoSchedule"
Save the file.
Deploy the Sample App
Deploy the test app using kubectl
:
To deploy the test app:
kubectl create -f CONFIG-FILE
Where
CONFIG-FILE
is the filename of your sample app deployment configuration file.
For example:kubectl create -f sample.yml
Validate the Fluent Bit Deployment
Validate your Fluent Bit configuration and confirm that Fluent Bit is functioning using the test app you created.
To confirm the sample app’s logs are being written to vRLI:
- Open vRLI.
- Open the vRLI > Interactive Analytics tab.
- Search the list for “logspewer”. The “logspewer” items are the log entries generated by the sample app.
Prepare the Working Environment
To prepare your working environment:
- Install
docker
on the local machine you will be working from. If you are working from a TKGI-provisioned Windows worker node docker is already installed. To configure Docker to create Windows containers:
- To open the Docker Desktop menu, click the Docker icon in the Windows system tray notification area.
- If the Switch to Windows containers… option is present on the Docker Desktop menu, click it.
The Docker Desktop menu should now display the Switch to Linux containers… option.
For more information, see Switch between Windows and Linux containers in Docker Desktop for Windows user manual in the Docker documentation.
- To open the Docker Desktop menu, click the Docker icon in the Windows system tray notification area.
Troubleshooting
‘Cannot Find Path’ Error When Creating the Fluent Bit Docker Container
Symptom
The error message Copy-Item : Cannot find path 'C:\Windows\System32\msvcp140.dll'
is displayed while Docker
installs the Microsoft Visual C++ Redistributable Update.
Description
When Docker builds the Fluent Bit container,
it installs the Microsoft Visual C++ Redistributable Update to the container.
To do this, it downloads the Redistributable Update as vc_redist.x64.exe
,
installs the update in the new Docker container, and copies three DLL files to the /fluent-bit/bin/
directory.
If a Copy-Item : Cannot find path
error is returned for either msvcp140.dll
, vccorlib140.dll
, or
vcruntime140.dll
the installation of vc_redist.x64.exe
has failed.
Workaround
To manually install the Microsoft Visual C++ Redistributable Update to the Fluent Bit container:
- Download the Microsoft Visual C++ Redistributable Update,
vc_redist.x64.exe
, from Microsoft. - Install
vc_redist.x64.exe
on your local Windows 2019 machine. - Copy the following files from
C:\Windows\System32\
to the directory containingdockerfile.windows
:msvcp140.dll
,vccorlib140.dll
, andvcruntime140.dll
. Modify your
Dockerfile.windows
file:Remove the following lines from the file:
RUN Write-Host ('Installing Visual C++ Redistributable Package'); ` Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; ` Copy-Item -Path /Windows/System32/msvcp140.dll -Destination /fluent-bit/bin/; ` Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; ` Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;
Replace those lines with the following:
RUN Write-Host ('Installing Visual C++ Redistributable Package'); ` Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '/norestart' -NoNewWindow -Wait; COPY msvcp140.dll /fluent-bit/bin/; COPY vccorlib140.dll /fluent-bit/bin/; COPY vcruntime140.dll /fluent-bit/bin/;
Save the file.
Re-run the docker build command as shown in Build a Windows Fluent Bit Docker Image above.
Please send any feedback you have to pks-feedback@pivotal.io.