Configuring Windows Worker-Based Kubernetes Clusters (Beta)

Page last updated:

This topic describes configuring Windows worker-based Kubernetes clusters in VMware Enterprise PKS.

Overview

In Enterprise PKS you can provision a Windows worker-based Kubernetes cluster on vSphere with Flannel.

To provision a Windows worker-based Kubernetes cluster:

  1. Verify your environment meets the Windows worker-based Kubernetes cluster Prerequisites.
  2. Configure a Windows Worker-Based Kubernetes Cluster.
  3. Upload the Windows Server Stemcell.
  4. Create a Windows Worker-Based Cluster.

For information about the architecture of Enterprise PKS Windows worker-based Kubernetes clusters, see Windows Worker-Based Kubernetes Cluster High Availability in Enterprise PKS Architecture.

Warning: Support for Windows-based Kubernetes clusters is in beta and supports only vSphere with Flannel.

Do not enable this feature if you are using Enterprise PKS with vSphere with NSX-T, Google Cloud Platform (GCP), Azure, or Amazon Web Services (AWS).

We are actively looking for feedback on this beta feature. To submit feedback, send an email to pcf-windows@pivotal.io.

Prerequisites

The following are required for creating a Windows worker-based Kubernetes cluster in Enterprise PKS:

Configure a Windows Worker-Based Kubernetes Cluster

  1. Configure a Windows worker plan as described in Plans, below.
  2. Configure Windows worker networking as described in Networking, below.
  3. Upload the Windows Server stemcell as described in Upload the Windows Server Stemcell, below.
  4. Click Apply Changes to complete the configuration changes.

Plans

A plan defines a set of resource types used for deploying a cluster.

Activate a Plan

Note: Before configuring your Windows worker plan, you must first activate and configure Plan 1. See Plans in Installing Enterprise PKS on vSphere for more information.

To activate and configure a plan, perform the following steps:

  1. Click the plan that you want to activate. You must activate and configure either Plan 11, Plan 12, or Plan 13 to deploy a Windows worker-based cluster.

  2. Select Active to activate the plan and make it available to developers deploying clusters.
    Plan pane configuration

  3. Under Name, provide a unique name for the plan.

  4. Under Description, edit the description as needed. The plan description appears in the Services Marketplace, which developers can access by using the PKS CLI.

  5. Select Enable HA Linux workers to enable high availability Linux worker clusters. A high availability Linux worker cluster consists of three Linux worker nodes.

  6. Under Master/ETCD Node Instances, select the default number of Kubernetes master/etcd nodes to provision for each cluster. You can enter 1, 3, or 5.

    Note: If you deploy a cluster with multiple master/etcd node VMs, confirm that you have sufficient hardware to handle the increased load on disk write and network traffic. For more information, see Hardware recommendations in the etcd documentation.

    In addition to meeting the hardware requirements for a multi-master cluster, we recommend configuring monitoring for etcd to monitor disk latency, network latency, and other indicators for the health of the cluster. For more information, see Configuring Telegraf in Enterprise PKS.

    WARNING: To change the number of master/etcd nodes for a plan, you must ensure that no existing clusters use the plan. Enterprise PKS does not support changing the number of master/etcd nodes for plans with existing clusters.

  7. Under Master/ETCD VM Type, select the type of VM to use for Kubernetes master/etcd nodes. For more information, including master node VM customization options, see the Master Node VM Size section of VM Sizing for Enterprise PKS Clusters.

  8. Under Master Persistent Disk Type, select the size of the persistent disk for the Kubernetes master node VM.

  9. Under Master/ETCD Availability Zones, select one or more AZs for the Kubernetes clusters deployed by Enterprise PKS. If you select more than one AZ, Enterprise PKS deploys the master VM in the first AZ and the worker VMs across the remaining AZs. If you are using multiple masters, Enterprise PKS deploys the master and worker VMs across the AZs in round-robin fashion.

  10. Under Maximum number of workers on a cluster, set the maximum number of Kubernetes worker node VMs that Enterprise PKS can deploy for each cluster. Enter any whole number in this field.
    Plan pane configuration, part two

  11. Under Worker Node Instances, specify the default number of Kubernetes worker nodes the PKS CLI provisions for each cluster. The Worker Node Instances setting must be less than, or equal to, the Maximum number of workers on a cluster setting.

    For high availability, create clusters with a minimum of three worker nodes, or two per AZ if you intend to use PersistentVolumes (PVs). For example, if you deploy across three AZs, you should have six worker nodes. For more information about PVs, see PersistentVolumes in Maintaining Workload Uptime. Provisioning a minimum of three worker nodes, or two nodes per AZ is also recommended for stateless workloads.

    For more information about creating clusters, see Creating Clusters.

    Note: Changing a plan’s Worker Node Instances setting does not alter the number of worker nodes on existing clusters. For information about scaling an existing cluster, see Scale Horizontally by Changing the Number of Worker Nodes Using the PKS CLI in Scaling Existing Clusters.

  12. Under Worker VM Type, select the type of VM to use for Kubernetes worker node VMs. For more information, including worker node VM customization options, see Worker Node VM Number and Size in VM Sizing for Enterprise PKS Clusters.

    Note: Enterprise PKS requires a Worker VM Type with an ephemeral disk size of 32 GB or more.

    Note: BOSH does not support persistent disks for Windows VMs. If specifying Worker Persistent Disk Type on a Windows worker is a requirement for you, submit feedback by sending an email to pcf-windows@pivotal.io.

  13. Under Worker Availability Zones, select one or more AZs for the Kubernetes worker nodes. Enterprise PKS deploys worker nodes equally across the AZs you select.

  14. Under Kubelet customization - system-reserved, enter resource values that Kubelet can use to reserve resources for system daemons. For example, memory=250Mi, cpu=150m. For more information about system-reserved values, see the Kubernetes documentation. Plan pane configuration, part two

  15. Under Kubelet customization - eviction-hard, enter threshold limits that Kubelet can use to evict pods when they exceed the limit. Enter limits in the format EVICTION-SIGNAL=QUANTITY. For example, memory.available=100Mi, nodefs.available=10%, nodefs.inodesFree=5%. For more information about eviction thresholds, see the Kubernetes documentation.

    WARNING: Use the Kubelet customization fields with caution. If you enter values that are invalid or that exceed the limits the system supports, Kubelet might fail to start. If Kubelet fails to start, you cannot create clusters.

  16. Under Kubelet customization - Windows pause image location, enter the location of your Windows pause image. The Kubelet customization - Windows pause image location default value of mcr.microsoft.com/k8s/core/pause:1.2.0 configures Enterprise PKS to pull the Windows pause image from the Microsoft Docker registry.

    The Microsoft Docker registry cannot be accessed from within air-gapped environments. If you want to deploy Windows pods in an air-gapped environment you must upload a Windows pause image to an accessible private registry, and configure the Kubelet customization - Windows pause image location field with the URI to this accessible Windows pause image. For more information about uploading a Windows pause image to a private registry, see Prepare a Windows Pause Image for an Internetless Environment, below.

  17. Under Errand VM Type, select the size of the VM that contains the errand. The smallest instance possible is sufficient, as the only errand running on this VM is the one that applies the Default Cluster App YAML configuration.

  18. (Optional) Under (Optional) Add-ons - Use with caution, enter additional YAML configuration to add custom workloads to each cluster in this plan. You can specify multiple files using --- as a separator. For more information, see Adding Custom Linux Workloads. Plan pane configuration

    Note: Windows in Kubernetes does not support privileged containers. See Feature Restrictions in the Kubernetes documentation for additional information.

  19. (Optional) Enable or disable one or more admission controller plugins: PodSecurityPolicy, and SecurityContextDeny. For more information see Using Admission Control Plugins for Enterprise PKS Clusters. Windows in Kubernetes does not support the DenyEscalatingExec Admission Plugin feature. See API in the Kubernetes documentation for additional information.

  20. Click Save.

Networking

To configure networking, do the following:

  1. Click Networking.
  2. Under Container Networking Interface, select Flannel. Networking pane configuration
  3. (Optional) Enter values for Kubernetes Pod Network CIDR Range and Kubernetes Service Network CIDR Range.
    • For Windows worker-based clusters the Kubernetes Service Network CIDR Range setting must remain 10.220.0.0/16. vSphere on Flannel does not support networking Windows containers. If customizing the Service Network CIDR range is a key requirement for you, submit feedback by sending an email to pcf-windows@pivotal.io.


      Networking pane configuration
      Production environments can deny direct access to public Internet services and between internal services by placing an HTTP or HTTPS proxy in the network path between Kubernetes nodes and those services.

      Configure Enterprise PKS to use your proxy and enable the following:

      • PKS API access to public Internet services and other internal services.
      • Enterprise PKS-deployed Kubernetes nodes access to public Internet services and other internal services.
      • Enterprise PKS Telemetry ability to forward Telemetry data to the CEIP and Telemetry program.

        Note: This setting does not set the proxy for running Kubernetes workloads or pods.

  4. To complete your global proxy configuration for all outgoing HTTP/HTTPS traffic from your Kubernetes clusters, perform the following steps:

    1. To proxy outgoing HTTP traffic, enter the URL of your HTTP proxy endpoint under HTTP Proxy URL. For example, http://myproxy.com:1234.
    2. (Optional) If your outgoing HTTP proxy uses basic authentication, enter the username and password in the HTTP Proxy Credentials fields.
    3. To proxy outgoing HTTPS traffic, enter the URL of your HTTP proxy endpoint under HTTPS Proxy URL. For example, http://myproxy.com:1234.

      Note: Using an HTTPS connection to the proxy server is not supported. HTTP and HTTPS proxy options can only be configured with an HTTP connection to the proxy server. You cannot populate either of the proxy URL fields with an HTTPS URL. The proxy host and port can be different for HTTP and HTTPS traffic, but the proxy protocol must be HTTP.

    4. (Optional) If your HTTPS proxy uses basic authentication, enter the username and password in the HTTPS Proxy Credentials fields.
    5. Under No Proxy, enter the comma-separated list of IP addresses that must bypass the proxy to allow for internal Enterprise PKS communication.

      The No Proxy list should include 127.0.0.1 and localhost.

      Also include the following in the No Proxy list:
      • Your Enterprise PKS environment’s CIDRs, such as the service network CIDR where your Enterprise PKS cluster is deployed, the deployment network CIDR, the node network IP block CIDR, and the pod network IP block CIDR.

      • The FQDN of any registry, such as the Harbor API FQDN, or component communicating with Enterprise PKS, using a hostname instead of an IP address.

      • Any additional IP addresses or domain names that should bypass the proxy.

        The No Proxy property for AWS accepts wildcard domains denoted by a prefixed *. or ..

        For example:
        127.0.0.1,localhost,
        *.example1.com,
        .example2.com,
        example3.com,
        198.51.100.0/24,
        203.0.113.0/24,
        192.0.2.0/24
        

        Note: By default the 10.100.0.0/8 and 10.200.0.0/8 IP address ranges, .internal, .svc,.svc.cluster.local, .svc.cluster, and your Enterprise PKS FQDN are not proxied. This allows internal Enterprise PKS communication.

        Do not use the _ character in the No Proxy field. Entering an underscore character in this field can cause upgrades to fail.

        Because some jobs in the VMs accept *. as a wildcard, while others only accept ., we recommend that you define a wildcard domain using both of them. For example, to denote example.com as a wildcard domain, add both *.example.com and example.com to the No Proxy property.

  5. Under Allow outbound internet access from Kubernetes cluster vms (IaaS-dependent), ignore the Enable outbound internet access checkbox.

  6. Click Save.

Upload the Windows Server Stemcell

  1. When prompted by Ops Manager to upload a stemcell, follow the instructions and provide your previously created vSphere stemcell 2019.15 or later for Windows Server version 2019.

Create a Windows Worker-Based Cluster

  1. To create a Windows worker-based cluster follow the steps in Creating Clusters.

Prepare a Windows Pause Image for an Air-Gapped Environment

To deploy a Windows pod, Kubelet deploys a Windows container image fetched from a Docker registry. Microsoft restricts distribution of Windows container base images and the fetched Windows container image is typically pulled from the Microsoft Docker registry.

To deploy Windows pods in an air-gapped environment you must have a Windows container image in a Docker registry accessible from your Enterprise PKS environment.

To prepare a Windows pause image for an air-gapped environment, perform the following:

  1. Create an accessible Windows Server 2019 machine in your environment.
  2. Install Docker on this Windows Server 2019 machine.
  3. Configure the machine’s Docker daemon to allow non-redistributable artifacts to be pushed to your private registry. For information about configuring your Docker daemon, see Allow push of nondistributable artifacts in the Docker documentation.
  4. Open a command line on the Windows machine.
  5. To download a Windows container image from the Microsoft Docker registry, run the following command:

    docker pull mcr.microsoft.com/k8s/core/pause:1.3.0
    
  6. To tag the Windows container image, run the following command:

    docker tag mcr.microsoft.com/k8s/core/pause:1.3.0  REGISTRY-ROOT/windows/pause:1.3.0
    

    Where REGISTRY-ROOT is your private registry’s URI.

  7. To upload the Windows container image to your accessible private registry, run the following command:

    docker push  PAUSE-IMAGE-URI
    

    Where PAUSE-IMAGE-URI is the URI to the Windows pause image in your private registry. Your pause image URI should follow the pattern: my.private.registry/windows/pause:1.3.0.

To configure Enterprise PKS to fetch your accessible Windows container image when deploying Windows pods, perform the following:

  1. Open the Enterprise PKS tile.
  2. Click the Windows worker Plan that you want to configure to use your accessible private registry.
  3. Modify the Kubelet customization - Windows pause image location property to be your pause image URI.

    For example:

    my.private.registry/windows/pause:1.3.0
    
  4. Click Save.


Please send any feedback you have to pks-feedback@pivotal.io.