Using Kubernetes Profiles

Page last updated:

This topic describes how VMware Enterprise PKS administrators and cluster managers create and use Kubernetes profiles for Kubernetes clusters provisioned by Enterprise PKS.

This topic also lists verified use cases for Kubernetes profiles on Enterprise PKS.

Overview

Kubernetes profiles enable cluster administrators and cluster managers to customize Kubernetes component settings for any clusters that they provision.

To use Kubernetes profiles, Enterprise PKS users:

  1. Create JSON configuration files that set configuration options for any Kubernetes components, such as kube-apiserver on the control plane or the kubelet on each node.
  2. Use the PKS CLI to create the Kubernetes profile in Enterprise PKS.
  3. Use the PKS CLI to apply the profile to clusters.

Verified uses for Kubernetes profiles include encrypting secrets in an etcd database, adding an OIDC provider, and using a ResourceQuota admission control plugin.

Who Creates and Manages Kubernetes Profiles

Users with the pks.clusters.admin or pks.clusters.manage roles can create and use Kubernetes profiles.

If user with the pks.clusters.admin-read-only role attempts to create a Kubernetes profile, they see the following error:

You do not have enough privileges to perform this action. Please contact the PKS administrator.

Validated vs Experimental Customizations

A Kubernetes profile configures settings for Kubernetes components in two JSON code blocks, customizations and experimental_customizations. See Kubernetes Profile Format for details. The code blocks differ as follows:

  • customizations block

    • PKS checks the validity of configurations in this block. If you run pks create-k8s-profile on a profile with invalid configurations in customizations, the command returns an error.
    • The PKS team supports clusters configured with tested, validated parameters in this block.
  • experimental_customizations block

    Warning: Experimental customizations are not validated or supported.

    • PKS imposes no restrictions on the contents of this block.
    • Configurations in this block are neither tested nor supported.
    • If a customer wants to use an unsupported configuration, they should contact the PKS team with the parameters that they want tested, validated, and supported.

“k8s” to “kubernetes” Alias in PKS CLI

In the PKS CLI, all commands that include k8s-profile are aliased to also use kubernetes-profile. For example, the pks k8s-profiles and pks kubernetes-profiles commands are equivalent.

For brevity, this documentation uses the k8s- versions.

Create a Kubernetes Profile

To create a Kubernetes profile in Enterprise PKS, you:

  1. Define a Kubernetes profile in a JSON configuration file, following the Kubernetes Profile Format below.

  2. Use the PKS CLI to define the Kubernetes profile within Enterprise PKS, as described in The create-network-profile Command, below.

Kubernetes Profile Format

To create a Kubernetes profile, you must first define it as a JSON file that specifies network parameters, listed in Kubernetes Profile Parameters below.

Here is the basic structure of a Kubernetes profile.

{
    "name": "my-profile-1",
    "description": "My profile description",
    "customizations": [
        {
           "component": "k8s-component-name",
           "arguments": {
                "key": "value,value,value"
            },
            "file-arguments": {
                "key": "local file path"
            }

        }
    ],
    "experimental_customizations": [
        {

        }
    ]
}

Note: This example Kubernetes profile is for illustration purposes only. It is not intended to be used as a template for Kubernetes profile definition.

Kubernetes Profile Parameters

The Kubernetes profile JSON can include the following parameters:

Parameter Type Description
name* String Name of the Kubernetes profile.
description String Description of the Kubernetes profile.
customizations* Map A block that defines supported, validated
K8s options using the component, arguments,
and file-arguments parameters below.
experimental_
customizations
Map A block that defines unsupported, unvalidated
K8s options using the component, arguments,
and file-arguments parameters below.
component* String The name of a K8s component,
e.g. kubelet, kube-apiserver.
arguments* Map Parameters for each component, as one or more Key:Value pairs.
Multiple values must be separated by commas, without spaces.
file-arguments* Map Parameters whose values are stored as files
on the local machine.

*Parameters marked with an * are mandatory.

Note: If you specify the same parameter in both customizations and experimental_customizations, the one in customization takes precedence.

The create-k8s-profile Command

After defining a Kubernetes profile in a JSON file as described in Kubernetes Profile Format, a cluster administrator or manager creates the Kubernetes profile by running the following PKS CLI command:

pks create-k8s-profile PATH-TO-YOUR-KUBERNETES-PROFILE-CONFIGURATION

Where PATH-TO-YOUR-KUBERNETES-PROFILE-CONFIGURATION is the path to the JSON file you created when defining the Kubernetes profile.

For example:

cat profile3-docs.json
{
    "name": "my-profile3",
    "description": "My profile description",
    "customizations": [
        {
            "component": "kube-apiserver",
            "arguments": {
                "service-node-port-range": "30000-40000"
            }
        }
    ],
    "experimental_customizations": [
        {
            "component": "kubelet",
            "arguments": {
                "maximum-dead-containers": "1000",
                "feature-gates": "APIListChunking=true,AttachVolumeLimit=false"
            }
        }
    ]
}
user ~/workspace: ./pks create-k8s-profile k8s-profile3.json 
Kubernetes profile my-profile3 successfully created 

Manage Kubernetes Profiles

Enterprise PKS cluster administrators and managers can perform the following operations on Kubernetes profiles and the clusters that use them.

List Kubernetes Profiles

To list available Kubernetes profiles, run the following command:

pks k8s-profiles

For example:

$ pks k8s-profiles
K8s-profile        Owner     Created Date
Basic-k8s-profile  Alana     Tue, 05 Nov 2019 16:28:15 PST

The command output differs by user role: - pks.cluster.admin see a list of Kubernetes profiles that all users created - pks.cluster.manage see a list of only the Kubernetes profiles that they created

Delete a Kubernetes Profile

To delete a Kubernetes profile, run the following command:

pks delete-k8s-profile KUBERNETES-PROFILE-NAME

Where KUBERNETES-PROFILE-NAME is the name of the Kubernetes profile you want to delete.

For example:

$ user ~/workspace: ./pks delete-k8s-profile my-profile3 
Are you sure you want to delete the kubernetes profile my-profile3? (y/n): y
Deletion of my-profile3 completed

Note: You cannot delete a Kubernetes profile that is in use. Before deleting a Kubernetes profile, you must disassociate it from all clusters or delete all clusters it is associated with.

Both pks.clusters.admin and pks.clusters.manage users can delete Kubernetes profiles. If a pks.clusters.admin-read-only user attempts to delete a Kubernetes profile, they see the following error:

You do not have enough privileges to perform this action. Please contact the PKS administrator.

View Kubernetes Profile Details

To view details of a Kubernetes profile, run the following command:

pks k8s-profile KUBERNETES-PROFILE-NAME

Where KUBERNETES-PROFILE-NAME is the name of the Kubernetes profile you want to view.

For example:

pks k8s-profile Basic-k8s-profile
Name:                    Basic-k8s-profile
Owner:                   Alana
Created Date             Tue, 05 Nov 2019 16:28:15 PST
Description:             Kubernetes profile for customer A
….
<KEY> :                  <VALUE>
  • Users with the pks.cluster.admin can view the details of any Kubernetes profile; users with the pks.cluster.manage role can view details of Kubernetes profiles that they created.
  • Once you create or update a cluster with an encryption profile, you cannot assign any other Kubernetes profiles to that cluster. Because decryption is not straightforward, applying another profile can have nondeterministic outcome.

Create a Cluster with a Kubernetes Profile

You can assign a Kubernetes profile to a Kubernetes cluster at the time of cluster creation.

To create an Enterprise PKS-provisioned Kubernetes cluster with a Kubernetes profile, run the following command:

pks create-cluster CLUSTER-NAME --external-hostname HOSTNAME --plan PLAN-NAME --kubernetes-profile KUBERNETES-PROFILE-NAME

Where:

  • CLUSTER-NAME is a unique name for your cluster.
  • HOSTNAME is your external hostname used for accessing the Kubernetes API.
  • PLAN-NAME is the name of the Enterprise PKS plan you want to use for your cluster.
  • KUBERNETES-PROFILE-NAME is the name of the Kubernetes profile you want to use for your cluster.

Assign a Kubernetes Profile to an Existing Cluster

PKS supports changing the Kubernetes profile for an already created cluster. You can use this procedure to:

  • assign a Kubernetes profile to a cluster that does not have one, or
  • change a cluster’s existing profile to a new one

This is the procedure to change a cluster’s Kubernetes profile:

  1. Do one of the following

    • Choose a new Kubernetes profile for the cluster. See List Kubernetes profiles.
    • Define and create a new Kubernetes profile as described in Create a Kubernetes Profile.
      • The name of the new Kubernetes profile must be unique and different from the previously assigned Kubernetes profile.
  2. Run the following command to update the cluster with the new Kubernetes profile:

    pks update-cluster CLUSTER-NAME --kubernetes-profile NEW-KUBERNETES-PROFILE-NAME
    

    Where:

    • CLUSTER-NAME is the name of the existing Kubernetes cluster
    • NEW-KUBERNETES-PROFILE-NAME is the name of the new Kubernetes profile you want to apply to the cluster.

Kubernetes Profile Use Cases

Kubernetes profiles let you customize Kubernetes configuration parameters at the time of cluster creation. Use cases for Kubernetes profiles include:

Topic Description
Encrypt Secrets in an etcd Database Use an encryption provider to encrypt secrets in a cluster’s etcd database.
Admission Control: ResourceQuota Use the ResourceQuota admission control plugin to restrict incoming requests by resource usage.
Set Service Node Port Range Use service-node-port-range to specify an IP range for for NodePort services.
Add an OIDC Provider Customize a cluster’s OIDC provider by deploying a dex connector or other OIDC provider to its pod.
Restrict Request Header Names Set requestheader-allowed-names for Apiserver client authentication.

Admission Control: ResourceQuota

To create a Kubernetes profile that includes the ResourceQuota admission control plugin:

  • Follow the Create a Kubernetes Profile instructions.
  • Include the following customizations in your profile configuration file:

      "customizations": [
          {
              "component": "kube-apiserver",
              "arguments": {
                  "enable-admission-plugins": PLUGINS-LIST
              }
          }
      ],
    

    Where PLUGINS-LIST is one of the following:

    • The string "ResourceQuota"
    • A comma-delimited string list of validated plugins that includes ResourceQuota.

For more information, see ResourceQuota in the Kubernetes documentation.

Set Service Node Port Range

To create a Kubernetes profile that uses service-node-port-range for NodePort type services:

  • Follow the Create a Kubernetes Profile instructions.
  • Include the following customizations in your profile configuration file:

      "customizations": [
          {
              "component": "kube-apiserver",
              "arguments": {
                  "service-node-port-range": PORT-RANGE
              }
          }
      ],
    

    Where PORT-RANGE is A CIDR notation IP range from which to assign service cluster IPs, such as 30000-40000.

If the specified PORT-RANGE is not valid, the pks create-k8s-profile command returns an error invalid value for service-node-port-range.

For more information, see Type NodePort in the Kubernetes documentation.

Restrict Request Header Names

To create a Kubernetes profile that uses requestheader-allowed-names for Apiserver client authentication:

  • Follow the Create a Kubernetes Profile instructions.
  • Include the following customizations in your profile configuration file:

      "customizations": [
          {
              "component": "kube-apiserver",
              "arguments": {
                  "requestheader-allowed-names": COMMON-NAMES
              }
          }
      ],
    

    Where COMMON-NAMES is a string list of valid Common Name values in the signed client certificate, such as "cn1.com,c2.com".

For more information, see Kubernetes Apiserver Client Authentication in the Kubernetes documentation.


Please send any feedback you have to pks-feedback@pivotal.io.