Preparing vSphere Before Deploying PKS

Page last updated:

This topic describes how to prepare your vSphere environment before deploying Pivotal Container Service (PKS).

Overview

Before you install PKS on vSphere without NSX-T integration, you must prepare your vSphere environment by creating the required service accounts and configuring DNS for the PKS API endpoint.

You must create the following service accounts in vSphere:

  • Master Node Service Account for the Kubernetes master node VMs.
  • BOSH/Ops Manager Service Account for BOSH Director operations.

WARNING: The PKS Master Node and BOSH/Ops Manager service accounts must be two separate accounts.

After creating the Master Node and BOSH/Ops Manager service accounts you must grant the accounts privileges in vSphere:

  • Master Node Service Account: Kubernetes master node VMs require storage permissions to create load balancers and attach persistent disks to pods. Creating a custom role for this service account allows vSphere to apply the same privileges to all Kubernetes master node VMs in your PKS installation.

  • BOSH/Ops Manager Service Account: BOSH Director requires permissions to create VMs. You can apply privileges directly to this service account without creating a role. You can also apply the default VMware Administrator System Role to this service account to achieve the appropriate permission level.

Pivotal recommends configuring each service account with the least permissive privileges and unique credentials.

Note: If your Kubernetes clusters span multiple vCenters, you must set the service account privileges correctly in each vCenter.

To prepare your vSphere environment, do the following:

  1. Create the Master Node Service Account
  2. Grant Storage Permissions
  3. Create the BOSH/Ops Manager Service Account
  4. Grant Permissions to the BOSH/Ops Manager Service Account
  5. Configure DNS for the PKS API

Prerequisites

Before you prepare your vSphere environment, you must fulfill the prerequisites in vSphere Prerequisites and Resource Requirements.

Create the Master Node Service Account

  1. From the vCenter console, create a service account for Kubernetes cluster master VMs.

  2. Grant the following Virtual Machine Object privileges to the service account:

    Privilege (UI)Privilege (API)
    Virtual Machine > Configuration > AdvancedVirtualMachine.Configuration.Advanced
    Virtual Machine > Configuration > SettingsVirtualMachine.Configuration.Settings

Grant Storage Permissions

Kubernetes master node VM service accounts require the following:

  • Read access to the folder, host, and datacenter of the cluster node VMs
  • Permission to create and delete VMs within the resource pool where PKS is deployed

Grant these permissions to the master node service account based on your storage configuration using one of the procedures below:

For more information about vSphere storage configurations, see vSphere Storage for Kubernetes in the VMware vSphere documentation.

Static Only Persistent Volume Provisioning

To configure your Kubernetes master node service account using static only Persistent Volume (PV) provisioning, do the following:

  1. Create a custom role that allows the service account to manage Kubernetes node VMs. Give this role a name. For example, manage-k8s-node-vms. For more information about custom roles in vCenter, see Create a Custom Role in the VMware vSphere documentation.

    1. Grant the following privileges at the VM Folder level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Virtual Machine > Configuration > Add existing diskVirtualMachine.Config.AddExistingDisk
      Virtual Machine > Configuration > Add new diskVirtualMachine.Config.AddNewDisk
      Virtual Machine > Configuration > Add or remove deviceVirtualMachine.Config.AddRemoveDevice
      Virtual Machine > Configuration > Remove diskVirtualMachine.Config.RemoveDisk
    2. Select the Propagate to Child Objects checkbox.
  2. (Optional) Create a custom role that allows the service account to manage Kubernetes volumes. Give this role a name. For example, manage-k8s-volumes.

    Note: This role is required if you create a Persistent Volume Claim (PVC) to bind with a statically provisioned PV, and the reclaim policy is set to delete. When the PVC is deleted, the statically provisioned PV is also deleted.

    1. Grant the following privilege at the Datastore level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Datastore > Low level file operationsDatastore.FileManagement
    2. Clear the Propagate to Child Objects checkbox.
  3. Grant the service account the existing Read-only role. This role includes the following privileges at the vCenter, Datacenter, Datastore Cluster, and Datastore Storage Folder levels:

    Privilege (UI)Privilege (API)
    Read-onlySystem.Anonymous
    System.Read
    System.View

  4. Continue to Create the BOSH/Ops Manager Service Account.

Dynamic Persistent Volume Provisioning (with Storage Policy-Based Volume Placement)

To configure your Kubernetes master node service account using dynamic PV provisioning with storage policy-based placement, do the following:

  1. Create a custom role that allows the service account to manage Kubernetes node VMs. Give this role a name. For example, manage-k8s-node-vms. For more information about custom roles in vCenter, see Create a Custom Role in the VMware vSphere documentation.

    1. Grant the following privileges at the Cluster, Hosts, and VM Folder levels using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Virtual Machine > Resource > Assign virtual machine to resource poolResource.AssignVMToPool
      Virtual Machine > Configuration > Add existing diskVirtualMachine.Config.AddExistingDisk
      Virtual Machine > Configuration > Add new diskVirtualMachine.Config.AddNewDisk
      Virtual Machine > Configuration > Add or remove deviceVirtualMachine.Config.AddRemoveDevice
      Virtual Machine > Configuration > Remove diskVirtualMachine.Config.RemoveDisk
      Virtual Machine > Inventory > Create newVirtualMachine.Inventory.Create
      Virtual Machine > Inventory > RemoveVirtualMachine.Inventory.Delete
    2. Select the Propagate to Child Objects checkbox.
  2. Create a custom role that allows the service account to manage Kubernetes volumes. Give this role a name. For example, manage-k8s-volumes.

    1. Grant the following privilege at the Datastore level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Datastore > Allocate spaceDatastore.AllocateSpace
      Datastore > Low level file operationsDatastore.FileManagement
    2. Clear the Propagate to Child Objects checkbox.
  3. Create a custom role that allows the service account to read the Kubernetes storage profile. Give this role a name. For example, k8s-system-read-and-spbm-profile-view.

    1. Grant the following privilege at the vCenter level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Profile-driven storage viewStorageProfile.View
    2. Clear the Propagate to Child Objects checkbox.
  4. Grant the service account the existing Read-only role. This role includes the following privileges at the vCenter, Datacenter, Datastore Cluster, and Datastore Storage Folder levels:

    Privilege (UI)Privilege (API)
    Read-onlySystem.Anonymous
    System.Read
    System.View

  5. Continue to Create the BOSH/Ops Manager Service Account.

Dynamic Volume Provisioning (without Storage Policy-Based Volume Placement)

To configure your Kubernetes master node service account using dynamic PV provisioning without storage policy-based placement, do the following:

  1. Create a custom role that allows the service account to manage Kubernetes node VMs. Give this role a name. For example, manage-k8s-node-vms. For more information about custom roles in vCenter, see Create a Custom Role in the VMware vSphere documentation.

    1. Grant the following privileges at the Cluster, Hosts, and VM Folder levels using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Virtual Machine > Configuration > Add existing diskVirtualMachine.Config.AddExistingDisk
      Virtual Machine > Configuration > Add new diskVirtualMachine.Config.AddNewDisk
      Virtual Machine > Configuration > Add or remove deviceVirtualMachine.Config.AddRemoveDevice
      Virtual Machine > Configuration > Remove diskVirtualMachine.Config.RemoveDisk
    2. Select the Propagate to Child Objects checkbox.
  2. Create a custom role that allows the service account to manage Kubernetes volumes. Give this role a name. For example, manage-k8s-volumes.

    1. Grant the following privilege at the Datastore level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Datastore > Allocate spaceDatastore.AllocateSpace
      Datastore > Low level file operationsDatastore.FileManagement
    2. Clear the Propagate to Child Objects checkbox.
  3. Grant the service account the existing Read-only role. This role includes the following privileges at the vCenter, Datacenter, Datastore Cluster, and Datastore Storage Folder levels:

    Privilege (UI)Privilege (API)
    Read-onlySystem.Anonymous
    System.Read
    System.View

Create the BOSH/Ops Manager Service Account

  1. From the vCenter console, create the BOSH/Ops Manager Service Account.
  2. If you are deploying both PAS and PKS within the same vSphere environment, create an additional BOSH/Ops Manager Service Account, so that there is one account for PAS and a separate account for PKS.

Grant Permissions to the BOSH/Ops Manager Service Account

There are two options for granting permissions to the BOSH/Ops Manager Service Account(s):

  • Grant minimal permissions. Grant each BOSH/Ops Manager Service Account the minimum required permissions as described in vSphere Service Account Requirements.
  • Grant Administrator Role permissions. Apply the default VMware Administrator Role to each BOSH/Ops Manager Service Account as described in vCenter Server System Roles .

    Warning: Applying the VMware Administrator Role to the BOSH/Ops Manager Service Account grants the account more privileges than are required. For optimal security always use the least privileged account.

Configure DNS for the PKS API

Navigate to your DNS provider and create an entry for a fully qualified domain name (FQDN) within your system domain. For example, api.pks.example.com.

When you configure the PKS tile, enter this FQDN in the PKS API pane.

After you deploy PKS, you map the IP address of the PKS API to this FQDN. You can then use this FQDN to access the PKS API from your local system.

Next Steps

After you complete the instructions provided in this topic, install one of the following:

  • Pivotal Ops Manager v2.3.1 or later
  • Pivotal Ops Manager v2.4.x

Note: You use Ops Manager to install and configure PKS. Each version of Ops Manager supports multiple versions of PKS. To confirm that your Ops Manager version supports the version of PKS that you install, see PKS Release Notes.

To install an Ops Manager version that is compatible with the PKS version you intend to use, follow the instructions in the corresponding version of the Ops Manager documentation.

Version
Ops Manager v2.3
Ops Manager v2.4

Please send any feedback you have to pks-feedback@pivotal.io.