LATEST VERSION: v1.1 - RELEASE NOTES
Pivotal Container Service v1.1

Preparing vSphere Before Deploying PKS

Page last updated:

Before you install Pivotal Container Service (PKS) on vSphere without NSX-T integration, you must prepare your vSphere environment. In addition to fulfilling the prerequisites specified in vSphere Prerequisites and Resource Requirements, you must create the following two service accounts in vSphere:

  • Master Node Service Account: You must create a service account for Kubernetes cluster master VMs.

  • BOSH/Ops Manager Service Account: You must create a service account for BOSH and Ops Manager.

After you create the service accounts listed above, you must grant them privileges in vSphere. Pivotal recommends configuring each service account with the least permissive privileges and unique credentials.

For the master node service account, you can create a custom role in vSphere based on your storage configuration. Kubernetes master node VMs require storage permissions to create load balancers and attach persistent disks to pods. Creating a custom role allows vSphere to apply the same privileges to all Kubernetes master node VMs in your PKS installation.

When you configure the Kubernetes Cloud Provider pane of the PKS tile, you enter the master node service account credentials in the vSphere Master Credentials fields.

For more information, see the Kubernetes Cloud Provider section of Installing PKS on vSphere.

For the BOSH/Ops Manager service account, you can apply privileges directly to the service account without creating a role. You can also apply the default VMware Administrator System Role to the service account to achieve the appropriate permission level.

Note: If your Kubernetes clusters span multiple vCenters, you must set the service account privileges correctly in each vCenter.

Step 1: Create the Master Node Service Account

  1. From the vCenter console, create a service account for Kubernetes cluster master VMs.

  2. Grant the following Virtual Machine Object privileges to the service account:

    Privilege (UI)Privilege (API)
    Virtual Machine > Configuration > AdvancedVirtualMachine.Configuration.Advanced
    Virtual Machine > Configuration > SettingsVirtualMachine.Configuration.Settings

Step 2: Grant Storage Permissions

Kubernetes master node VM service accounts require the following:

  • Read access to the folder, host, and datacenter of the cluster node VMs
  • Permission to create and delete VMs within the resource pool where PKS is deployed

Grant these permissions to the master node service account based on your storage configuration using one of the procedures below:

For more information about vSphere storage configurations, see vSphere Storage for Kubernetes in the VMware vSphere documentation.

Static Only Persistent Volume Provisioning

To configure your Kubernetes master node service account using static only Persistent Volume (PV) provisioning, do the following:

  1. Create a custom role that allows the service account to manage Kubernetes node VMs. Give this role a name. For example, manage-k8s-node-vms. For more information about custom roles in vCenter, see Create a Custom Role in the VMware vSphere documentation.

    1. Grant the following privileges at the VM Folder level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Virtual Machine > Configuration > Add existing diskVirtualMachine.Config.AddExistingDisk
      Virtual Machine > Configuration > Add new diskVirtualMachine.Config.AddNewDisk
      Virtual Machine > Configuration > Add or remove deviceVirtualMachine.Config.AddRemoveDevice
      Virtual Machine > Configuration > Remove diskVirtualMachine.Config.RemoveDisk
    2. Select the Propagate to Child Objects checkbox.
  2. (Optional) Create a custom role that allows the service account to manage Kubernetes volumes. Give this role a name. For example, manage-k8s-volumes.

    Note: This role is required if you create a Persistent Volume Claim (PVC) to bind with a statically provisioned PV, and the reclaim policy is set to delete. When the PVC is deleted, the statically provisioned PV is also deleted.

    1. Grant the following privilege at the Datastore level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Datastore > Low level file operationsDatastore.FileManagement
    2. Clear the Propagate to Child Objects checkbox.
  3. Grant the service account the existing Read-only role. This role includes the following privileges at the vCenter, Datacenter, Datastore Cluster, and Datastore Storage Folder levels:

    Privilege (UI)Privilege (API)
    Read-onlySystem.Anonymous
    System.Read
    System.View

  4. Continue to Step 3: Create the BOSH/Ops Manager Service Account.

Dynamic Persistent Volume Provisioning (with Storage Policy-Based Volume Placement)

To configure your Kubernetes master node service account using dynamic PV provisioning with storage policy-based placement, do the following:

  1. Create a custom role that allows the service account to manage Kubernetes node VMs. Give this role a name. For example, manage-k8s-node-vms. For more information about custom roles in vCenter, see Create a Custom Role in the VMware vSphere documentation.

    1. Grant the following privileges at the Cluster, Hosts, and VM Folder levels using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Virtual Machine > Resource > Assign virtual machine to resource poolResource.AssignVMToPool
      Virtual Machine > Configuration > Add existing diskVirtualMachine.Config.AddExistingDisk
      Virtual Machine > Configuration > Add new diskVirtualMachine.Config.AddNewDisk
      Virtual Machine > Configuration > Add or remove deviceVirtualMachine.Config.AddRemoveDevice
      Virtual Machine > Configuration > Remove diskVirtualMachine.Config.RemoveDisk
      Virtual Machine > Inventory > Create newVirtualMachine.Inventory.Create
      Virtual Machine > Inventory > RemoveVirtualMachine.Inventory.Delete
    2. Select the Propagate to Child Objects checkbox.
  2. Create a custom role that allows the service account to manage Kubernetes volumes. Give this role a name. For example, manage-k8s-volumes.

    1. Grant the following privilege at the Datastore level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Datastore > Allocate spaceDatastore.AllocateSpace
      Datastore > Low level file operationsDatastore.FileManagement
    2. Clear the Propagate to Child Objects checkbox.
  3. Create a custom role that allows the service account to read the Kubernetes storage profile. Give this role a name. For example, k8s-system-read-and-spbm-profile-view.

    1. Grant the following privilege at the vCenter level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Profile-driven storage viewStorageProfile.View
    2. Clear the Propagate to Child Objects checkbox.
  4. Grant the service account the existing Read-only role. This role includes the following privileges at the vCenter, Datacenter, Datastore Cluster, and Datastore Storage Folder levels:

    Privilege (UI)Privilege (API)
    Read-onlySystem.Anonymous
    System.Read
    System.View

  5. Continue to Step 3: Create the BOSH/Ops Manager Service Account.

Dynamic Volume Provisioning (without Storage Policy-Based Volume Placement)

To configure your Kubernetes master node service account using dynamic PV provisioning without storage policy-based placement, do the following:

  1. Create a custom role that allows the service account to manage Kubernetes node VMs. Give this role a name. For example, manage-k8s-node-vms. For more information about custom roles in vCenter, see Create a Custom Role in the VMware vSphere documentation.

    1. Grant the following privileges at the Cluster, Hosts, and VM Folder levels using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Virtual Machine > Configuration > Add existing diskVirtualMachine.Config.AddExistingDisk
      Virtual Machine > Configuration > Add new diskVirtualMachine.Config.AddNewDisk
      Virtual Machine > Configuration > Add or remove deviceVirtualMachine.Config.AddRemoveDevice
      Virtual Machine > Configuration > Remove diskVirtualMachine.Config.RemoveDisk
    2. Select the Propagate to Child Objects checkbox.
  2. Create a custom role that allows the service account to manage Kubernetes volumes. Give this role a name. For example, manage-k8s-volumes.

    1. Grant the following privilege at the Datastore level using either the vCenter UI or API:
      Privilege (UI)Privilege (API)
      Datastore > Allocate spaceDatastore.AllocateSpace
      Datastore > Low level file operationsDatastore.FileManagement
    2. Clear the Propagate to Child Objects checkbox.
  3. Grant the service account the existing Read-only role. This role includes the following privileges at the vCenter, Datacenter, Datastore Cluster, and Datastore Storage Folder levels:

    Privilege (UI)Privilege (API)
    Read-onlySystem.Anonymous
    System.Read
    System.View

Step 3: Create the BOSH/Ops Manager Service Account

  1. From the vCenter console, create a service account for BOSH and Ops Manager.

  2. Grant the permissions below to the BOSH and Ops Manager service account.

    Note: The privileges listed in this section describe the minimum required permissions to deploy BOSH. You can also apply the default VMware Administrator System Role to the service account to achieve the appropriate permission level, but the default role includes more privileges than those listed below.

vCenter Root Privileges

Grant the following privileges on the root vCenter server entity to the service account:

Privilege (UI)Privilege (API)
Read-onlySystem.Anonymous
System.Read
System.View
Manage custom attributesGlobal.ManageCustomFields

vCenter Datacenter Privileges

Grant the following privileges on any entities in a datacenter where you deploy PKS:

Role Object

Privilege (UI)Privilege (API)
Users inherit the Read-Only role from the vCenter root levelSystem.Anonymous
System.Read
System.View

Datastore Object

Grant the following privileges must at the datacenter level to upload and delete virtual machine files:

Privilege (UI)Privilege (API)
Allocate spaceDatastore.AllocateSpace
Browse datastoreDatastore.Browse
Low level file operationsDatastore.FileManagement
Remove fileDatastore.DeleteFile
Update virtual machine filesDatastore.UpdateVirtualMachineFiles

Folder Object

Privilege (UI)Privilege (API)
Delete folderFolder.Delete
Create folderFolder.Create
Move folderFolder.Move
Rename folderFolder.Rename

Global Object

Privilege (UI)Privilege (API)
Set custom attributeGlobal.SetCustomField
Host Object
Privilege (UI)Privilege (API)
Modify clusterHost.Inventory.EditCluster

Inventory Service Object

Privilege (UI)Privilege (API)
vSphere Tagging > Create vSphere TagInventoryService.Tagging.CreateTag
vSphere Tagging > Delete vSphere TagInventoryService.Tagging.EditTag
vSphere Tagging > Edit vSphere TagInventoryService.Tagging.DeleteTag

Network Object

Privilege (UI)Privilege (API)
Assign networkNetwork.Assign

Resource Object

Privilege (UI)Privilege (API)
Assign virtual machine to resource poolResource.AssignVMToPool
Migrate powered off virtual machineResource.ColdMigrate
Migrate powered on virtual machineResource.HotMigrate

vApp Object

Grant these privileges at the resource pool level.

Privilege (UI)Privilege (API)
ImportVApp.Import
vApp application configurationVApp.ApplicationConfig

Virtual Machine Object

Configuration

Privilege (UI)Privilege (API)
Add existing diskVirtualMachine.Config.AddExistingDisk
Add new diskVirtualMachine.Config.AddNewDisk
Add or remove deviceVirtualMachine.Config.AddRemoveDevice
AdvancedVirtualMachine.Config.AdvancedConfig
Change CPU countVirtualMachine.Config.CPUCount
Change resourceVirtualMachine.Config.Resource
Configure managedByVirtualMachine.Config.ManagedBy
Disk change trackingVirtualMachine.Config.ChangeTracking
Disk leaseVirtualMachine.Config.DiskLease
Display connection settingsVirtualMachine.Config.MksControl
Extend virtual diskVirtualMachine.Config.DiskExtend
MemoryVirtualMachine.Config.Memory
Modify device settingsVirtualMachine.Config.EditDevice
Raw deviceVirtualMachine.Config.RawDevice
Reload from pathVirtualMachine.Config.ReloadFromPath
Remove diskVirtualMachine.Config.RemoveDisk
RenameVirtualMachine.Config.Rename
Reset guest informationVirtualMachine.Config.ResetGuestInfo
Set annotationVirtualMachine.Config.Annotation
SettingsVirtualMachine.Config.Settings
Swapfile placementVirtualMachine.Config.SwapPlacement
Unlock virtual machineVirtualMachine.Config.Unlock

Guest Operations

Privilege (UI)Privilege (API)
Guest Operation Program ExecutionVirtualMachine.GuestOperations.Execute
Guest Operation ModificationsVirtualMachine.GuestOperations.Modify
Guest Operation QueriesVirtualMachine.GuestOperations.Query

Interaction

Privilege (UI)Privilege (API)
Answer questionVirtualMachine.Interact.AnswerQuestion
Configure CD mediaVirtualMachine.Interact.SetCDMedia
Console interactionVirtualMachine.Interact.ConsoleInteract
Defragment all disksVirtualMachine.Interact.DefragmentAllDisks
Device connectionVirtualMachine.Interact.DeviceConnection
Guest operating system management by VIX APIVirtualMachine.Interact.GuestControl
Power offVirtualMachine.Interact.PowerOff
Power onVirtualMachine.Interact.PowerOn
ResetVirtualMachine.Interact.Reset
SuspendVirtualMachine.Interact.Suspend
VMware Tools installVirtualMachine.Interact.ToolsInstall

Inventory

Privilege (UI)Privilege (API)
Create from existingVirtualMachine.Inventory.CreateFromExisting
Create newVirtualMachine.Inventory.Create
MoveVirtualMachine.Inventory.Move
RegisterVirtualMachine.Inventory.Register
RemoveVirtualMachine.Inventory.Delete
UnregisterVirtualMachine.Inventory.Unregister

Provisioning

Privilege (UI)Privilege (API)
Allow disk accessVirtualMachine.Provisioning.DiskRandomAccess
Allow read-only disk accessVirtualMachine.Provisioning.DiskRandomRead
Allow virtual machine downloadVirtualMachine.Provisioning.GetVmFiles
Allow virtual machine files uploadVirtualMachine.Provisioning.PutVmFiles
Clone templateVirtualMachine.Provisioning.CloneTemplate
Clone virtual machineVirtualMachine.Provisioning.Clone
CustomizeVirtualMachine.Provisioning.Customize
Deploy templateVirtualMachine.Provisioning.DeployTemplate
Mark as templateVirtualMachine.Provisioning.MarkAsTemplate
Mark as virtual machineVirtualMachine.Provisioning.MarkAsVM
Modify customization specificationVirtualMachine.Provisioning.ModifyCustSpecs
Promote disksVirtualMachine.Provisioning.PromoteDisks
Read customization specificationsVirtualMachine.Provisioning.ReadCustSpecs

Snapshot Management

Privilege (UI)Privilege (API)
Create snapshotVirtualMachine.State.CreateSnapshot
Remove snapshotVirtualMachine.State.RemoveSnapshot
Rename snapshotVirtualMachine.State.RenameSnapshot
Revert snapshotVirtualMachine.State.RevertToSnapshot

Next Steps

To install PKS on vSphere, follow the procedures in Deploying Ops Manager to vSphere.


Please send any feedback you have to pks-feedback@pivotal.io.

Create a pull request or raise an issue on the source for this page in GitHub