Manually Installing the vSphere CSI Driver

Page last updated:

This topic explains how to manually integrate Cloud Native Storage (CNS) with VMware Tanzu Kubernetes Grid Integrated Edition on vSphere using the vSphere Container Storage Interface (CSI) driver. This integration enables TKGI clusters to use external container storage.

This topic provides procedures for installing CSI on a TKGI cluster, verifying the installation and resizing PersistentVolumes.

Note: CSI can only be installed on a Linux TKGI cluster.

Overview

Cloud Native Storage (CNS) provides comprehensive data management for stateful, containerized apps, enabling apps to survive restarts and outages. Stateful containers can use vSphere storage primitives such as standard volume, PersistentVolume, and dynamic provisioning, independent of VM and container lifecycle.

vSphere storage backs the volumes, and you can set a storage policy directly on the volumes. After you create the volumes, you can use the vSphere client to review the volumes and their backing virtual disks, and monitor their storage policy compliance.

To create PersistentVolumes using CNS on vSphere, see:


To resize a PersistentVolume, see:


To upgrade the vSphere CSI driver, see:


For more information, see Getting Started with VMware Cloud Native Storage.

Prerequisites for Using the vSphere CSI Driver with TKGI

Before you use the vSphere CSI Driver, your environment must meet the following requirements:

  • A Linux TKGI cluster. CSI on a Windows TKGI cluster is not supported.
  • The vSphere CSI Driver must be a compatible version: The vSphere CSI Driver you use must be compatible with the Kubernetes version installed with TKGI. For more information see Compatibility Matrix for vSphere Container Storage Plug-in in the vSphere CSI Driver documentation.

    Note: VMware recommends using vSphere CSI Driver v2.3 with TKGI v1.12.

    For instructions on how to upgrade the vSphere CSI Driver version, see Upgrade the vSphere CSI Driver on a TKGI Cluster below.
  • The vSphere CSI Driver requirements have been met: For more information, see Prerequisites for Installing the vSphere Container Storage Plug-in.
  • Plan Configuration: The Allow Privileged setting must be enabled in the TKGI tile for the plans you use with the vSphere CSI Driver. To enable this setting, see Installing TKGI on vSphere.
  • The following must have access to your vCenter:
    • Cluster master nodes.
    • Cluster worker nodes. You must enable this access in order for vSphere CSI Driver components on worker nodes to provision disks.
    • All Pods that are running vSphere CSI Driver components.
  • vSphere v7.0 or vSphere v6.7U3: vSphere must be vSphere v7.0 or later or vSphere v6.7U3 or later. For more information, see vSphere definition.

Install the vSphere CSI Driver on a TKGI Cluster

This section provides procedures for manually installing CSI on a TKGI cluster. Installing CSI on a TKGI cluster requires a number of steps:

Create a TKGI Cluster

To create a TKGI-provisioned Kubernetes cluster:

  1. Create a cluster using TKGI create-cluster:

    tkgi create-cluster CLUSTER-NAME --external-hostname EXTERNAL-HOSTNAME  \
    --plan PLAN-NAME --num-nodes 3 --network-profile single-tier-profile
    

    Where:

    • CLUSTER-NAME is the name you want to apply to your new cluster.

      Note: Use only lowercacse characters when naming your cluster if you manage your clusters with Tanzu Mission Control (TMC). Clusters with names that include an uppercase character cannot be attached to TMC.

    • EXTERNAL-HOSTNAME is the address from which to access Kubernetes API.
    • PLAN-NAME is the name of the plan to base the new cluster on. To provide the ability to resize the cluster’s PersistentVolume chose a plan where Allow Privileged is selected. For more information, see Plans in Installing Tanzu Kubernetes Grid Integrated Edition on vSphere with NSX-T.
    > tkgi create-cluster tkgi-cluster-5-shared-t1 --external-hostname tkgi-cluster-5-shared-t1 --plan large --num-nodes 3 --network-profile single-tier-profile
    
  2. Confirm that all of the VMs in the Kubernetes cluster have hardware compatible with VMware version 15.

    VM Instance Summary pane in vCenter

Customize Your CSI Driver Configuration

You must customize your vSphere CSI Driver configuration files before manually deploying the vSphere CSI Driver.

Note: For vSphere CSI Driver v2.3.0 and later a single vsphere-csi-driver.yaml manifest file is used instead of the separate vsphere-csi-controller-deployment.yaml and vsphere-csi-node-ds.yaml manifest files used in prior versions.

To download and customize the vSphere CSI Driver configuration files:

  1. Download the v2.3.0 or later vSphere CSI Driver manifest files from https://github.com/kubernetes-sigs/vsphere-csi-driver/tree/v2.3.0/manifests/vanilla in the kubernetes-sigs/vsphere-csi-driver GitHub repository.

  2. Modify the vSphere CSI Driver vsphere-csi-driver.yaml configuration file:

    1. Remove the following nodeselector from the vsphere-csi-controller Deployment section:

        nodeSelector:
          node-role.kubernetes.io/master: ""
      
    2. In the vsphere-csi-node DaemonSet section, replace all occurrences of /var/lib/kubelet with /var/vcap/data/kubelet.

Install the vSphere CSI Driver

To install the vSphere CSI Driver:

  1. Complete the pre-installation configuration steps in Deploy the vSphere Container Storage Plug-in on a Native Kubernetes Cluster:
    1. Create vmware-system-csi Namespace for vSphere Container Storage Plug-in
    2. Taint Kubernetes Primary Node for the vSphere Container Storage Plug-in Installation
    3. Create a Kubernetes Secret for vSphere Container Storage Plug-in
  2. Make copies of your customized csi-vsphere.conf configuration file and customize a copy for each of your Kubernetes clusters:
    • Configure the cluster-id value. If you deploy the vSphere CSI Driver to multiple Kubernetes clusters, you must use a unique cluster-id value in each cluster. You can use the cluster’s UUID as the cluster-id value. The cluster’s UUID is in the information returned by running pks cluster CLUSTER-NAME.
  3. Complete the installation and verification steps:
    1. Install vSphere Container Storage Plug-in

Create a vSphere Storage Class

To create a vSphere Storage Class:

  1. Open vCenter.
  2. Open the vSAN Datastore Summary pane.

    vSAN Datastore Summary pane in vCenter

  3. Determine the datastoreurl value for your Datastore.

  4. Create the following YAML:

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: demo-sts-storageclass
      annotations:
          storageclass.kubernetes.io/is-default-class: "true"
    provisioner: csi.vsphere.vmware.com
    allowVolumeExpansion: ALLOW-EXPANSION
    parameters:
      datastoreurl: "DATASTORE-URL"
    

    Where:

    • ALLOW-EXPANSION defines whether the cluster’s PersistentVolume size is either resizable or static. Set to true for resizable and false for static size.
    • DATASTORE-URL is the URL to your Datastore. For a non-vSAN datastore, the datastoreurl value looks like ds:///vmfs/volumes/5e66e525-8e46bd39-c184-005056ae28de/.

    For example:

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: demo-sts-storageclass
      annotations:
          storageclass.kubernetes.io/is-default-class: "true"
    provisioner: csi.vsphere.vmware.com
    allowVolumeExpansion: true
    parameters:
      datastoreurl: "ds:///vmfs/volumes/vsan:52d8eb4842dbf493-41523be9cd4ff7b7/"
    

Resize a PersistentVolume

PersistentVolumes created using the CNS v2.3.0 StorageClass procedures above can be resized. CNS supports PersistentVolume resizing of only off-line volumes.

To resize a PersistentVolume:

  1. Stop all pods connected to the PersistentVolume.
  2. Wait until all pods have stopped. To determine if the pods have stopped running:

    kubectl  get statefulset -n NAMESPACE
    

    Where NAMESPACE is the namespace the PersistentVolume is in.

    For example:

    > kubectl  get statefulset -n csi-namespace
    NAME          READY   AGE
    postgres-wl   0/0     15m
    > kubectl get pods -n csi-namespace No resources found in csi-resize-test namespace.
    > kubectl get pvc -n csi-namespace NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE postgredb-postgres-wl-0 Bound pvc-c22811a5-0650-4492-a7f0-0d21a4971c5a 1Gi RWO postgres-sc-csi-resize 15m

  3. Wait until all pods have stopped.

  4. Edit the PersistentVolume:

    kubectl edit pvc STATEFUL-SET-NAME -n NAMESPACE
    

    Where:

    • STATEFUL-SET-NAME is the name of the Pod with the PersistentVolume you are resizing.
    • NAMESPACE is the namespace the Pod is in.

    For example:

    > kubectl  edit pvc postgredb-postgres-wl-0 -n csi-namespace
    
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
      storage: 1Gi
    storageClassName: postgres-sc-csi-resize
    volumeMode: Filesystem
    volumeName: pvc-c22811a5-0650-4492-a7f0-0d21a4971c5a
    
  5. Modify the storage size.

    Note: Kubernetes does not support shrinking of PersistentVolumes.

  6. Restart the Pod using kubectl scale:

    kubectl scale statefulset STATEFUL-SET-NAME  -n NAMESPACE --replicas=1
    

    Where:

    • STATEFUL-SET-NAME is the name of the Pod with the PersistentVolume you are resizing.
    • NAMESPACE is the namespace the Pod is in.

    For example:

    > kubectl scale statefulset postgres-wl  -n csi-namespace --replicas=1
    
  7. Confirm the Pod is running:

    kubectl get statefulset -n NAMESPACE
    kubectl get pods -n NAMESPACE
    kubectl get pvc -n NAMESPACE
    kubectl get pv | grep STATEFUL-SET-NAME
    

    Where:

    • STATEFUL-SET-NAME is the name of the Pod with the PersistentVolume you are resizing.
    • NAMESPACE is the namespace the Pod is in.

    For example:

    > kubectl  get statefulset -n csi-namespace
    NAME          READY   AGE
    postgres-wl   1/1     24m  
    
    > kubectl get pods -n csi-namespace
    NAME            READY   STATUS    RESTARTS   AGE
    postgres-wl-0   2/2     Running   0          49s  
    
    > kubectl  get pvc -n csi-namespace
    NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS             AGE
    postgredb-postgres-wl-0   Bound    pvc-c22811a5-0650-4492-a7f0-0d21a4971c5a   5Gi        RWO            postgres-sc-csi-resize   24m  
    
    > kubectl  get pv | grep test
    pvc-c22811a5-0650-4492-a7f0-0d21a4971c5a   5Gi        RWO            Delete           Bound    csi-namespace/postgredb-postgres-wl-0   postgres-sc-csi-resize            23m
    
  8. Confirm the Pod’s PersistentVolume has been resized:

    kubectl  exec -it STATEFUL-SET-NAME -n NAMESPACE -- sh
    

    Where:

    • STATEFUL-SET-NAME is the name of the Pod with the PersistentVolume you are resizing.
    • NAMESPACE is the namespace the Pod is in.

    For example:

    > kubectl  exec -it postgres-wl-0 -n csi-namespace -- sh
    # df -h
    Filesystem      Size  Used Avail Use% Mounted on
    overlay          30G  1.8G   27G   7% /
    tmpfs            64M     0   64M   0% /dev
    tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
    /dev/sdb2        48G  3.1G   42G   7% /etc/hosts
    /dev/sdc1        30G  1.8G   27G   7% /etc/hostname
    shm              64M  8.0K   64M   1% /dev/shm
    /dev/sdd        4.9G  847M  3.9G  18% /var/lib/postgresql/data
    tmpfs           7.9G   12K  7.9G   1% /run/secrets/kubernetes.io/serviceaccount
    
    

For more information about resizing PersistentVolumes, see Resizing Persistent Volumes using Kubernetes in the Kubernetes documentation.

Upgrade the vSphere CSI Driver on a TKGI Cluster

If your TKGI cluster is using a vSphere CSI Driver earlier than v2.3.0, you must upgrade the CSI driver on the cluster.

To upgrade the vSphere CSI Driver installed on your clusters:

  1. Review the prerequisites for the vSphere CSI Driver version you are installing. For more information, see Prerequisites for using the vSphere CSI Driver with TKGI above.
  2. Review the configuration requirements for the vSphere CSI Driver version you are installing. For more information, see Install the vSphere CSI Driver above.
  3. Follow the steps in Upgrade vSphere Container Storage Plug-in.

Please send any feedback you have to pks-feedback@pivotal.io.