Page last updated:

Kubernetes Storage Considerations (TKGI Storage)

TKGI manages multiple Kubernetes clusters spaced across multiple vSphere clusters. Each kubernetes cluster’s storage requirements depend on the size of the cluster and applications deployed. Detailed guidance on these storage requirements can be found in the TKGI Documentation.

VxRails used in TA4V include vSAN, which is well applied to TKGI needs on a per vSphere cluster basis, or in the stretched cluster approach (MVP2), vSAN Fault Domains. This will provide storage for the TKGI installation in each AZ but there’s one additional need for storage: stateful apps. You can not look to vSAN to satisfy the storage needs for cross-cluster common storage.

TKGI storage options to support stateful apps include:

  • NFS, iSCSI and fiber channel datastores from external sources that are common to all clusters/host groups

vSAN does not support sharing datastores across vSphere clusters. As fail-over across availability zones is desired in the multiple cluster architecture, deploying external shared storage across those vSphere clusters / availability zones such as NFS/iSCSI/FC for both static and dynamic persistent volume provisioning for stateful workloads is recommended. Please refer to TKGI Documentation for the implementation details of persistent volumes.

In summary: You must provide a shared storage target for TKGI to use across all clusters in use for stateful apps that can not be a cluster’s vSAN, unless there is only one cluster used by TKGI, or shared storage is not drawn from vSAN, or no stateful apps are to be deployed.

Cloud Foundry Storage Considerations (TAS for VMs Storage)

VMware recommends the following capacity allocation for TAS for VMs installations:

  • For production use, at least 8 TB of available capacity, either as one 8 TB store or a number of smaller volumes adding up to 8 TB. High churn or heavily used development foundations may require significantly more storage to accommodate new code and buildpacks. A “development” PaaS installation will see siginifcantly higher storage consumption than a “production” one due to rapid app prototyping and blob creation.
  • For small installations without many tiles, 4-6 TB may suffice.
  • The primary consumer of storage is the NFS/WebDAV blobstore included in TAS for VMs. This job can be externalized to a hardware or emulated software S3-compatible blob store (see below).
  • Do not use the included TAS for VMs blob store VM in any production PaaS Foundation.

TAS for VMs Blobstore Options

Elastic Cloud Storage

Dell EMC ECS object storage features a flexible software-defined architecture that provides the simplistic benefits of the public cloud without the risk, compliance, and data sovereignty concerns.

TA4V can use the ECS platform as its blobstore location to store application code, buildpacks, and applications. The blobstore uses the S3 protocol and can be managed by ECS nodes. The ECS platform provides software-defined object storage that is designed for modern cloud-scale storage requirements.

With Dell EMC ECS, VMware recommends that you use unversioned buckets for file storage by clearing the Use versioning for backup and restore checkbox in the File Storage pane of the TAS for VMs tile. For information, see Step 3: Configure PAS File Storage in this Dell EMC ECS whitepaper. You can use mirroring as an alternative to versioning.

Virtustream Storage Cloud

Virtustream Storage Cloud is a hyper-scale, enterprise-class, public cloud object storage platform, built for resiliency and performance. Virtustream provides a unique value proposition with enterprise-class off-premises primary and backup storage.

Pivotal Internal Blobstore

Pivotal provides an internal WebDAV blobstore that TAS for VMs uses to store persistent data. The TAS for VMs internal blobstore is deployed as a single virtual machine in TAS for VMs. It does not offer high availability but can be backed up using BOSH Backup and Restore (BBR). It is designed for small, non-production deployments. For TAS for VMs production-level deployments, Dell EMC and VMware recommend using an external S3-compatible blobstore.