Release Notes

Page last updated:

This topic contains release notes for Tanzu Kubernetes Grid Integrated Edition (TKGI) v1.11.

Warning: Before installing or upgrading to Tanzu Kubernetes Grid Integrated Edition v1.11, review the Breaking Changes below.

TKGI v1.11.4

Release Date: September 16, 2021

Product Snapshot

Release Details
Version v1.11.4
Release date September 16, 2021
Component Version
Antrea v0.11.4
cAdvisor v0.39.1
CSI Driver for vSphere v2.2.1
CoreDNS v1.7.0+vmware.15*
Docker Linux: v20.10.7
Windows: v19.03.17
etcd v3.4.13
Harbor v2.3.2*
Kubernetes v1.20.10*
Metrics Server v0.3.6
NCP v3.1.2.3*
Percona XtraDB Cluster (PXC) v0.37.0
UAA v74.5.25*
Velero v1.6.0
VMware Cloud Foundation (VCF) v4.3, v4.2.1**
Wavefront Wavefront Collector: v1.3.4
Wavefront Proxy: v9.7
Compatibilities Versions
Ops Manager See VMware Tanzu Network.
NSX-T See VMware Product Interoperability Matrices.
vSphere
Windows stemcells v2019.37 or later
Xenial stemcells See VMware Tanzu Network.

* Components marked with an asterisk have been updated.
** VCF v4.2.1 is supported with TKGI v1.11.4, but is untested with this TKGI version.

Upgrade Path

The supported upgrade paths to TKGI v1.11.4 are from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Resolved Issues

TKGI v1.11.4 has the following resolved issues:

Upcoming Deprecations

For information about upcoming deprecations, see Upcoming Deprecations in the TKGI 1.11.0 Release Notes.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition v1.11.3 are also in Tanzu Kubernetes Grid Integrated Edition v1.11.4. See the TKGI v1.11.3 Known Issues below.


The BOSH VM in Clusters Using the Integrated vSphere CSI Driver Might Enter a Failing State While Rebooting

Symptom

As you restart a BOSH worker VM in a cluster using the integrated vSphere CSI Driver, the VM might enter a failing state.

Explanation

The csi-node process on the BOSH worker VM is running but is in an Execution failed state.

Workaround

To return the BOSH VM to a running state, restart the csi-node process:

  1. Determine which BOSH VM is in the failing state.
  2. SSH into the failed BOSH VM.
  3. Run:

    monit restart csi-node
    


TKGI Management Console v1.11.4

Release Date: September 16, 2021

Note: Tanzu Kubernetes Grid Integrated Edition Management Console provides an opinionated installation of TKGI. The supported versions may differ from or be more limited than what is generally supported by TKGI.

Product Snapshot

Element Details
Version v1.11.4
Release date September 16, 2021
Installed Tanzu Kubernetes Grid Integrated Edition version v1.11.4
Installed Ops Manager version v2.10.16
Component Version
Installed Kubernetes version v1.20.10*
Installed Harbor Registry version v2.3.2*
Linux stemcell v621.141*
Windows stemcells v2019.37 and later

* Components marked with an asterisk have been updated.

Upgrade Path

The supported upgrade path to Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.4 is from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Features and Resolved Issues

This release of the TKGI Management Console includes no new features or resolved issues.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.3 are also in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.4. See the TKGI Management Console v1.11.3 Known Issues below.


TKGI v1.11.3

Release Date: August 16, 2021

Product Snapshot

Release Details
Version v1.11.3
Release date August 16, 2021
Component Version
Antrea v0.11.4*
cAdvisor v0.39.1
CSI Driver for vSphere v2.2.1*
CoreDNS v1.7.0+vmware.12*
Docker Linux: v20.10.7
Windows: v19.03.17*
etcd v3.4.13
Harbor v2.2.3*
Kubernetes v1.20.8*
Metrics Server v0.3.6
NCP v3.1.2.2*
Percona XtraDB Cluster (PXC) v0.37.0*
UAA v74.5.24*
Velero v1.6.0
VMware Cloud Foundation (VCF) v4.3, v4.2.1**
Wavefront Wavefront Collector: v1.3.4
Wavefront Proxy: v9.7
Compatibilities Versions
Ops Manager See VMware Tanzu Network.
NSX-T See VMware Product Interoperability Matrices.
vSphere
Windows stemcells v2019.37 or later
Xenial stemcells See VMware Tanzu Network.

* Components marked with an asterisk have been updated.
** VCF v4.2.1 is supported with TKGI v1.11.3, but is untested with this TKGI version.

Upgrade Path

The supported upgrade paths to TKGI v1.11.3 are from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Resolved Issues

TKGI v1.11.3 has the following resolved issues:

Upcoming Deprecations

For information about upcoming deprecations, see Upcoming Deprecations in the TKGI 1.11.0 Release Notes.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition v1.11.2 are also in Tanzu Kubernetes Grid Integrated Edition v1.11.3. See the TKGI v1.11.2 Known Issues below.


TKGI Management Console v1.11.3

Release Date: August 16, 2021

Note: Tanzu Kubernetes Grid Integrated Edition Management Console provides an opinionated installation of TKGI. The supported versions may differ from or be more limited than what is generally supported by TKGI.

Product Snapshot

Element Details
Version v1.11.3
Release date August 16, 2021
Installed Tanzu Kubernetes Grid Integrated Edition version v1.11.3
Installed Ops Manager version v2.10.16
Component Version
Installed Kubernetes version v1.20.8*
Installed Harbor Registry version v2.2.3*
Linux stemcell v621.136*
Windows stemcells v2019.37* and later

* Components marked with an asterisk have been updated.

Upgrade Path

The supported upgrade path to Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.3 is from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Features and Resolved Issues

This release of the TKGI Management Console includes no new features or resolved issues.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.2 are also in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.3. See the TKGI Management Console v1.11.2 Known Issues below.


TKGI v1.11.2

Release Date: June 24, 2021

Product Snapshot

Release Details
Version v1.11.2
Release date June 24, 2021
Component Version
Antrea v0.11.3
cAdvisor v0.39.1
CSI Driver for vSphere v2.2.0
CoreDNS v1.7.0+vmware.10
Docker Linux: v20.10.7*
Windows: v19.03.14
etcd v3.4.13
Harbor v2.2.2*
Kubernetes v1.20.6
Metrics Server v0.3.6
NCP v3.1.2.1
Percona XtraDB Cluster (PXC) v0.35.0
UAA v74.5.22
Velero v1.6.0
VMware Cloud Foundation (VCF) v4.3, v4.2.1**
Wavefront Wavefront Collector: v1.3.4
Wavefront Proxy: v9.7
Compatibilities Versions
Ops Manager See VMware Tanzu Network***.
NSX-T See VMware Product Interoperability Matrices.
vSphere
Windows stemcells v2019.34 or later
Xenial stemcells See VMware Tanzu Network.

* Components marked with an asterisk have been updated.
** VCF v4.2.1 is supported with TKGI v1.11.2, but is untested with this TKGI version.
*** Do not use TKGI with Ops Manager v2.10.15. For more information, see v2.10.15 in Ops Manager v2.10 Release Notes.

Upgrade Path

The supported upgrade paths to TKGI v1.11.2 are from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Warning: Do not upgrade TKGI to TKGI v1.11.2 if your environment was initially created using Ops Manager v1.8.3 or earlier. For more information, see CA Certificates Initially Created by Ops Manager v1.8.3 and Earlier Are Not Supported below.

Resolved Issues

TKGI v1.11.2 has the following resolved issues:

Upcoming Deprecations

For information about upcoming deprecations, see Upcoming Deprecations in the TKGI 1.11.0 Release Notes.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition v1.11.1 are also in Tanzu Kubernetes Grid Integrated Edition v1.11.2. See the TKGI v1.11.1 Known Issues below.

Warning: Do not use TKGI with Ops Manager v2.10.15. For more information, see v2.10.15 in Ops Manager v2.10 Release Notes.


“drain-cluster-windows” Errand Does Not Exist

This issue is fixed in TKGI v1.11.3.

Symptom

When you delete a Linux cluster, the drain-cluster-windows errand reports the following error:

Errand 'drain-cluster-windows' doesn't exist

Explanation

When you delete a cluster, the following errands run:

  • drain-cluster
  • drain-cluster-windows

Both errands run regardless of whether the cluster is a Linux or Windows cluster.

On a Linux cluster, the Windows cluster and the drain-cluster-windows errand do not exist. The log shows the following:

Task 572
Task 572 | 11:39:54 | Preparing deployment: Preparing deployment
Task 572 | 11:39:54 | Deprecation: Global 'properties' are deprecated. Please define 'properties' at the job level.
Task 572 | 11:39:56 | Preparing deployment: Preparing deployment (00:00:02)
 L Error: Errand 'drain-cluster-windows' doesn't exist
Task 572 | 11:39:56 | Error: Errand 'drain-cluster-windows' doesn't exist
Task 572 Started Tue Jun 29 11:39:54 UTC 2021
Task 572 Finished Tue Jun 29 11:39:56 UTC 2021
Task 572 Duration 00:00:02
Task 572 error

On a Windows cluster, both the drain-cluster and drain-cluster-windows errands exist, and no error messages are generated.

Workaround

No workaround is necessary. The error message is informational, accurate, and does not negatively impact deleting the cluster.


Kubernetes Pods on NSX-T Become Stuck in a Creating State

Symptom

During an NSX-T upgrade or while increasing the number of ESXi nodes in your NSX-T environment, the pods in your TKGI Kubernetes clusters become stuck in a creating state. The connections between nsx-node-agent and hyperbus repeatedly close, log Couldn't connect to 'tcp://...' (error: 111-Connection refused), and have a status of COMMUNICATION_ERROR.

Explanation

For information and workaround steps for this Known Issue, see Issue 2795268: Connection between nsx-node-agent and hyperbus flips and Kubernetes pod is stuck at creating state in NSX Container Plugin 3.1.2 Release Notes in the VMware documentation.


Worker Nodes Are Occasionally Not Added to an Azure Back-End Pool

This issue is fixed in TKGI v1.11.4.

Symptom

The Kubernetes services of type LoadBalancer occasionally become inaccessible for a few of your clusters while recreating clusters on Azure. On the few affected clusters, the newly recreated worker VMs were not added back to the load balancer back-end pools used by type LoadBalancer services.

Explanation

For information on this known Kubernetes issue, see Azure Exclude node from LoadBalancer lead to failed reconciliation of services in the Kubernetes GitHub repository.


TKGI Management Console v1.11.2

Release Date: June 24, 2021

Note: Tanzu Kubernetes Grid Integrated Edition Management Console provides an opinionated installation of TKGI. The supported versions may differ from or be more limited than what is generally supported by TKGI.

Product Snapshot

Element Details
Version v1.11.2
Release date June 24, 2021
Installed Tanzu Kubernetes Grid Integrated Edition version v1.11.2
Installed Ops Manager version v2.10.14*
Component Version
Installed Kubernetes version v1.20.6
Installed Harbor Registry version v2.2.2*
Linux stemcell v621.130
Windows stemcells v2019.34 and later

* Components marked with an asterisk have been updated.

Upgrade Path

The supported upgrade path to Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.2 is from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Features and Resolved Issues

This release of the TKGI Management Console includes no new features or resolved issues.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.0 are also in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.2. See the TKGI Management Console v1.11.0 Known Issues below.


TKGI v1.11.1

Release Date: June 14, 2021

Product Snapshot

Release Details
Version v1.11.1
Release date June 14, 2021
Component Version
Antrea v0.11.3
cAdvisor v0.36.0
CSI Driver for vSphere v2.2.0
CoreDNS v1.7.0+vmware.8
Docker Linux: v19.03.14
Windows: v19.03.14
etcd v3.4.13
Harbor v2.2.1
Kubernetes v1.20.6
Metrics Server v0.3.6
NCP v3.1.2.1
Percona XtraDB Cluster (PXC) v0.33.0
UAA v74.5.22
Velero v1.6.0
VMware Cloud Foundation (VCF) v4.3, v4.2.1**
Wavefront Wavefront Collector: v1.3.4
Wavefront Proxy: v9.7
Compatibilities Versions
Ops Manager See VMware Tanzu Network***.
NSX-T See VMware Product Interoperability Matrices.
vSphere
Windows stemcells v2019.34 or later
Xenial stemcells See VMware Tanzu Network.

* Components marked with an asterisk have been updated.
** VCF v4.2.1 is supported with TKGI v1.11.1, but is untested with this TKGI version.
*** Do not use TKGI with Ops Manager v2.10.15. For more information, see v2.10.15 in Ops Manager v2.10 Release Notes.

Upgrade Path

The supported upgrade paths to TKGI v1.11.1 are from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Warning: Do not upgrade TKGI to TKGI v1.11.1 if your environment was initially created using Ops Manager v1.8.3 or earlier. For more information, see CA Certificates Initially Created by Ops Manager v1.8.3 and Earlier Are Not Supported below.

Resolved Issues

TKGI v1.11.1 has the following resolved issues:

Upcoming Deprecations

For information about upcoming deprecations, see Upcoming Deprecations in the TKGI 1.11.0 Release Notes.

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition v1.11.0 are also in Tanzu Kubernetes Grid Integrated Edition v1.11.1. See the TKGI v1.11.0 Known Issues below.

Warning: Do not use TKGI with Ops Manager v2.10.15. For more information, see v2.10.15 in Ops Manager v2.10 Release Notes.


TKGI Management Console v1.11.1

Release Date: June 14, 2021

Note: Tanzu Kubernetes Grid Integrated Edition Management Console provides an opinionated installation of TKGI. The supported versions may differ from or be more limited than what is generally supported by TKGI.

Product Snapshot

Element Details
Version v1.11.1
Release date June 14, 2021
Installed Tanzu Kubernetes Grid Integrated Edition version v1.11.1
Installed Ops Manager version v2.10.11
Component Version
Installed Kubernetes version v1.20.6
Installed Harbor Registry version v2.2.1
Linux stemcell v621.125
Windows stemcells v2019.34 and later

* Components marked with an asterisk have been updated.

Upgrade Path

The supported upgrade path to Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.1 is from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Features and Resolved Issues

TKGI Management Console v1.11.1 has the following resolved issues:

Known Issues

Except where noted, the known issues in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.0 are also in Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.1. See the TKGI Management Console v1.11.0 Known Issues below.


TKGI v1.11.0

Release Date: May 27, 2021

Product Snapshot

Release Details
Version v1.11.0
Release date May 27, 2021
Component Version
Antrea v0.11.3*
cAdvisor v0.36.0*
CSI Driver for vSphere v2.2.0*
CoreDNS v1.7.0+vmware.8
Docker Linux: v19.03.14
Windows: v19.03.14
etcd v3.4.13
Harbor v2.2.1
Kubernetes v1.20.6*
Metrics Server v0.3.6
NCP v3.1.2*
Percona XtraDB Cluster (PXC) v0.33.0
UAA v74.5.22
Velero v1.6.0*
VMware Cloud Foundation (VCF) v4.3, v4.2.1**
Wavefront Wavefront Collector: v1.3.4*
Wavefront Proxy: v9.7*
Compatibilities Versions
Ops Manager See VMware Tanzu Network***.
NSX-T See VMware Product Interoperability Matrices.
vSphere
Windows stemcells v2019.34 or later
Xenial stemcells See VMware Tanzu Network.

* Components marked with an asterisk have been updated.
** VCF v4.2.1 is supported with TKGI v1.11.0, but is untested with this TKGI version.
*** Do not use TKGI with Ops Manager v2.10.15. For more information, see v2.10.15 in Ops Manager v2.10 Release Notes.

Upgrade Path

The supported upgrade paths to TKGI v1.11.0 are from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Warning: Do not upgrade TKGI to TKGI v1.11.0 if your environment was initially created using Ops Manager v1.8.3 or earlier. For more information, see CA Certificates Initially Created by Ops Manager v1.8.3 and Earlier Are Not Supported below.

Warning: If you have nodes with more than 30 stateful Pods or nodes with a total of more than 50 Pods, do not upgrade to TKGI v1.11.0. For more information, see Nodes with More Than 30 Stateful Pods Are in a NotReady State below.

Breaking Changes

TKGI v1.11.0 has the following breaking changes:

  • The built-in Clair container image scanner is deprecated in favor of Trivy. If you have enabled Clair, do one of the following before upgrading to Harbor tile v2.2.1:
    • In the Harbor tile, select the Trivy scanner as the default in “Interrogation Service”.
    • Install the Clair scanner outside the Harbor tile VM and configure the Clair scanner as the default scanner in “Interrogation Service”. For more information about installing Clair, see Getting Started With Clair in the Clair documentation. For information about upgrading TKGI with Clair enabled, see Update Default Scanner in Upgrade Preparation Checklist for Tanzu Kubernetes Grid Integrated Edition v1.10.

Features

TKGI v1.11.0 has the following features and enhancements:

Support for Per-Cluster CA and Custom CA

TKGI v1.11 updates the security of the platform by removing the shared cluster CA and introducing a per-cluster CA. By default a new or updated cluster is given its own unique CA that is used to sign the TLS certificates specific to that cluster. No action is needed to take advantage of the per-cluster CA; it is injected automatically for any Kubernetes cluster provisioned by TKGI v1.11 and later. For more information, see TKGI Certificates.

A byproduct of the introduction of the per-cluster CA is that you can now use a custom CA when you create a cluster. While this is an advanced use case, it may be appropriate for some customers with specific security requirements. See Use a Custom CA for Kubernetes Clusters and Creating Clusters for more information.

Automatic vSphere CSI Driver Integration

TKGI v1.11.0 supports automatic vSphere CSI Driver installation on vSphere. Automatic vSphere CSI Driver integration provides CNS as a process, allowing CNS volumes to be deployed to clusters without requiring access to IaaS credentials. For more information, see Deploying Cloud Native Storage (CNS) on vSphere.

Add DNS Server IPs to Clusters Using Network Profiles

Supports using Network Profiles to add additional DNS server IPs to existing clusters. For information about using a Network Profile to update a cluster with additional DNS server IPs, see Confirm the Network Profile Property Supports Updates in Creating and Deleting Network Profiles (NSX-T Only).

Kubernetes Host Port Support

Supports the Kubernetes Host Port feature. The Kubernetes Host Port feature allows you to expose an application to be externally accessible through a single port. Host Port support requires a Host Port-compatible CNI. Host Port support does not require privileged mode. For more information, see Creating and Deleting Network Profiles (NSX-T Only).

Supports Kubernetes NodeLocal DNSCache

Supports enabling the NCP 3.1 Kubernetes NodeLocal DNSCache feature. NodeLocal DNSCache improves cluster DNS performance by running a DNS caching agent on cluster nodes. The agent runs on the nodes as a DaemonSet. For more information, see Creating and Deleting Network Profiles (NSX-T Only).

CoreDNS is Highly Available

CoreDNS in TKGI v1.11.0 and later is highly available. CoreDNS functions in multi-node Pods are automatically distributed across Pods across your designated AZs. CoreDNS in an existing TKGI multi-node Pod is automatically reconfigured for high availability when the cluster is upgraded to TKGI v1.11.

Support for BOSH VM Extensions

TKGI v1.11.0 supports configuring new and existing TKGI-provisioned Kubernetes clusters using BOSH VM Extensions. For more information, see Using BOSH VM Extensions.

VMware vSphere Bitfusion Interoperability

VMware vSphere Bitfusion supports TKGI v1.11.0.

Bitfusion decouples physical resources from servers within an environment. The Bitfusion platform can share graphical processing units (GPUs) in a virtualized infrastructure, as a pool of network-accessible resources, rather than isolated resources per server. Bitfusion works across AI frameworks, clouds, networks, and in environments such as virtual machines and containers.

For information about integrating your TKGI clusters with Bitfusion, see the VMware Kubernetes Integration with Bitfusion solutions guide. For information about Bitfusion, see the VMware vSphere Bitfusion Documentation.

Additional Features

TKGI v1.11.0 includes the following additional features:

  • Returns an error if a network profile configuration changes a parameter that does not support modification.
  • Supports integration with Tanzu Mission Control. For more information about Tanzu Mission Control, see the Tanzu Mission control documentation.

Resolved Issues

TKGI v1.11.0 has the following resolved issues:

Upcoming Deprecations

The following TKGI features will be deprecated or removed in future versions of TKGI:

  • Flannel Support: Support for the Flannel Container Networking Interface (CNI) will be deprecated in TKGI v1.13. VMware recommends that you upgrade your Flannel CNI-configured clusters to the Antrea CNI. For more information about Flannel CNI deprecation, see About Upgrading from the Flannel CNI to the Antrea CNI in About Tanzu Kubernetes Grid Integrated Edition Upgrades.

  • Docker Support: kubelet support for Docker images has been deprecated and Docker support will be completely removed in Kubernetes v1.24. TKGI support for containerd is scheduled to begin in TKGI v1.12.

  • VCP Support: VCP support has been deprecated and support will be completely removed in a future TKGI version.

  • DenyEscalatingExec: DenyEscalatingExec support has been deprecated in Kubernetes and support will be completely removed in Kubernetes v1.21. For more information, see DenyEscalatingExec in the Kubernetes documentation.

Known Issues

Warning: Do not use TKGI with Ops Manager v2.10.15. For more information, see v2.10.15 in Ops Manager v2.10 Release Notes.

TKGI v1.11.0 has the following known issues:


Nodes with More Than 30 Stateful Pods Are in a NotReady State

This issue is fixed in TKGI v1.11.1.

Warning: If you have nodes with more than 30 stateful Pods or nodes with a total of more than 50 Pods, do not upgrade to TKGI v1.11.0.

Symptom

Nodes that have more than 50 Pods or more than 30 stateful Pods are in a NotReady state, and you see the following errors in kubelet logs:

Error syncing pod ... skipping: rpc error: code = DeadlineExceeded desc = context deadline exceeded
skipping pod synchronization - PLEG is not healthy: pleg was last seen active...
skipping pod synchronization - PLEG is not healthy: pleg was last seen active...
failed to collect filesystem stats - rootDiskErr: could not stat .. to get inode usage: stat ...: no such file or directory, extraDiskErr: could not stat

Explanation

The runc component has introduced instability into nodes consisting of 50 or more Pods or 30 or more statefulset Pods. For more information, see loading seccomp filter: invalid argument in the runc GitHub repository.


Pods Stop After Upgrading From NSX-T v3.0.2 to v3.1.0 on vSphere 7.0 and 7.0.1

Symptom

Your TKGI-provisioned Pods stop after upgrading from NSX-T v3.0.2 to NSX-T v3.1.0 on vSphere 7.0 and 7.0.1.

Explanation

For information, see Issue 2603550: Some VMs are vMotioned and lose network connectivity during UA nodes upgrade in the VMware NSX-T Data Center 3.1.1 Release Notes.

Workaround

To avoid the loss of network connectivity during UA node upgrade, ensure DRS is set to manual mode during your upgrade from NSX-T v3.0.2 to v3.1.0.

If you upgraded to NSX-T v3.1.0 with DRS in automation mode, run the following on the affected Pods’ master VMs to restore Pod connectivity:

monit restart ncp

For more information about upgrading NSX-T v3.0.2 to NSX-T v3.1.0, see Upgrade NSX-T Data Center to v3.0 or v3.1.


Error: Could Not Execute “Apply-Changes” in Azure Environment

Symptom

After clicking Apply Changes on the TKGI tile in an Azure environment, you experience an error ’…could not execute “apply-changes”…’ with either of the following descriptions:

  • {“errors”:{“base”:[“undefined method 'location’ for nil:NilClass”]}}
  • FailedError.new(“Resource Groups in region ’#{location}’ do not support Availability Zones”))

For example:

INFO | 2020-09-21 03:46:49 +0000 | Vessel::Workflows::Installer#run | Install product (apply changes)
2020/09/21 03:47:02 could not execute "apply-changes": installation failed to trigger: request failed: unexpected response from /api/v0/installations:
HTTP/1.1 500 Internal Server Error
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Mon, 21 Sep 2020 17:51:50 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Referrer-Policy: strict-origin-when-cross-origin
Server: Ops Manager
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: f5fc99c1-21a7-45c3-7f39
X-Runtime: 9.905591
X-Xss-Protection: 1; mode=block

44
{"errors":{"base":["undefined method `location' for nil:NilClass"]}}
0

Explanation

The Azure CPI endpoint used by Ops Manager has been changed and your installed version of Ops Manager is not compatible with the new endpoint.

Workaround

Run the following Ops Manager CLI command:

om --skip-ssl-validation --username USERNAME --password PASSWORD --target https://OPSMAN-API curl --silent --path /api/v0/staged/director/verifiers/install_time/IaasConfigurationVerifier -x PUT -d '{ "enabled": false }'

Where:

  • USERNAME is the account to use to run Ops Manager API commands.
  • PASSWORD is the password for the account.
  • OPSMAN-API is the IP address for the Ops Manager API

For more information, see Error 'undefined method location’ is received when running Apply Change on Azure in the VMware Tanzu Knowledge Base.


VMware vRealize Operations Does Not Support Windows Worker-Based Kubernetes Clusters

VMware vRealize Operations (vROPs) does not support Windows worker-based Kubernetes clusters and cannot be used to manage TKGI-provisioned Windows workers.


TKGI Wavefront Requires Manual Installation for Windows Workers

To monitor Windows-based worker node clusters with a Wavefront collector and proxy, you must first install Wavefront on the clusters manually, using Helm. For instructions, see the Wavefront section of the Monitoring Windows Worker Clusters and Nodes topic.


Pinging Windows Worker Kubernetes Clusters Does Not Work

TKGI-provisioned Windows worker-based Kubernetes clusters inherit a Kubernetes limitation that prevents outbound ICMP communication from workers. As a result, pinging Windows workers does not work.

For information about this limitation, see Limitations > Networking in the Windows in Kubernetes documentation.


Velero Does Not Support Backing Up Stateful Windows Workloads

You can use Velero to back up stateless TKGI-provisioned Windows workers only. You cannot use Velero to back up stateful Windows applications. For more information, see Velero on Windows in Basic Install in the Velero documentation.


Tanzu Mission Control Integration Not Supported on GCP

TKGI on Google Cloud Platform (GCP) does not support Tanzu Mission Control (TMC) integration, which is configured in the Tanzu Kubernetes Grid Integrated Edition tile > the Tanzu Mission Control pane.

If you intend to run TKGI on GCP, skip this pane when configuring the Tanzu Kubernetes Grid Integrated Edition tile.


TMC Data Protection Feature Requires Privileged TKGI Containers

TMC Data Protection feature supports privileged TKGI containers only. For more information, see Plans in the Installing TKGI topic for your IaaS.


Windows Worker Kubernetes Clusters with Group Managed Service Account Do Not Support Compute Profiles

Windows worker-based Kubernetes clusters integrated with group Managed Service Account (gMSA) cannot be managed using compute profiles.


Windows Worker Kubernetes Clusters on Flannel Do Not Support Compute Profiles

On vSphere with NSX-T networking you can use compute profiles with both Linux and Windows worker‑based Kubernetes clusters. On vSphere with Flannel networking, you can apply compute profiles only to Linux clusters.


TKGI CLI Does Not Prevent Reducing the Control Plane Node Count

TKGI CLI does not prevent accidentally reducing a cluster’s control plane node count using a compute profile.

Warning: Reducing a cluster’s control plane node count can destroy the cluster. Do not scale out or scale in existing master nodes by reconfiguring the TKGI tile or by using a compute profile. Reducing a cluster’s number of control plane nodes may remove a master node and cause the cluster to become inactive.


Windows Cluster Nodes Not Deleted After VM Deleted

Symptom

After you delete a VM using the management console of your infrastructure provider, you notice a Windows worker node that had been on that VM is now in a notReady state.

Solution

  1. To identify the leftover node:

    kubectl get no -o wide
    
  2. Locate nodes on the returned list that are in a notReady state and have the same IP address as another node in the list.

  3. To manually delete a notReady node:

    kubectl delete node NODE-NAME
    

    Where NODE-NAME is the name of the node in the notReady state.


502 Bad Gateway After OIDC Login

Symptom

You experience a “502 Bad Gateway” error from the NSX load balancer after you log in to OIDC.

Explanation

A large response header has exceeded your NSX-T load balancer maximum response header size. The default maximum response header size is 10,240 characters and should be resized to 50,000.

Workaround

If you experience this issue, manually reconfigure your NSX-T request_header_size and response_header_size to 50,000 characters. For information about configuring NSX-T default header sizes, see OIDC Response Header Overflow in the Knowledge Base.


Difficulty Changing Proxy for Windows Workers

You must configure a global proxy in the Tanzu Kubernetes Grid Integrated Edition tile > Networking pane before you create any Windows workers that use the proxy.

You cannot change the proxy configuration for Windows workers in an existing cluster.


Character Limitations in HTTP Proxy Password

For vSphere with NSX-T, the HTTP Proxy password field does not support the following special characters: & or ;.


Error After Modifying Your Harbor Storage Configuration

Symptom

You receive the following error after modifying your existing Harbor installation’s storage configuration:

Error response from daemon: manifest for ... not found: manifest unknown: manifest unknown

Explanation

Harbor does not support modifying an existing Harbor installation’s storage configuration.

Workaround

To modify your Harbor storage configuration, re-install Harbor. Before starting Harbor, configure the new Harbor installation with the desired configuration.


Unexplained Errors After Interrupting a Log Stream When Using Antrea Networking

This issue is fixed in TKGI v1.11.3.

Symptom

While using Antrea networking, you observe unexplainable errors after you interrupt a log stream started using kubectl logs -f POD-NAME. The errors could include any of the following:

  • kubectl returns the error: “Error from server (TooManyRequests): the server has received too many”.
  • kube-apiserver returns an http code 429.

Explanation

When using Antrea networking there is a chance that konnectivity-agent will become unstable after interrupting your kubectl log steam.

Workaround

To resolve the issue:

  1. Log in to the master VM:

    bosh -d DEPLOYMENT-NAME ssh master/0
    
  2. Change to root:

    sudo -i
    
  3. Restart proxy-server:

    monit restart proxy-server
    
  4. Wait for proxy-server restart:

    monit summary
    


Ingress Controller Statefulset Fails to Start After Resizing Worker Nodes

Symptom

Permissions are removed from your cluster’s files and processes after resizing the persistent disk during a cluster upgrade. The ingress controller statefulset fails to start.

Explanation

When resizing a persistent disk, Bosh migrates the data from the old disk to the new disk but does not copy the files’ extended attributes.

Workaround

To resolve the problem, complete the steps in Ingress controller statefulset fails to start after resize of worker nodes with permission denied in the VMware Tanzu Knowledge Base.


Azure Default Security Group Is Not Automatically Assigned to Cluster VMs

Symptom

You experience issues when configuring a load balancer for a multi-master Kubernetes cluster or creating a service of type LoadBalancer. Additionally, in the Azure portal, the VM > Networking page does not display any inbound and outbound traffic rules for your cluster VMs.

Explanation

As part of configuring the Tanzu Kubernetes Grid Integrated Edition tile for Azure, you enter Default Security Group in the Kubernetes Cloud Provider pane. When you create a Kubernetes cluster, Tanzu Kubernetes Grid Integrated Edition automatically assigns this security group to each VM in the cluster. However, on Azure the automatic assignment may not occur.

As a result, your inbound and outbound traffic rules defined in the security group are not applied to the cluster VMs.

Workaround

If you experience this issue, manually assign the default security group to each VM NIC in your cluster.


One Plan ID Longer than Other Plan IDs

Symptom

One of your plan IDs is one character longer than your other plan IDs.

Explanation

In TKGI, each plan has a unique plan ID. A plan ID is normally a UUID consisting of 32 alphanumeric characters and 4 hyphens. However, the Plan 4 ID consists of 33 alphanumeric characters and 4 hyphens.

Solution

You can safely configure and use Plan 4. The length of the Plan 4 ID does not affect the functionality of Plan 4 clusters.

If you require all plan IDs to have identical length, do not activate or use Plan 4.


The TKGI API FQDN Must Not Include Trailing Whitespace

Symptom

Your TKGI logs include the following error:

'uaa'. Errors are:- Error filling in template 'uaa.yml.erb' (line 59: Client redirect-uri is invalid: uaa.clients.pks_cli.redirect-uri Client redirect-uri is invalid: uaa.clients.pks_cluster_client.redirect-uri)

Explanation

The TKGI API fully-qualified domain name (FQDN) for your cluster contains leading or trailing whitespace.

Workaround

Do not include whitespace in the TKGI tile API Hostname (FQDN) field.


Database Cluster Stops After a Database Instance is Stopped

Symptom

After you stop one instance in a multiple-instance database cluster, the cluster stops, or communication between the remaining databases times out, and the entire cluster becomes unreachable.

The following might be in your UAA log:

WSREP has not yet prepared node for application use

Explanation

The database cluster is unable to recover automatically because a member is no longer available to reconcile quorum.


Windows Workloads With Attached Dynamic PVs Must Be Removed before Deleting a Cluster

This issue is fixed in TKGI v1.11.2.

Symptom

Your tkgi delete-cluster operation hangs while draining a Windows worker VM containing a Pod bound to one or more dynamic persistent volumes (PVs).

Workaround

Before deleting your Windows cluster, remove all workloads.


Velero Back Up Fails for vSphere PVs Attached to Clusters on Kubernetes v1.20 and Later

Symptom

Backing up vSphere persistent volumes using Velero fails and your Velero backup log includes the following error:

rpc error: code = Unknown desc = Failed during IsObjectBlocked check: Could not translate selfLink to CRD name

Explanation

This is a known issue when backing up clusters on Kubernetes v1.20 and later using the Velero Plugin for vSphere v1.1.0 or earlier.

Workaround

To resolve the problem, complete the steps in Velero backups of vSphere persistent volumes fail on Kubernetes clusters version 1.20 or higher (83314) in the VMware Tanzu Knowledge Base.


NSX-T Pre-Check Errand Fails Due to Edge Node CPU Count Configuration

This issue is fixed in TKGI v1.8.0 and later and was incorrectly included in the TKGI v1.11 Release Notes.

Symptom

You have configured your NSX-T Edge Node VM as medium size, and the NSX-T Pre-Check Errand fails with the following error: “ERROR: NSX-T Precheck failed due to Edge Node … no of cpu cores is less than 8”.

Explanation

The NSX-T Pre-Check Errand is erroneously returning the “cpu cores is less than 8” error.

Solution

You can safely configure your NSX-T Edge Node VMs as medium size and ignore the error.


Fluent Bit ClusterLogSink Returns 'TCP Connection Failed’ While CEIP Telemetry Services Are Disabled

This issue is fixed in TKGI v1.11.3.

Symptom

Fluent Bit ClusterLogSink logs the errors “TCP connection failed” and “no upstream connections available” after you disable CEIP Telemetry services:

[2020/10/14 10:38:50] [error] [io] TCP connection failed: telemetry.pks.internal:24224 (Connection timed out)
[2020/10/14 10:38:50] [error] [out_fw] no upstream connections available
[2020/10/14 10:41:08] [error] [io] TCP connection failed: telemetry.pks.internal:24224 (Connection timed out)
[2020/10/14 10:41:08] [error] [out_fw] no upstream connections available
[2020/10/14 10:43:51] [error] [io] TCP connection failed: telemetry.pks.internal:24224 (Connection timed out)
[2020/10/14 10:43:51] [error] [out_fw] no upstream connections available
[2020/10/14 10:43:51] [ warn] [engine] Task cannot be retried: task_id=0 thread_id=2 output=forward.0

Explanation

Fluent Bit intermittently attempts to connect to the Telemetry Server during ClusterLogSink. The connection attempts fail while the CEIP Telemetry services are disabled, and Fluent Bit logs the failed connection attempts.


Creating Two Windows Clusters at the Same Time Fails

Symptom

The first time that you try to create two Windows clusters at the same time, the creation of one of the clusters fails. If you run pks cluster CLUSTER-NAME to examine the last action taken on the cluster, you see the following:

Last Action: Create
Last Action State: failed
Last Action Description: Instance provisioning failed: There was a problem completing your request.
. . .
operation: create, error-message: Failed to acquire lock
. . .
locking task id is 111, description: 'create deployment'

Explanation

This is a known issue that occurs the first time that you create two Windows clusters concurrently.

Workaround

Recreate the failed cluster. This issue only occurs the first time that you create two Windows clusters concurrently.


Cluster Fails to Restart

This issue is fixed in TKGI v1.11.3.

Symptom

A stopped cluster fails to restart and you see the following error in the NSX Container Plug-in (NCP) logs:

Response body {'httpStatus': 'BAD_REQUEST', 'module_name': 'nsx-search', 'error_code': 60513, 'error_message': 'The result set is too large. Please refine the search criteria.'}

Explanation

This problem occurs on clusters with a network policy with more than twelve port-protocol fields.

Workaround

Modify the cluster network policy to have twelve or fewer port-protocol fields.


You Cannot Use vRealize Log Insight to Monitor NCP

This issue is fixed in TKGI v1.11.3.

Symptom

vRealize Log Insight (vRLI) monitoring does not include NCP stdout or stderror.

Explanation

A bug in the vRLI configuration prevents inclusion of NCP stdout and stderror in vRLI.

Workaround

To write NCP logs for a cluster to vRLI:

  1. Confirm you have admin access to the master node.
  2. Open /var/vcap/jobs/fluentd/config/config.d/nsx-t.conf.
  3. Replace:

    expression /^(?<num>[\w]+) (?<time>[^ ]+) (?<nsx_uuid>[^ ]+) (?<nsx_component>[^ ]+) (?<pid>[^ ]+) - \[.* level="(?<severity>[^\"]+)".*\] (?<message>.+)/
    

    with:

    expression /^(?<time>[^ ]+) (?<nsx_uuid>[^ ]+) (?<nsx_component>[^ ]+) (?<pid>[^ ]+) - \[.* level="(?<severity>[^\"]+)".*\] (?<message>.+)/
    

Note: This workaround affects only the cluster you modified and does not persist.


The TKGI API VM fluentd.stdout.log Grows While vRealize Log Insight is Disabled

This issue is fixed in TKGI v1.11.2.

Symptom

While vRealize Log Insight (vRLI) is disabled, the log for fluentd.stdout on the TKGI API VM grows quite large and rolls over frequently.

Explanation

The fluentd job on the TKGI API VM automatically starts when vRLI is disabled. This fluentd job runs continuously, frequently writing inaccessible hostname errors to stdout for the non-existent vRLI endpoint. The fluentd stdout on the TKGI API VM is logged in: /var/vcap/sys/log/pks-vrli-control-plane-fluentd/fluentd.stdout.log.


'Invalid Gateway for Network’ Error after Updating a Cluster with Both nodes_dns And IP Subnet Range Parameters

This issue is fixed in TKGI v1.11.3.

Symptom

After updating a cluster with a network profile that specifies a nodes_dns parameter and an IP subnet range other than the typical x.x.x.x/24 range, for example, 1.1.1.64/26, you see the following error for the updated cluster:

operation: update, error-message: Invalid gateway for network 'pks-...': must be inside the range

Subsequently, when you create or update a valid cluster, the process returns the same “Invalid gateway for network” error message.

Explanation

The error is reported because TKGI has incorrectly updated the gateway IP to x.x.x.1, which is outside of your designated IP range.

Additionally, BOSH validates all cloud-configs when creating deployments. The invalid cloud-config created by the error cluster blocks subsequent create and update cluster processes, including processes for valid clusters, and returns the same “Invalid gateway for network” error message.

Avoid updating and resizing clusters with an IP subnet range other than x.x.x.x/24.

For additional information, see TKGI cluster update failing with error: “Invalid gateway for network 'xxx’: must be inside the range” (85101) in the Knowledge Base.

Workaround

If you’ve already encountered this issue, delete the cluster with this error to avoid the subsequent BOSH issue.

To update an existing cluster:

  • Remove the nodes_dns parameter from the network profile and update the cluster with the modified network profile.

To create a new cluster, do one of the following:

  • Create the cluster using a typical x.x.x.x/24 IP subnet range.
  • Configure nodes_dns for the cluster through your Tanzu Kubernetes Grid Integrated Edition tile configuration and remove the nodes_dns parameter from the network profile.


Wavefront Collector is Duplicated or Not Collecting from TKGI Namespaces

This issue is fixed in TKGI v1.11.3.

Symptom

After upgrading TKGI to TKGI v1.10.0 or later, you observe that two Wavefront collectors are running instead of one. If one of the collectors stops, the remaining collector alternates between collecting metrics as itself and as the stopped collector. The pks.kubernetes.daemonset. and pks.kubernetes.deployment. namespaces may stop reporting.

Explanation

Two instances of Wavefront collector are currently installed and running: a v1.20 DaemonSet collector and a v1.26 Deployment collector. Both Wavefront collectors are running and collecting metrics. Only the v1.26 Deployment collector instance should be running.

Workaround

You must remove the v1.20 DaemonSet Wavefront collector and leave the v1.26 Deployment collector running.

To remove the DaemonSet Wavefront collector:

  1. Save the DaemonSet YAML creation file as wf_daemonset.yaml. Keep a copy of the file in case you need to restore the DaemonSet Wavefront collector.

  2. To delete the Wavefront Collector DaemonSet:

    kubectl delete daemonset wavefront-collector -n pks-system
    

If you later decide that you need to restore the DaemonSet collector, you can restore using the DaemonSet yaml creation file you saved earlier.

  1. To restore the DaemonSet collector:

    kubectl apply -f wf_daemonset.yaml 
    kubetcl get daemonset/wavefront-collector -n pks-system -o yaml > wf_daemonset.yaml
    


Tags Applied Using tkgi cli Commands Are Removed during tkgi update-cluster

This issue is fixed in TKGI v1.11.3.

Symptom

The tags you applied to a cluster using the tkgi cli --tags parameter are removed whenever you run tkgi update-cluster.

Workaround

Specify the --tags parameter when updating a cluster using tkgi update-cluster:

  1. To confirm the existing tags on a cluster:

    tkgi cluster CLUSTER-NAME
    

    Where CLUSTER-NAME is the name of the cluster to update.

    Note the tags listed under Tags:.

  2. When running tkgi update-cluster, include the --tags parameter, specifying all of the existing tags on the cluster.


CA Certificates Initially Created by Ops Manager v1.8.3 and Earlier Are Not Supported

This issue is fixed in TKGI v1.11.3.

Symptom

While upgrading TKGI to TKGI v1.11.0, v1.11.1 or v1.11.2, you might observe the following error:

Config Server failed to generate value for '/microbosh-.../tls-kubernetes-2018' with type 'certificate'. HTTP Code '404', Error: 'The request could not be completed because the CA does not exist or you do not have sufficient authorization.'

Explanation

TKGI v1.11.0, v1.11.1, and v1.11.2 do not support the CA Certificates named with the default microbosh- prefix applied to certificates by Ops Manager v1.8.3 and earlier.


TKGI MC Unable to Manage TKGI after Restoring the TKGI Control Plane from Backup

Symptom

After you restore Ops Manager and the TKGI API VM from backup, TKGI functions normally, but your TKGI MC tabs include the following error: “…product ‘pivotal-container service’ is not deployed…”.

Explanation

TKGI MC is associated with an Ops Manager with a specific name. If you rename Ops Manager with a new name while restoring, your TKGI MC will not recognize the restored Ops Manager and cannot manage it.


Telegraf Agent Unable to Read From localhost Port 8080

This issue is fixed in TKGI v1.11.4.

Symptom

Your Telegraf agents fail to read from localhost port 8080. Your cluster master node /var/vcap/sys/log/metric-sink/telegraf.stderr.log files contain logged errors similar to the following:

Error in plugin: error making HTTP request to http://localhost:8080/metrics: Get http://localhost:8080/metrics: dial tcp 127.0.0.1:8080: connect: connection refused

Explanation

Support for the unsecured port 8080 was removed in Kubernetes v1.20.


Deleted Clusters are Listed in Cluster Lists

Symptom

After running tkgi delete-cluster and cluster deletion has completed, the deleted cluster continues to be listed when running tkgi clusters.


SAML Authentication Requests Are Always Signed

This issue is fixed in TKGI v1.11.4.

Symptom

After you disable the SAML Identity Provider Sign Authentication Requests setting on the Tanzu Kubernetes Grid Integrated Edition tile, SAML IdP authentication requests continue to be signed.


BOSH Director Logs the Error 'Duplicate vm extension name 'disk_enable_uuid’’

Symptom

After you uninstall TKGI, then reinstall TKGI in the same environment, BOSH Director logs errors similar to the following:

.../gems/bosh-director-0.0.0/lib/bosh/director/deployment_plan/cloud_manifest_parser.rb:120:in `parse_vm_extensions': Duplicate vm extension name 'disk_enable_uuid' (Bosh::Director::DeploymentDuplicateVmExtensionName)

Explanation

The pivotal-container-service cloud-config was not removed when you uninstalled the TKGI tile, and it remained active. When you reinstalled the TKGI tile, an additional pivotal-container-service cloud-config was created, causing the metrics_server to fall into a crash-loop state.

Workaround

For more information, see “Duplicate vm extension name 'disk_enable_uuid’” error when metrics_server runs on Director VM in Tanzu Kubernetes Grid Integrated Edition in the VMware Tanzu Community Knowledge Base.


Node Hostnames Are Not Resolved for BYO DNS in Azure Environments

Symptom

If you use a bring-your-own DNS configuration in Azure environments, node hostnames are not resolved on your TKGI clusters, and kubectl returns a “no such host” error:

Error from server: Get "...": dial tcp: lookup ... on ... no such host

Explanation

On Azure, the Azure CPI uses BOSH agent_id GUIDs as the VM name. Additionally, the Azure Kubernetes cloud provider mandates the VM name matches the hostname and that the hostname is resolvable without using search domains.

TKGI on Azure relies on Dynamic DNS updates via DHCP to allow resolvable K8s node names. Environments with a bring-your-own DNS server configuration must provide an alternative method for updating DNS entries.


TKGI Management Console v1.11.0

Release Date: May 27, 2021

Note: Tanzu Kubernetes Grid Integrated Edition Management Console provides an opinionated installation of TKGI. The supported versions may differ from or be more limited than what is generally supported by TKGI.

Product Snapshot

Element Details
Version v1.11.0
Release date May 27, 2021
Installed Tanzu Kubernetes Grid Integrated Edition version v1.11.0
Installed Ops Manager version v2.10.11
Component Version
Installed Kubernetes version v1.20.6*
Installed Harbor Registry version v2.2.1*
Linux stemcell v621.125*
Windows stemcells v2019.34 and later*

* Components marked with an asterisk have been updated.

Upgrade Path

The supported upgrade path to Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.0 is from Tanzu Kubernetes Grid Integrated Edition v1.10.0 and later.

Warning: If you have nodes with more than 30 stateful Pods or nodes with a total of more than 50 Pods, do not upgrade to TKGI v1.11.0. For more information, see Nodes with More Than 30 Stateful Pods Are in a NotReady State above.

Features and Resolved Issues

TKGI Management Console v1.11.0 has the following resolved issues:

Known Issues

The Tanzu Kubernetes Grid Integrated Edition Management Console v1.11.0 has the following known issues:

Warning: If you have nodes with more than 30 stateful Pods or nodes with a total of more than 50 Pods, do not upgrade to TKGI v1.11.0. For more information, see Nodes with More Than 30 Stateful Pods Are in a NotReady State above.


vRealize Log Insight Integration Does Not Support HTTPS Connections

Symptom

The Tanzu Kubernetes Grid Integrated Edition Management Console integration to vRealize Log Insight does not support connections to the HTTPS port on the vRealize Log Insight server.

Workaround

  1. Use SSH to log in to the Tanzu Kubernetes Grid Integrated Edition Management Console appliance VM.
  2. Open the file /lib/systemd/system/pks-loginsight.service in a text editor.
  3. Add -e LOG_SERVER_ENABLE_SSL_VERIFY=false.
  4. Set -e LOG_SERVER_USE_SSL=true.

    The resulting file should look like the following example:

    ExecStart=/bin/docker run --privileged --restart=always --network=pks
    -v /var/log/journal:/var/log/journal
    --name=pks-loginsight
    -e TYPE=gear2-vm
    -e LOG_SERVER_HOST=${LOGINSIGHT_HOST}
    -e LOG_SERVER_PORT=${LOGINSIGHT_PORT}
    -e LOG_SERVER_ENABLE_SSL_VERIFY=false
    -e LOG_SERVER_USE_SSL=true
    -e LOG_SERVER_AGENT_ID=${LOGINSIGHT_ID}
    pksoctopus/vrli-journald:v07092019
    
  5. Save the file and run systemctl daemon-reload.

  6. To restart the vRealize Log Insight service, run systemctl restart pks-loginsight.service.

Tanzu Kubernetes Grid Integrated Edition Management Console can now send logs to the HTTPS port on the vRealize Log Insight server.


vSphere HA causes Management Console ovfenv Data Corruption

Symptom

If you enable vSphere HA on a cluster, if the TKGI Management Console appliance VM is running on a host in that cluster, and if the host reboots, vSphere HA recreates a new TKGI Management Console appliance VM on another host in the cluster. Due to an issue with vSphere HA, the ovfenv data for the newly created appliance VM is corrupted and the new appliance VM does not boot up with the correct network configuration.

Workaround

  • In the vSphere Client, right-click the appliance VM and select Power > Shut Down Guest OS.
  • Right-click the appliance again and select Edit Settings.
  • Select VM Options and click OK.
  • Verify under Recent Tasks that a Reconfigure virtual machine task has run on the appliance VM.
  • Power on the appliance VM.


Base64 encoded file arguments are not decoded in Kubernetes profiles

Symptom

Some file arguments in Kubernetes profiles are base64 encoded. When the management console displays the Kubernetes profile, some file arguments are not decoded.

Workaround

Run echo "$content" | base64 --decode


Network profiles not immediately selectable

Symptom

If you create network profiles and then try to apply them in the Create Cluster page, the new profiles are not available for selection.

Workaround

Log out of the management console and log back in again.


Real-Time IP information not displayed for network profiles

Symptom

In the cluster summary page, only default IP pool, pod IP block, node IP block values are displayed, rather than the real-time values from the associated network profile.

Workaround

None


Error After Modifying Your Harbor Storage Configuration

Symptom

You receive the following error after modifying your existing Harbor installation’s storage configuration:

Error response from daemon: manifest for ... not found: manifest unknown: manifest unknown

Explanation

Harbor does not support modifying an existing Harbor installation’s storage configuration.

Workaround

To modify your Harbor storage configuration, re-install Harbor. Before starting Harbor, configure the new Harbor installation with the desired configuration.


Windows Stemcells Must be Re-Imported After Upgrading Ops Manager

Symptom

After upgrading Ops Manager, your Management Console does not recognize a Windows stemcell imported when using the prior version of Ops Manager.

Workaround

If your Management Console does not recognize a Windows stemcell after upgrading Ops Manager:

  1. Re-import your previously imported Windows stemcell.
  2. Apply Changes to TKGI MC.


Your New Clusters Are Not Shown In Tanzu Mission Control

Symptom

After you create a cluster, Tanzu Mission Control does not include the cluster in cluster lists. You have a “Resource not found” error similar to the following in your BOSH logs:

Cluster Name in TMC: cluster-1
Cluster Name Prefix: tkgi-my-prefix-
Group Name in TMC: my-prefix-clusters
Cluster Description in TMC: VMware Enterprise PKS Attaching cluster ''tkgi-my-prefix-cluster-1'' to TMC 
Fetching token successful 
request POST:/v1alpha1/clusters, 
response 404 Not Found:{"error":"Resource not found - clustergroup(my-prefix-clusters) 
org id(d859dc9f-g622-426d-8c91-939a9f13dea9)",
"code":5,"message":"Resource not found - clustergroup(my-prefix-clusters)

Explanation

The cluster group you assign a cluster to must be defined in Tanzu Mission Control before you assign your cluster to the cluster group in the TKGI Management Console.

Workaround

To resolve the problem, complete the steps in Attaching a Tanzu Kubernetes Grid Integrated (TKGI) cluster to Tanzu Mission Control (TMC) fails with “Resource not found - clustergroup(cluster-group-name)” in the VMware Tanzu Knowledge Base.


Management Console Incorrectly Displays Kubernetes Version as 1.20.5

TKGI v1.11 installs Kubernetes v1.20.6, but the TKGI Management Console incorrectly displays the shipped Kubernetes version as 1.20.5.


Please send any feedback you have to pks-feedback@pivotal.io.