Configure NSX Load Balancer for each vCenter Server.

<span style="color:#FF0000">NEW:</span> Attempts to run the Remove Disk operation on a vSAN Direct Datastore fail with the VimFault - Cannot complete the operation error

Generally, you can observe this error when one of the following scenarios occurs:

Workaround:

After the reconfiguration task for the vSphere Pod completes, run the Remove Disk operation again. The Remove Disk operation successfully proceed.

An expansion of a Supervisor cluster PVC in offline or online mode does not result in an expansion of a corresponding Tanzu Kubernetes cluster PVC

A pod that uses the Tanzu Kubernetes cluster PVC cannot use the expanded capacity of the Supervisor cluster PVC because the filesystem has not been resized.

Workaround: Resize the Tanzu Kubernetes cluster PVC to a size equal or greater than the size of the Supervisor cluster PVC.

Static provisioning in Kubernetes does not verify if the PV and backing volume sizes are equal. If you statically create a PVC in a Tanzu Kubernetes cluster, and the PVC size is less than the size of the underlying corresponding Supervisor cluster PVC, you might be able to use more space than the space you request in the PV. If the size of the PVC you statically create in the Tanzu Kubernetes cluster is greater than the size of the underlying Supervisor cluster PVC, you might notice No space left on device error even before you exhaust the requested size in the Tanzu Kubernetes cluster PV.

Workaround:

Attempts to create a PVC from a supervisor namespace or a TKG cluster fail if the external csi.vsphere.vmware.com provisioner loses its lease for leader election

When you try to create a PVC from a supervisor namespace or a TKG cluster using the kubectl command, your attempts might not succeed. The PVC remains in the Pending state. If you describe the PVC, the Events field displays the following information in a table layout:

Workaround:

Kubernetes will create a new instance of the CSI controller and all sidecars will be reinitialized.

All PVC operations, such as create, attach, detach, or delete a volume, fail while CSI cannot connect to vCenter Server

In addition to operation failures, the Volume health information and the StoragePool CR cannot be updated in the Supervisor cluster. The CSI and Syncer logs display errors about not being able to connect to vCenter Server.

CSI connects to vCenter Server as a specific solution user. The password for this SSO user is rotated by wcpsvc once every 24 hours and the new password is transferred in to a Secret that the CSI driver reads to connect to vCenter Server. If the new password fails to be delivered to Secret, the stale password remains in the Supervisor cluster, and the CSI driver fails its operations.

This problem affects vSAN Data Persistence Platform and all CSI volume operations.

Workaround:

Typically the WCP Service delivers the updated password to CSI that runs in the Kubernetes cluster. Occasionally, the password delivery doesn't happen due to a problem, for example, a connectivity issue or an error in earlier part of the sync process. The CSI continues to use the old password and eventually locks the account due to too many authentication failures.

Ensure that the WCP cluster is in a healthy and running state. No errors should be reported for that cluster on Workload Management page. After problems causing the sync to fail are resolved, force a password refresh to unlock the locked account.

To force reset of the password:

The wcpsvc resets the password, which unlocks the account and delivers the new password to the cluster.

When a Supervisor Cluster is upgraded, it can trigger a rolling update of all the Tanzu Kubernetes clusters to propagate any new configuration settings. During this process, a previously "Running" TKC Cluster might hang in the "Updating" phase. A "Running" Tanzu Kubernetes cluster only indicates the availability of the control plane and it is possible that the required control plane and worker nodes have not been successfully created. Such a Tanzu Kubernetes cluster might fail the health checks that are performed during the rolling update that initiates upon completion of the Supervisor Cluster upgrade. This results in the Tanzu Kubernetes cluster hanging in the "Updating" phase and can be confirmed by looking at the events on the KubeadmControlPlane resources associated with the Tanzu Kubernetes Cluster. The events emitted by the resource will be similar to the one below:

Warning ControlPlaneUnhealthy 2m15s (x1026 over 5h42m) kubeadm-control-plane-controller Waiting for control plane to pass control plane health check to continue reconciliation: machine's (gc-ns-1597045889305/tkg-cluster-3-control-plane-4bz9r) node (tkg-cluster-3-control-plane-4bz9r) was not checked

Workaround : None.

When a VI Admin deletes a storage class on from the vCenter Server namespace, access to that storage class is not removed for any Tanzu Kubernetes cluster that is already using it.

Workaround:

The embedded container registry SSL certificate is not copied to Tanzu Kubernetes cluster nodes

When the embedded container registry is enabled for a Supervisor Cluster, the Harbor SSL certificate is not included in any Tanzu Kubernetes cluster nodes created on that SC, and you cannot connect to the registry from those nodes.

Workaround: Copy and paste the SSL certificate from the Supervisor Cluster control plane to the Tanzu Kubernetes cluster worker nodes.

When multiple vCenter Server instances are configured in an Embedded Linked Mode setup, the UI allows the user to select a content library created on a different vCenter Server instance. Selecting such a library results in virtual machine images not being available for DevOps users to provision a Tanzu Kubernetes cluster. In this case, `kubectl get virtualmachineimages` does not return any results.

Workaround: When you associate a content library with the Supervisor Cluster for Tanzu Kubernetes cluster VM images, choose a library that is created in the same vCenter Server instance where the Supervisor Cluster resides. Alternatively, create a local content library which also supports air-gapped provisioning of Tanzu Kubernetes clusters.

You cannot provision new Tanzu Kubernetes clusters, or scale out existing clusters, because the Content Library subscriber cannot synchronize with the publisher.

When you set up a Subscribed Content Library for Tanzu Kubernetes cluster OVAs, an SSL certificate is generated, and you are prompted to manually trust the certificate by confirming the certificate thumbprint. If the SSL certificate is changed after the initial library setup, the new certificate must be trusted again by updating the thumbprint.

Edit the settings of the Subscribed Content Library. This will initiate a probe of the subscription URL even though no change is requested on the library. The probe will discover that the SSL certificate is not trusted and prompt you to trust it.

Tanzu Kubernetes Release version 1.16.8 is incompatible with vCenter Server 7.0 Update 1.

The Tanzu Kubernetes Release version 1.16.8 is incompatible with vCenter Server 7.0 Update 1. You must update Tanzu Kubernetes clusters to a later version before performing a vSphere Namespaces update to U1.

Before performing a vSphere Namespaces update to the vCenter Server 7.0 Update 1 release, update each Tanzu Kubernetes cluster running version 1.16.8 to a later version. Refer to the List of Tanzu Kubernetes Releases in the documentation for more information.

After upgrading the Workload Control Plane to vCenter Server 7.0 Update 1, new VM Class sizes are not available.

Description : After upgrading to vSphere 7.0.1, and then performing a vSphere Namespaces update of the Supervisor Cluster, for Tanzu Kubernetes clusters, running the command "kubectl get virtualmachineclasses" does not list the new VM class sizes 2x-large, 4x-large, 8x-large.

Workaround : None. The new VM classes sizes can only be used with a new installation of the Workload Control Plane.

The Tanzu Kubernetes Release version 1.17.11 vmware.1-tkg.1 times out connecting to the cluster DNS server when using the Calico CNI.

The Tanzu Kubernetes Release version v1.17.11+vmware.1-tkg.1 has a Photon OS kernel issue that prevents the image from working as expected with the Calico CNI.

Workaround : For Tanzu Kubernetes Release version 1.17.11, the image identified as "v1.17.11+vmware.1-tkg.2.ad3d374.516" fixes the issue with Calico. To run Kubernetes 1.17.11, use this version instead of "v1.17.11+vmware.1-tkg.1.15f1e18.489". Alternatively, use a different Tanzu Kubernetes Release, such as version 1.18.5 or 1.17.8 or 1.16.14.

When vSphere with Tanzu networking is configured with NSX-T Data Center, updating an "ExternalTrafficPolicy: Local" Service to "ExternalTrafficPolicy: Cluster" will render this Service's LB IP inaccessible on SV Masters

When a LoadBalancer type Kubernetes Service is initially created in workload clusters with ExternalTrafficPolicy: Local , and later updated to ExternalTrafficPolicy: Cluster , access to this Service's LoadBalancer IP on the Supervisor Cluster VMs will be dropped.

Workaround: Delete the Service and recreate it with ExternalTrafficPolicy: Cluster .

A known issue exists in the Kubernetes upstream project where occasionally kube-controller-manager goes into a loop resulting in high CPU usage which might effect functionality of Tanzu Kubernetes clusters. You might notice that the process, kube-controller-manager, is consuming a larger than expected amount of CPU and is outputting repeated logs indicating failed for updating Node.Spec.PodCIDRs .

Workaround: Delete the kube-controller-manager pod that sits inside the control plane node with such an issue. The pod will be recreated and the issue should not reappear.

Kubelet's configuration file is generated at the time kubeadm init is run and then replicated during cluster upgrades. At the time of 1.16, kubeadm init generates a config file that set resolvConf to /etc/resolv.conf that was then overwritten by a the command-line flag --resolv-conf pointing at /run/systemd/resolve/resolv.conf . During 1.17 and 1.18, kubeadm continues to configure Kubelet with the correct --resolv-conf . As of 1.19, kubeadm no longer configures the command line flag and instead relies on the Kubelet configuration file. Due to the replication process during cluster upgrades, a 1.19 cluster upgraded from 1.16 will include a config file where resolvConf points at /etc/resolv.conf instead of /run/systemd/resolve/resolv.conf .

Workaround: Before upgrading a Tanzu Kubernetes cluster to 1.19, reconfigure the Kubelet configuration file to point to the correct resolv.conf . Manually duplicate the ConfigMap kubelet-config-1.18 to kubelet-config-1.19 in the kube-system namespace then modify that new ConfigMap's data to point resolvConf at /run/systemd/resolve/resolv.conf .

When the Supervisor Cluster networking is configured with NSX-T, after updating a service from "ExternalTrafficPolicy: Local" to "ExternalTrafficPolicy: Cluster", requests made on the Supervisor Cluster control plane nodes to this service's load balancer IP fail

When you create a service on a Tanzu Kubernetes cluster with ExternalTrafficPolicy: Local and later updated the service to ExternalTrafficPolicy: Cluster , kube-proxy creates an IP table rule incorrectly on the Supervisor Cluster control plane nodes to block traffic destined to the service's LoadBalancer IP. For example, if this service has LoadBalancer IP 192.182.40.4, the following IP table rule is created on any one of the control plane nodes:

-A KUBE-SERVICES -d 192.182.40.4/32 -p tcp -m comment --comment "antrea-17-171/antrea-17-171-c1-2bfcfe5d9a0cdea4de6eb has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

As a result, access to that IP is dropped.

Workaround: Delete the service and create it anew with ExternalTrafficPolicy: Cluster .

After you enable HTTP Proxy and/or Trust settings in the TkgServiceConfiguration specification, all pre-existing clusters without Proxy/Trust settings will inherit the global Proxy/Trust settings when they are updated.

You can edit the TkgServiceConfiguration specification to configure the TKG Service, including specifying the default CNI, HTTP Proxy, and Trust certificates. Any configuration changes you make to the TkgServiceConfiguration specification apply globally to any Tanzu Kuberentes cluster provisioned or updated by that service. You cannot opt-out of the global configuration using per-cluster settings.

For example, if you edit the TkgServiceConfiguration specification and enable an HTTP Proxy, all new clusters provisioned by that cluster inherit those proxy settings. In addition, all pre-existing clusters without a proxy server inherit the global proxy configuration when the cluster is modified or updated. In the case of HTTP/S proxy, which supports per-cluster configuration, you can update the cluster spec with a different proxy server, but you cannot remove the global proxy setting. If the HTTP Proxy is set globally, you must either use it or overwrite it with a different proxy server.

Workaround : Understand that the TkgServiceConfiguration specification applies globally. If you don't want all clusters to use an HTTP Proxy, don't enable it at the global level. Do so at the cluster level.

In very large Supervisor Cluster deployments with many Tanzu Kubernetes Clusters and VMs, vmop-controller-manager pods might fail due to OutOfMemory resulting in the inability to lifecycle manage Tanzu Kubernetes Clusters

Within the Supervisor Cluster, the vmop-controller-manager pod is responsible for managing the lifecycle of the VMs that make up Tanzu Kubernetes Clusters. At very large numbers of such VMs (>850 VMs per Supervisor Cluster), the vmop-controller-manager pod can go into an OutOfMemory CrashLoopBackoff. When this occurs, lifecycle management of Tanzu Kubernetes Clusters are disrupted until the vmop-controller-manager pod resumes operations.

Reduce the total number Tanzu Kubernetes Cluster worker nodes managed in a Supervisor Cluster either by deleting clusters or scaling down clusters.

Upgrading from a pre-6.5 version of vSphere to vSphere 7 with Tanzu can throw an error indicating that the PhotonOS type is not supported.

After successfully upgrading a vSphere 6 environment to vSphere 7 with Tanzu, trying to deploy a Tanzu Kubernetes cluster results in an error message that PhotonOS is "not supported". Specifically:

failed to create or update VirtualMachine: admission webhook "default.validating.virtualmachine.vmoperator.vmware.com" denied the request: GuestOS not supported for osType vmwarePhoton64Guest on image v1.19.7---vmware.1-tkg.1.fc82c41'

If you have upgraded to vSphere 7 with Tanzu from a vSphere version that predates vSphere 6.5 (such as vSphere 6), you need to make sure that the default VM compatibility level is set to at least "ESXi 6.5 and later" for PhotonOS to show up as a supported guest operating system. To do this, select the vSphere cluster where Workload Management is enabled, right-click and choose Edit Default VM Compatibility. Select "ESXi 6.5 and later."

After provisioning a Tanzu Kubernetes cluster using a small size VM, and then deploying a workload to that cluster, the worker node enters a NotReady state and a continuous loop of attempting to respawn the node.

Description : On a small or extra small worker node for a cluster, the /bin/opm process may consume an inordinate portion of the VM memory, leading to an out-of-memory error for the worker node.

Workaround : Avoid using the small or extra small VM class for worker nodes, even for ephemeral development or test clusters. As a best practice, the minimum VM class for a worker node in any environment is medium. For more information, see Default Virtual Machine Classes in the documentation.

Synchronizing Tanzu Kubernetes releases from a Subscribed Content Library fails with the following HTTP request error message: "cannot authenticate SSL certificate for host wp-content.vmware.com."

Description : When you configure a Subscribed Content Library for Tanzu Kubernetes cluster OVAs, an SSL certificate is generated for wp-content.vmware.com. You are prompted to manually trust the certificate by confirming the certificate thumbprint. If the SSL certificate is changed after the initial library setup, the new certificate must be trusted again by updating the thumbprint. The current SSL certificate for the Content Library expires month-end June 2021. Customers who subscribed to wp-content.vmware.com will see their Content Library synchronization failing, and need to update the thumbprint by performing the steps in the Workaround. For additional guidance, see the associated VMware Knowledge Base article at https://kb.vmware.com/s/article/85268 .

Workaround : Log in to the vCenter Server using the vSphere Client. Select the Subscribed Content Library and click Edit Settings . This action will initiate a probe of the subscription URL even though no change is requested on the library. The probe will discover that the SSL certificate is not trusted and prompt you to trust it. In the Actions drop-down menu that appears, select Continue and the fingerprint is updated. Now you can proceed with synchronizing the contents of the library.

Customers may see failures after applying an update to the cluster involving both a VMClass modification and a node scale up.

Please modify the TKC configuration to only modify one of the two fields and reapply the change.

Symptom: An unexpected rolling update of the cluster occurred after updating a label or a taint in the cluster spec.

Description: When using the TKGS v1alpah2 API, modifying spec.topology.nodePools[*].labels or spec.topology.nodePools[*].taints of one node pool will trigger a rolling update of that node pool.

Workaround: None

Symptom: After a cluster update, the taints and labels manually added to a node pool are no longer present.

Description: Using the TKGS v1alpha2 API, during a cluster update the system does not retain spec.topology.nodePools[*].taints and spec.topology.nodePools[*].labels that were manually added and present before the update.

Workaround: After the update completes, manually add back the labels and taints fields.

Symptom: Tanzu Kubernetes cluster is stuck in a creating phase as one of the control plane VMs did not get an IP address.

Description:Tanzu Kubernetes cluster is stuck in a creating phase as one of the control plane VMs did not get an IP address.

Workaround: Use Tanzu Kubernetes release 1.20.7 or later to avoid this issue.

Symptom: During the creation of a Tanzu Kubernetes cluster, the process is stuck in an updating state with the reason "WaitingForNetworkAddress."

Description: The control plane VMs and worker nodes are powered on but are not have an IP assigned to them. This may be caused by vmware-tools running out of memory and not getting restarted.

Workaround: The specific issue is fixed in Tanzu Kubernetes releases v1.20.7 and later. In addition, you can fix the issue by increasing the VM memory. The best-effort-xsmall VM class provides only 2G of memory. Use a VM class with more memory to deploy Tanzu Kubernetes clusters.

Description: After deleting a vSphere namespace, the process is stuck in a "Removing" state for several hours without progressing.

Workaround: Use kubectl to check for remaining Kubernetes objects that haven't been deleted from the namespace yet. If there are remaining objects related to kubeadm-control-plane, restart the capi-kubeadm-control-plane-controller-manager pods. This action should requeue the objects to be deleted.

NOTE: This is an advanced operation. Contact VMware Support before performing this workaround.

Starting with 7.0.3 MP2 release TKr < 1.18.x will be incompatible, but TKC API still allows creation of TKCs with version less than v1.18.x, it does not block the creation of a TKC object when it is created with an incompatible TKr. These TKC will never get created.

Delete TKCs which are not in a running state and were created with incompatible TKr. And recreate with a compatible version.

A vDPP UI plugin does not get deployed in the vSphere Client and an error message indicates that the plugin download was attempted from a Supervisor cluster that no longer exists

The problem might occur when you deploy a vDPP UI plugin on a cluster, and then remove this cluster. However, the stale entries related to this vDPP UI plugin remain in the vCenter Extension Manager. If you later create a new cluster, your attempts to install the same version of the vDPP UI plugin on this cluster fail because the vSphere client uses the stale entries to download the plugin from the old non-existing cluster.

Workaround:

A vSphere pod in a Supervisor Cluster remains in pending state without the vm-uuid annotation

Occasionally, a pod might remain in pending state for a long time. This typically occurs when you use a vSAN Data Persistence platform (vDPP), and can be caused by an internal error or user actions.

When you use the kubectl command to query the pod instance, the vmware-system-vm-uuid/vmware-system-vm-moid annotation is missing in the metadata annotations.

Workaround: Use the kubectl delete pod pending_pod_name command to delete the pod. If the pod is a part of a stateful set, the pod is recreated automatically.

An instance of a vDPP service fails to deploy when host local PVCs of its two replica pods are bound to the same cluster node

Occasionally, an instance of a vSAN Data Persistence Platform service, such as MinIO or Cloudian, might get into a state when its two replica pods have their host local PVCs allocated on the same cluster node. Normally, no two replicas of the same instance can have host local storage on the same cluster node. If this happens, one of the replica pods might remain in a pending state for an indefinite time causing the instance deployment failures.

The symptoms you observe during the failures include all of the following:

The failure scenarios that can lead to this problem are the following:

Workaround:

The service operator should rebuild the volume data on the new node where the pod gets scheduled. This can take some time to complete, depending on the size of the volume and amount of valid data on it.

After an ESXi node exits maintenance in Ensure accessibility mode, the taint applied to the node during the maintenance might still remain on the node

This issue might occur when you use vSAN Data Persistence Platform (vDPP) to create instances of partner services. After the ESXi node exits maintenance in Ensure accessibility mode, you can still find the remaining taint node.vmware.com/drain=planned-downtime:NoSchedule on the node.

Typically, the issue occurs when these actions take place:

1. A partner service is enabled on the Supervisor Cluster and instances of the service are created.

2. An ESXi node is put into maintenance in Ensure accessibility mode.

3. The node successfully enters maintenance mode in Ensure accessibility.

4. The taint node.vmware.com/drain=planned-downtime:NoSchedule is applied on the node.

5. The node exits maintenance mode.

After the node exits maintenance mode, use the following command to ensure that no taint remains on the node:

kubectl describe node | grep Taints

Workaround:

If the taint node.vmware.com/drain=planned-downtime:NoSchedule is present on the host, manually remove the taint:

kubectl taint nodes nodeName key=value:Effect -

Note: Make sure to use a hyphen at the end of the command.

Follow this example:

kubectl taint nodes wdc-10-123-123-1.eng.vmware.com node.vmware.com/drain=planned-downtime:NoSchedule-

After an APD recovery, persistent service pods that run on the vSAN Data Persistence platform might remain in pending state with AttachVolume errors

After ADP and recovery of ESXi hosts, a vDPP service instance pod might remain in pending state. If you run the kubectl -n instance_namespace describe pod name_of_pod_in_pending command, you can see the AttachVolume errors.

If the pod remains in pending state for more than 15 minutes, it is unlikely to come out of this state.

Workaround: Delete the pending pod using the following command: kubectl -n instance_namespace delete pod name_of_pod_in_pending . The pod is recreated and moves to running state.

<span style="color:#FF0000">NEW:</span> Attempts to deploy a vSphere Pod and a VM Service VM that have the same name cause errors and unpredictable behavior

You might observe failures and errors or other problematic behavior. You can experience these problems when you have a vSphere Pod running in a namespace and then try to deploy a VM Service virtual machine with the same name in the same namespace, or visa versa.

Workaround: Do not use the same names for the vSphere Pods and VMs in the same namespace.

When attempting to use a VM image that is not compatible with VM Service, the following message is encountered at VM creation

Error from server (GuestOS not supported for osType on image or VMImage is not compatible with v1alpha1 or is not a TKG Image): error when creating : admission webhook "default.validating.virtualmachine.vmoperator.vmware.com" denied the request: GuestOS not supported for osType on image or VMImage is not compatible with v1alpha1 or is not a TKG Image

Workaround: Only use VM images from the VMware Marketplace that have been validated to work with VM Service

If the name of an OVF image in a Content Library is not DNS-subdomain name compliant, VM Service will not create a VirtualMachineImage from the OVF image. As a result, kubectl get vmimage will not list the images in a namespace and developers will not be able to consume this image.

Workaround: Use DNS subdomain name compliant names for OVF images.

OVF images with the same name in different Content Libraries do not show up as different VirtualMachineImages

Workaround: Use unique names for OVF images across Content Libraries that are configured with VM Service.

Virtual Machines created by VM Service do not allow access to the remote console. As a result, administrators cannot log into the VM using the Launch Web Console and Launch Remote Console options in the vSphere web client.

Workaround: Administrators can access the VM console by logging into the ESXi host on which the VM is located.

VMs with vGPU and pass-through devices that are managed by VM Service are powered off when the ESXi host enters maintenance mode

VM Service allows DevOps engineers to create VMs that are attached to vGPU or pass-through devices. Normally, when an ESXi host enters maintenance mode, DRS generates a recommendation for such VMs that are running on the host. A vSphere administrator can then accept the recommendation to trigger the vMotion.

However, VMs with vGPU and pass-through devices that are managed by VM Service are automatically powered off. This may temporarily affect workloads running in these VMs. The VMs are automatically powered on after the host exists the maintenance mode.

Workaround: None.