How to enable Changed Block Tracking for Guest Cluster on vSphere with Tanzu

Changed Block Tracking is a VMware feature that tracks changes in virtual disks. K10 uses this feature in vSphere with Tanzu Guest Clusters to efficiently backup Persistent Volumes.

Enabling Changed Block Tracking in each Supervisor Cluster where K10 will be used, is recommended.  Once it has been enabled on a Supervisor Cluster all Guest Clusters will be automatically configured for CBT.

CBT is normally enabled on virtual machines via the vCenter UI, however security constraints in the vSphere with Tanzu Supervisor Cluster prevent this.  The Velero vSphere Operator is used as the special path to enable this feature on vSphere with Tanzu clusters. Kasten K10 doesn’t use Velero to perform actual backup, it is only required in order to enable the Changed Block Tracking feature on VMs provisioned via vSphere with Tanzu.

 

Prerequisites:

vSphere 7.0.3

Tanzu Advanced

Linux system with network connectivity to vCenter

 

Steps:

Enable the Velero vSphere Operator

Log in to the vCenter UI

Select the Supervisor Cluster

Go to the “Configure” tab

Go to Services under Supervisor Services

Enable the Velero vSphere Operator

 

It will prompt for a repository endpoint, username and password. For an "air-gapped" environment using a private repository for containers, supply the parameters here.  To download the necessary containers from the Internet, leave the fields blank.

 

Patch VM Operator config map

In 7.0.3 the VM Operator does not allow the Velero vSphere Operator service account to enable changed block tracking on VMs.  To use CBT the vmware-system-tkg-system-service-accounts config map needs to be modified.

 

1. Determine the namespace of the Velero vSphere Operator

From the vCenter UI, the Velero vSphere Operator namespace will be visible as a namespace in the Supervisor cluster.  The name will begin with svc-velero-vsphere

The namespace can also be found using kubectl from the Supervisor Cluster.

This information will be used in step 4.

kubectl get ns | grep 'svc-velero-vsphere'

 

2. Login to vCenter Service Appliance using ssh.

If ssh access to VCSA is not enabled, please follow the instructions here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-D58532F7-E48C-4BF2-87F9-99BA89BF659A.htm

 

3. Decrypt the password for the Supervisor Control Plane VM

/usr/lib/vmware-wcp/decryptK8Pwd.py

The output will show the password and IP address of the Supervisor Control Plane VM.  For multiple Supervisor Clusters, select the appropriate one from the list.

Read key from file

Connected to PSQL

Cluster: domain-c106:6519bc6a-e519-4df9-84e1-da0a75598c05

IP: <Supervisor Control Plane IP>

PWD: <Supervisor Control Plane VM password>

Connect over SSH to the Supervisor Control Plane VM (This can be performed from the VCSA shell or another system that has connectivity).

Use the password obtained above to login.

ssh root@<Supervisor Control Plane IP>

 

4. Add the default service account for the Velero vSphere Operator’s namespace to the config map.  Use the Velero vSphere Operator namespace name obtained in step 1.

kubectl patch -n vmware-system-tkg cm vmware-system-tkg-system-service-accounts --type merge -p '{"data":{"system.serviceaccount.<velero operator namespace>.default": "true"}}'

 

5. Restart the TKG Controller Manager

 

kubectl rollout restart deployment vmware-system-tkg-controller-manager -n vmware-system-tkg

 

Login to vSphere with Tanzu

Follow the instructions here on logging into the vSphere with Tanzu using the kubectl command and vSphere plugin.

https://docs.vmware.com/en/VMware-vSphere/7.0/vmware-vsphere-with-tanzu/GUID-F5114388-1838-4B3B-8A8D-4AE17F33526A.html

 

Install the Velero vSphere Operator CLI

Instructions on installing are here: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/docs/supervisor.md#install

This requires an Intel based Linux system

 

Enable Changed Block Tracking (CBT) by installing Velero with the appropriate options

K10 only leverages Velero to interact with the Velero vSphere Operator, to be able to enable CBT, and does not use Velero for any part of the backup or restore process.

 

Create a namespace for the tool via the vSphere UI:

  • Go to Workload Management, select the Namespaces tab
  • Click “New Namespace”
  • Using the vsphere-backup-infra namespace for this install is recommended, as this is not going to be a fully configured Velero install and will not be used (or work) for any actual backup operations using Velero.

 

Run the below velero-sphere command to enable CBT on guest cluster VMs:

 

velero-vsphere install --namespace vsphere-backup-infra --version v1.5.1 --plugins vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0 --no-secret --use-volume-snapshots=false --no-default-backup-location --enable-cbt-in-guests

 

This “installation” of Velero is only used to enable CBT.  This is the minimum install with only the Velero Plugin for vSphere and no other options. 

 

If Velero is already installed in Supervisor Cluster, enable CBT with this command, where <guest cluster namespace> is the name of the Guest Cluster. This commands needs to be executed while connected to the Supervisor Cluster:

 

velero-vsphere configure --enable-cbt-in-guests -n <guest cluster namespace>

 

 

Verify that Changed Block Tracking (CBT) has been enabled on Guest Cluster VMs

A Guest Cluster runs on multiple VMs managed by the Supervisor Cluster.  Verify that CBT has been turned on for those VMs by checking the status.changeblocktracking field of the virtualmachine resource in the Supervisor Cluster namespace where the Guest Cluster was created.

 

While connected to the Supervisor cluster:

1. List VMs

kubectl get -n <guest cluster namespace> virtualmachine

 

2. Check the virtual machines

kubectl get -n <guest cluster namespace> virtualmachine <guest cluster worker node VM name> -o jsonpath='{.status.changeBlockTracking}'

This will return true if Changed Block Tracking has been successfully enabled on the VM.