Changed Block Tracking is a VMware feature that tracks changes in virtual disks. K10 uses this feature in vSphere with Tanzu Guest Clusters to efficiently backup Persistent Volumes.
Enabling Changed Block Tracking in each Supervisor Cluster where K10 will be used, is recommended. Once it has been enabled on a Supervisor Cluster all Guest Clusters will be automatically configured for CBT.
CBT is normally enabled on virtual machines via the vCenter UI, however security constraints in the vSphere with Tanzu Supervisor Cluster prevent this. The Velero vSphere Operator is used as the special path to enable this feature on vSphere with Tanzu clusters. Kasten K10 doesn’t use Velero to perform actual backup, it is only required in order to enable the Changed Block Tracking feature on VMs provisioned via vSphere with Tanzu.
Prerequisites:
vSphere 7.0.3
Tanzu Advanced
Linux system with network connectivity to vCenter
Steps:
Enable the Velero vSphere Operator
Log in to the vCenter UI
Select the Supervisor Cluster
Go to the “Configure” tab
Go to Services under Supervisor Services
Enable the Velero vSphere Operator
It will prompt for a repository endpoint, username and password. For an "air-gapped" environment using a private repository for containers, supply the parameters here. To download the necessary containers from the Internet, leave the fields blank.
Patch VM Operator config map
In 7.0.3 the VM Operator does not allow the Velero vSphere Operator service account to enable changed block tracking on VMs. To use CBT the vmware-system-tkg-system-service-accounts config map needs to be modified.
1. Determine the namespace of the Velero vSphere Operator
From the vCenter UI, the Velero vSphere Operator namespace will be visible as a namespace in the Supervisor cluster. The name will begin with svc-velero-vsphere
The namespace can also be found using kubectl from the Supervisor Cluster.
This information will be used in step 4.
kubectl get ns | grep 'svc-velero-vsphere'
2. Login to vCenter Service Appliance using ssh.
If ssh access to VCSA is not enabled, please follow the instructions here: https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenter.configuration.doc/GUID-D58532F7-E48C-4BF2-87F9-99BA89BF659A.html
3. Decrypt the password for the Supervisor Control Plane VM
/usr/lib/vmware-wcp/decryptK8Pwd.py
The output will show the password and IP address of the Supervisor Control Plane VM. For multiple Supervisor Clusters, select the appropriate one from the list.
Read key from file
Connected to PSQL
Cluster: domain-c106:6519bc6a-e519-4df9-84e1-da0a75598c05
IP: <Supervisor Control Plane IP>
PWD: <Supervisor Control Plane VM password>
Connect over SSH to the Supervisor Control Plane VM (This can be performed from the VCSA shell or another system that has connectivity).
Use the password obtained above to login.
ssh root@<Supervisor Control Plane IP>
4. Add the default service account for the Velero vSphere Operator’s namespace to the config map. Use the Velero vSphere Operator namespace name obtained in step 1.
kubectl patch -n vmware-system-tkg cm vmware-system-tkg-system-service-accounts --type merge -p '{"data":{"system.serviceaccount.<velero operator namespace>.default": "true"}}'
5. Restart the TKG Controller Manager
kubectl rollout restart deployment vmware-system-tkg-controller-manager -n vmware-system-tkg
Login to vSphere with Tanzu
Follow the instructions here on logging into the vSphere with Tanzu using the kubectl command and vSphere plugin.
Install the Velero vSphere Operator CLI
Instructions on installing are here: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/docs/supervisor.md#install
This requires an Intel based Linux system
Enable Changed Block Tracking (CBT) by installing Velero with the appropriate options
K10 only leverages Velero to interact with the Velero vSphere Operator, to be able to enable CBT, and does not use Velero for any part of the backup or restore process.
Create a namespace for the tool via the vSphere UI:
- Go to Workload Management, select the Namespaces tab
- Click “New Namespace”
- Using the vsphere-backup-infra namespace for this install is recommended, as this is not going to be a fully configured Velero install and will not be used (or work) for any actual backup operations using Velero.
Run the below velero-sphere command to enable CBT on guest cluster VMs:
velero-vsphere install --namespace vsphere-backup-infra --version v1.5.1 --plugins vsphereveleroplugin/velero-plugin-for-vsphere:1.1.0 --no-secret --use-volume-snapshots=false --no-default-backup-location --enable-cbt-in-guests
This “installation” of Velero is only used to enable CBT. This is the minimum install with only the Velero Plugin for vSphere and no other options.
If Velero is already installed in Supervisor Cluster, enable CBT with this command, where <guest cluster namespace> is the name of the Guest Cluster. This commands needs to be executed while connected to the Supervisor Cluster:
velero-vsphere configure --enable-cbt-in-guests -n <guest cluster namespace>
Verify that Changed Block Tracking (CBT) has been enabled on Guest Cluster VMs
A Guest Cluster runs on multiple VMs managed by the Supervisor Cluster. Verify that CBT has been turned on for those VMs by checking the status.changeblocktracking field of the virtualmachine resource in the Supervisor Cluster namespace where the Guest Cluster was created.
While connected to the Supervisor cluster:
1. List VMs
kubectl get -n <guest cluster namespace> virtualmachine
2. Check the virtual machines
kubectl get -n <guest cluster namespace> virtualmachine <guest cluster worker node VM name> -o jsonpath='{.status.changeBlockTracking}'
This will return true if Changed Block Tracking has been successfully enabled on the VM.