Changed Block Tracking and vRDM support

[mdornfeld]

I get the following "notice" when I back up an SQL DB VM. It reads as follows:

***********************
Verifying changed block tracking... One or more VM disks have incorrect changed block tracking configuration. To resolve this, open VMware Infrastructure Client, right-click the VM, choose Edit Settings, Options tab, select General, click Configuration Parameters, and set all entries with ‘ctkEnabled’ substring to false. Veeam Backup will then automatically re-enable changed block tracking with the correct settings during the next job run. Backing up file "[VM_OS_APP4] wkshptdb01/wkshptdb01-flat.vmdk" Unable to establish direct connection to the shared storage (SAN). Please ensure that: - HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly. - SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server. - Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation). Direct SAN connection is not available, failing over to network mode... Backing up file "[VM_OS_APP4] wkshptdb01/wkshptdb01_1-flat.vmdk" Unable to establish direct connection to the shared storage (SAN). Please ensure that: - HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly. - SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server. - Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation). Direct SAN connection is not available, failing over to network mode... Backing up file "[VM_OS_APP4] wkshptdb01/wkshptdb01_2-rdm.vmdk" Unable to establish direct connection to the shared storage (SAN). Please ensure that: - HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly. - SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server. - Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation). Direct SAN connection is not available, failing over to network mode...
**********************

The test Veeam server (Beta3) is a VM, without direct access to the SAN, so the second part of the error makes sense, but the first part concerns me. For the part that reads "open VMware Infrastructure Client, right-click the VM, choose Edit Settings, Options tab, select General, click Configuration Parameters, and set all entries with ‘ctkEnabled’ substring to false.". I can't just turn this VM off and change the setting right now, but I looked in another VM that I could turn off and it didn't have any settings called "ctkEnabled". I should mention that both the VM and the test VM I tried both had Virtual RDM's. I am running vSphere4 on VM Hardware 7.

- Is this setting only in some VM's?
- What causes the setting to be there?
- What am I all affecting by changing this setting?

Thanks Anton,
Matt

[Gostev]

1. This setting is present and enabled in all VMs for which you want to be able to use ESX4 changed block tracking functionality.
2. This setting is automatically enabled (set to TRUE) for all VMs included in the Veeam Backup job automatically when VM is first added to the job (during the first backup cycle for this VM).
3. By disabling it as we request in the message above, you take VM configuration to the state when Veeam Backup can correctly enable it again. This is only needed in some special situations, when certain combination of settings in these parameters make the VMware API call that enables changed block tracking for a VM fail to do it properly. Generally, you will almost never see this message, for all "clean" VMs Veeam Backup will enable changed block tracking settings silently for you.

Thanks!

[Gostev]

By the way, changed block tracking is not supported on vRDM with BETA3. It is supported in build 77, but there are 2 generations of vRDM disks, v1 and v2, and changed block tracking is only supported for v2 vRDMs. There should be an error about changed block tracking logged for v1 vRDM disks (in build 77).

[mdornfeld]

Oh oh...
How I can tell which version my RDM's are? If they are v1, is there a way to update them to v2?

[Gostev]

Good question

[Gostev]

So... there is a way, but it inlvolves browsing deep through ESX MOB (managed object browser) and I would not try to repeat the sequence of click even after I saw devs doing this a few times.

Nevertheless, with build 77 you should get special error saying that disk is not supported for changed block tracking, this would mean that the disk is v1. In that case, you would need to re-create the disk with vSphere VIC. It appears that new disks are created as v2.

If your virtual disk were created before you installed vSphere, then most likely it is v1 vRDM.

[mdornfeld]

I'm not sure if crying is allowed on forums... but if it is, I am. That sounds like a horribly bad project coming my way.

Regardless, thanks for the info and I still very much appreciate that Veeam is supporting v2 RDM's for change block tracking. I'm counting on this functionality to achieve RPO's.

[Gostev]

I hear you... but unfortunately it is VMware limitation - similarly to regular VM disks where changed block tracking requires the latest virtual hardware version 7.

[Gostev]

This thread was moved here from the closed beta forum.

[mdornfeld]

So I opened a case with VMware tech support, and they of course had no idea that there we multiple versions of RDM's. I presented them the information that Anton gave me... so they're going to research.

I have run into the problem, as expected, where I get this error in Veeam 4.0 "Verifying changed block tracking... Disk "Hard disk 3" does not support changed block tracking." Hard Disk 3 is of course a vRDM.

The only thing, is I created this VM and virtual RDM AFTER I was already on vSphere. It must have something to do with how I created or formatted the vRDM or something.

I guess from here I'm going to create some RDM's of a whole 1 MB each, and see which ones Veeam 4.0 thinks are v2 and which ones Veeam thinks are v1, because I have no other way of telling. While I realize this is no longer Veeam's issue, any help you could provide would be invaluable.

[Gostev]

Matt, as long as it is connected with with Veeam Backup any how, it is partially our issue too
I will send you more technical details directly now, including links to VMware documentation, so you could share those with VMware.

[mdornfeld]

If anybody runs across this post with the same issue, I have been working with Veeam to resolve this. Currently vRDM's are not supported with CBT by Veeam for their initial Veeam 4.0 release. Veeam developement is working on incorporating this functionality for a future release, which if they don't know already, I very much appreciate their efforts in doing so.

Thank you Veeam.

[Gostev]

According to the latest build's history the required changes were made to the code, so let's see what our QC folks have to say about this now

[mdornfeld]

Yay! I'll even pull out Mr. Green for this

Also, I think the API components update from VMware explains a lot of my earlier confusion. Thank your dev's for looking into this again.

[mdornfeld]

Anybody looking at this shoud reference:
http://www.veeam.com/forums/viewtopic.php?f=15&t=2347

vRDM CBT is in the 4.1 release.

[Gostev]

Matt, actually this is hidden forum accessible by focus group participants only, but I will copy your post here:

Veeam Backup 4.1 Beta1
I'm running the Veeam Beta inside a VM, over network mode, so my results should be much much better on my actual server. Note that the server I backed up is test server with hardly anything on it, but it gets the point across. Regardless, I'm seeing the results that are very wonderful.

Did a FLR restore and that works. Tomorrow I'll do a full restore of the server and see where that gets me.

1st Backup
wkcrmdb02
Success
11/30/2009 12:44:17 PM
11/30/2009 3:53:30 PM
7 7 125.00 GB 125.00 GB
11 MB/s
3:09:12

2nd Backup
wkcrmdb02
Success
11/30/2009 4:11:02 PM
11/30/2009 4:13:59 PM
7 7 125.00 GB 125.00 GB
722 MB/s
0:02:57

[MattG]

I am trying to backup a VM with a C: vmdk and a 2TB vRDM E: drive. The C: drive backs up with CBT, the E: drive backs up without CBT.

How can I get CBT to backup the vRDM drive?

Thanks,
-Matt

[Vitaliy S.]

Matt, there is no special configuration for making CBT work with vRDM disks, let our technical team look at your log files.