Max virtual disk size that allows hotadd

[averylarry]

Case # 00154561

I'm expecting to hear back from Tier 2 (or above) as to the technical "why", but I thought maybe I could get a faster answer here.

I'm being told that hotadd has a smaller virtual disk size limit than VMware snapshot.

From here:
http://kb.vmware.com/selfservice/micros ... Id=1012384

the maximum file size of vmfs5 is 2TB-512B. But (and I found this out the hard way), you cannot snapshot a virtual disk that big. There is overhead associated with a snapshot. To almost be precise, it is a maximum overhead of about 16GB for a 2TB-512B virtual disk.

So if you want to snapshot a virtual disk, the actual maximum size is 2032GB. Straight from the chart referenced aboved. I created a 2030GB virtual disk, but Veeam tech support says it is too big for hotadd, even though I can manually take a snapshot and hotadd it.

So -- I would like to know the actual technical limit so I can (eventually, since I don't have 2TB of free space until I can buy a new SAN in January) create an appropriate virtual disk that Veeam can successfully hotadd.

[foggy]

Ted, have you tried to run Veeam jobs for this disk? I'm pretty sure that if you can hotadd it manually, Veeam can do that either.

The actual technical limit is referenced in the same article - it is 2,181,972,430,848 bytes for VMFS5 (which is ~2032GB).

[tsightler]

Yes, I agree. There were some technical limits with hotadd regarding size with earlier versions of the VDDK, however, with current versions you should be able to backup a VMDK of 2032GB.

However, it is very important to remember that, with vSphere 5.0 and above, by default, the snapshot delta's are stored on the same disk with the VMDK file. That means you need to have enough space not just for the initial snapshot overhead (16GB), but also enough to contain any changes that might be written to the disk while the snapshot is open. You can redirect the delta files to another disk, but this has some impact on features like sVmotion and storage DRS.

[averylarry]

Very well, I will wait and see what tier 2 tech support says.

[sergeadam]

For practical purposes, I create all my lage disks at 1.9TB. You do need enough free space to hold the snapshot.

[averylarry]

This is my reply from tech support:

We have concluded our tests. The limitation is 1.98 TB even.

1.98TB even is 2027.52 GB which worked successfully.
Once we incremented the VMDK to 2028 GB it failed with the same error.

This should be a concrete answer for the VDDK 5.1 limitation so you can use this when creating new disks you intend to backup using hot-add. Let me know if you need anything else or if you have further questions regarding this.

They also said direct san mode does not have this limitation (it's a limitation of the 32 bit vddk supplied by VMware).

I would consider this important enough to add to the documentation as a limitation of hotadd.

Further, as an enhancement request, it would be nice if hotadd wouldn't fail on ALL disks just because one of the disks is too big. Additionally, even if I exclude my 1.983 Tb disk from the backup job, the job still fails to hotadd (though it is implied that it would not fail if the disk was independent).

[averylarry]

PS -- I don't really think I should have been merged since this has nothing to do with snapshots, but with a separate vddk/hotadd limitation that is more restrictive than vmware snapshot limitations. The snapshot limitation is a red herring

[foggy]

I would consider this important enough to add to the documentation as a limitation of hotadd.

Ted, thanks for posting back the testing results. I'll ask support guys and technical writers to update the corresponding sources.

Further, as an enhancement request, it would be nice if hotadd wouldn't fail on ALL disks just because one of the disks is too big. Additionally, even if I exclude my 1.983 Tb disk from the backup job, the job still fails to hotadd (though it is implied that it would not fail if the disk was independent).

This is a known limitation of vStorage API: it is not possible to hotadd VM disks partially, all disks of the backed up VM are hotadded during its processing.

[averylarry]

Thank you. Now I just have to wait until next year when I buy a new SAN and then create a new (slightly smaller) virtual disk and manually copy all the data (using multiple passes of robocopy) and then carry my disaster recovery server back to my main office to re-sync the (almost) 2Tb of data.

Or maybe I'll try to squeeze a direct san proxy onto my not-enough-ports SAN switch somehow. Oh well, at least all the jobs are working well enough failing back to network mode.

[tsightler]

If your SAN supports iSCSI connectivity you could configure a virtual proxy to use Direct SAN access with the Microsoft iSCSI initiator. It's usually still faster than using network mode.

[averylarry]

If your SAN supports iSCSI connectivity you could configure a virtual proxy to use Direct SAN access with the Microsoft iSCSI initiator. It's usually still faster than using network mode.

Yes indeed. But I still need physical network cards, physical switch ports, and proper virtual networking across my cluster OR a physical server. I don't really have any of those things, and I don't want to try and piggy-back onto my software iSCSI networking in VMware.

Eventually I'll probably configure direct san mode -- but it'll likely have to wait until I get 10Gb SAN networking up and running first.