Slow backup of empty disk

[m.novelli]

I'm facing a strange issue with a customer: he has many VM backupped with Veeam, and two of them have big disks (1 TB each) almost empty now. Veeam take 4 hours to backup each of those two VM, while should be lightning fast because the disk are empty.
I've run sdelete on the disks and performed an active full backup a couple of times, but the backup performance are sooooo slow.

Infrastructure: Veeam Backup 6.1 with patch 1 and Vmware vSphere 5.0 U1

Any ideas guys? Many thanks for your time

Marco

[foggy]

Marco, what does the bottleneck statistics say for those jobs? What kind of VMs are backed up, what transport mode is used? Please, describe the setup more precisely.

[m.novelli]

Bottleneck source, about 65-75 MB/s active full backup of the empty disks

One VM is an Oracle databaese: Busy: Source 99% > Proxy 27% > Network 0% > Target 0%
One VM is DC + file server: Busy: Source 99% > Proxy 21% > Network 0% > Target 0%

Transport mode is network

Source server is a Dell T710 with RAID10 SATA storage, dual LAN 1 Gbit connected. Controller Dell H700 with 1 GB cache. Dual processor Intel E5630

Destination server is a Dell R310 with RAID5 disks, dual LAN 1 Gbit connected. Controller Dell H700 with 512 MB cache. Single processor Intel X3440

Marco

[tsightler]

Well, we still have to read all of the data, even if it's zero. What's the performance of his disks? You mention the controller, but not the number and expected speed of the disks. Based on the bottleneck stats we're retrieving the data as fast as we can and spend most of the time waiting on the disks. Is your proxy able to access the same datastore for hotadd? If not we are falling to network mode so that's probably about the speed I'd expect.

[m.novelli]

Ah, I supposed that even a full active backup was skipping the empty space on a VMDK. Don't know why but I was sure about that

Marco

[Gostev]

That is because you heard it from me, and right on these forums there is a topic around here where are explain leveraging CBT for full backups.

Active full backup does indeed skip zeroed blocks, this is one functionality CBT still provides even for full image runs. However, this functionality does not always work reliably (VMware issue), in which case our code has to read the complete image. For more details, please look up the existing topic.

[tsightler]

So this is probably semantics, but I'd argue that we don't really skip "zero" blocks even in this case, we skip blocks in the VMDK that have never been written, thus they contain zeros. If you create a 1TB VMDK and immediately back up up, Veeam will skip pretty much the entire disk. On the other hand, if you create a 1TB VMDK, immediately write a bunch of zeros to it with a tool like SDELETE or DD, and then back it up, we'll read the entire disk because we don't know that you simply wrote a bunch of zeros to the disk, we only know that the blocks were written to. We will read these blocks from the VMDK, see that they contain zeros, and then "skip" any further processing of them, but from a VMware perspective they are allocated and thus we have to read them to find out what is in them.

Based on your original post, I assumed that these disks did, at one time, have data on them since you said they were "almost empty now" and that you ran SDELETE against them. Running SDELETE will "zero" the blocks, but we'll have to read them to see that they are zero.

[m.novelli]

tsightler, your argument is pretty logical, but I suppose Gostev is right, Vmware / Veeam in 2012 should be smart enough to skip zeroed blocks

Maybe upgrading to vSphere 5.1 and resetting CBT will solve this odd behavior

Marco

[tsightler]

I think I read that you can storage vMotion the host and it will "correct" the issue, but I don't believe that any other method works because no other method can know for sure that the blocks are truly zero without reading them first (the storage vMotion reads them as well but "skips" them if they are zero).

[Gostev]

No, the issue I am talking about will not be corrected by Storage VMotion, but rather by complex procedure requiring powering off VM a few times (I don't remember the specifics by now).

[deduplicat3d]

If anyone is interested, here is the topic