Comprehensive data protection for all workloads
Post Reply
m.novelli
Veeam ProPartner
Posts: 355
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Slow backup of empty disk

Post by m.novelli » Sep 05, 2012 1:13 pm

I'm facing a strange issue with a customer: he has many VM backupped with Veeam, and two of them have big disks (1 TB each) almost empty now. Veeam take 4 hours to backup each of those two VM, while should be lightning fast because the disk are empty.
I've run sdelete on the disks and performed an active full backup a couple of times, but the backup performance are sooooo slow.

Infrastructure: Veeam Backup 6.1 with patch 1 and Vmware vSphere 5.0 U1

Any ideas guys? Many thanks for your time

Marco

foggy
Veeam Software
Posts: 18264
Liked: 1561 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow backup of empty disk

Post by foggy » Sep 05, 2012 1:25 pm

Marco, what does the bottleneck statistics say for those jobs? What kind of VMs are backed up, what transport mode is used? Please, describe the setup more precisely.

m.novelli
Veeam ProPartner
Posts: 355
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Slow backup of empty disk

Post by m.novelli » Sep 05, 2012 2:08 pm

Bottleneck source, about 65-75 MB/s active full backup of the empty disks

One VM is an Oracle databaese: Busy: Source 99% > Proxy 27% > Network 0% > Target 0%
One VM is DC + file server: Busy: Source 99% > Proxy 21% > Network 0% > Target 0%

Transport mode is network

Source server is a Dell T710 with RAID10 SATA storage, dual LAN 1 Gbit connected. Controller Dell H700 with 1 GB cache. Dual processor Intel E5630

Destination server is a Dell R310 with RAID5 disks, dual LAN 1 Gbit connected. Controller Dell H700 with 512 MB cache. Single processor Intel X3440

Marco

tsightler
VP, Product Management
Posts: 5420
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow backup of empty disk

Post by tsightler » Sep 05, 2012 2:42 pm

Well, we still have to read all of the data, even if it's zero. What's the performance of his disks? You mention the controller, but not the number and expected speed of the disks. Based on the bottleneck stats we're retrieving the data as fast as we can and spend most of the time waiting on the disks. Is your proxy able to access the same datastore for hotadd? If not we are falling to network mode so that's probably about the speed I'd expect.

m.novelli
Veeam ProPartner
Posts: 355
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Slow backup of empty disk

Post by m.novelli » Sep 05, 2012 3:06 pm

Ah, I supposed that even a full active backup was skipping the empty space on a VMDK. Don't know why but I was sure about that :-)

Marco

Gostev
SVP, Product Management
Posts: 24793
Liked: 3540 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup of empty disk

Post by Gostev » Sep 05, 2012 3:51 pm

That is because you heard it from me, and right on these forums :D there is a topic around here where are explain leveraging CBT for full backups.

Active full backup does indeed skip zeroed blocks, this is one functionality CBT still provides even for full image runs. However, this functionality does not always work reliably (VMware issue), in which case our code has to read the complete image. For more details, please look up the existing topic.

tsightler
VP, Product Management
Posts: 5420
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow backup of empty disk

Post by tsightler » Sep 05, 2012 5:03 pm

So this is probably semantics, but I'd argue that we don't really skip "zero" blocks even in this case, we skip blocks in the VMDK that have never been written, thus they contain zeros. If you create a 1TB VMDK and immediately back up up, Veeam will skip pretty much the entire disk. On the other hand, if you create a 1TB VMDK, immediately write a bunch of zeros to it with a tool like SDELETE or DD, and then back it up, we'll read the entire disk because we don't know that you simply wrote a bunch of zeros to the disk, we only know that the blocks were written to. We will read these blocks from the VMDK, see that they contain zeros, and then "skip" any further processing of them, but from a VMware perspective they are allocated and thus we have to read them to find out what is in them.

Based on your original post, I assumed that these disks did, at one time, have data on them since you said they were "almost empty now" and that you ran SDELETE against them. Running SDELETE will "zero" the blocks, but we'll have to read them to see that they are zero.

m.novelli
Veeam ProPartner
Posts: 355
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Slow backup of empty disk

Post by m.novelli » Sep 06, 2012 7:42 am

tsightler, your argument is pretty logical, but I suppose Gostev is right, Vmware / Veeam in 2012 should be smart enough to skip zeroed blocks :-)

Maybe upgrading to vSphere 5.1 and resetting CBT will solve this odd behavior

Marco

tsightler
VP, Product Management
Posts: 5420
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow backup of empty disk

Post by tsightler » Sep 06, 2012 11:53 am

I think I read that you can storage vMotion the host and it will "correct" the issue, but I don't believe that any other method works because no other method can know for sure that the blocks are truly zero without reading them first (the storage vMotion reads them as well but "skips" them if they are zero).

Gostev
SVP, Product Management
Posts: 24793
Liked: 3540 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup of empty disk

Post by Gostev » Sep 06, 2012 6:34 pm

No, the issue I am talking about will not be corrected by Storage VMotion, but rather by complex procedure requiring powering off VM a few times (I don't remember the specifics by now).

deduplicat3d
Expert
Posts: 100
Liked: 11 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: Slow backup of empty disk

Post by deduplicat3d » Sep 07, 2012 1:05 am

If anyone is interested, here is the topic

Post Reply

Who is online

Users browsing this forum: supremejumping and 26 guests