Full backup speed decreases as it runs

VMware specific discussions

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Mon Jan 23, 2012 6:03 pm

If I use a Linux target instead of a Windows target the job runs correctly without the decreasing speed. I would perfer to use a Winodws target though.
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Jan 23, 2012 9:13 pm

Hmm, I would not expect the target type to matter, because the data processing agent is cross-platform (same code). What really matters is the performance of storage you are writing to. However, if both Linux and Windows server have exactly identical hardware, then I would say you should investigate this with our support. Thanks!
Gostev
Veeam Software
 
Posts: 21396
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Tue Jan 24, 2012 4:38 pm

same hardware, I removed Windows and put Linux on it
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby mbutch » Tue Jan 24, 2012 10:01 pm

Wow, it must be something with the CBT. I followed those instructions, and after 40 minutes I'm still at 84MBps. That's awesome.

What is not is the fact that I know how to do this to each VM.
mbutch
Novice
 
Posts: 3
Liked: never
Joined: Sun Jan 22, 2012 4:55 pm
Full Name: Matthew Butch

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Wed Jan 25, 2012 2:41 pm

In my case, these jobs are a full backup, CBT shouldn't be affecting them, I don't think?
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby tsightler » Wed Jan 25, 2012 2:49 pm

Actually Veeam still attempts to use CBT for full backups to detect unused areas of the disk, etc. Also, even during a full backup the snapshot CBT file has to be updated so, while it does feel like a long shot, it might be worth a try since other seem to consistently see improvements.
tsightler
Veeam Software
 
Posts: 4769
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Wed Jan 25, 2012 2:52 pm

tsightler wrote:Actually Veeam still attempts to use CBT for full backups to detect unused areas of the disk, etc. Also, even during a full backup the snapshot CBT file has to be updated so, while it does feel like a long shot, it might be worth a try since other seem to consistently see improvements.

All I did was switch the destination storage from Windows to Linux and the problem went away, so CBT wouldn't be the issue than right?
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby tsightler » Wed Jan 25, 2012 4:33 pm

I would generally agree with that statement, but seeing multiple users report that it helped them make it worth it even as a long shot (perhaps a very long shot).
tsightler
Veeam Software
 
Posts: 4769
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Full backup speed decreases as it runs

Veeam Logoby wahrheit2004 » Mon Feb 27, 2012 7:47 am

Hi,

I have the same issue with direct SAN (Network Failback disabled) Veeam B&R 6.0 Patch 3 and KB1113 (How to reset CBT) fix.
I am Backing up a 4,9TB VM running "full backup" (Win 2008 R2 with 13 virtual Disks). SAN is connected by 2x 10GBit/s iSCSI (1 per controller), Backup Machine is
Physical Server Win2008 R2 with Backup proxy, accessing via MPIO iSCSI connected by 2x 10GBit/s to SAN. Server has 32GB RAM, 2x 4 Core XEON 2,66 Ghz.
Backup ist stored to DAS (16x 1TB SATA RAID 6). If I copy files from SAN to DAS I have 650MB/s.

Realtimestatistics says Bottleneck "target" but CPU is only used @ 1core and here 80% kernel time.
Current backup speed is slow 15MB/s.

Backing up a 2nd VM (Win2008, 1HDD, 167GB) as "full backup" took "just" 10 minutes and 2 sec.

I opened a Veeam Level 3 ticket @ ID 5175226 and sent them my logs.
wahrheit2004
Influencer
 
Posts: 10
Liked: 2 times
Joined: Mon Feb 27, 2012 7:33 am
Full Name: DW

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Feb 27, 2012 9:01 am

wahrheit2004 wrote:Realtimestatistics says Bottleneck "target" but CPU is only used @ 1core and here 80% kernel time.

Actually, "target" means the bottleneck is the target storage write speed. Nothing to deal with CPU load.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby wahrheit2004 » Tue Feb 28, 2012 7:18 am

Yes and no. what I did to verify is to copy a 4.3GB DVD ISO Image to the same lokal Volume Veeam uses for it's backups.
Copying the .ISO from a remote LAN_Server with a 1GBit/s LAN Card I reach 60MB/s write.
If I duplicate it from local volume to local volume my speed is 46MB/s in write, handling read and write I/O from same Volume.

Btw.:
From my backup-job above the Real-Time statistics show 57% @ 10MB/s - Duration 45hrs 47min currently.
wahrheit2004
Influencer
 
Posts: 10
Liked: 2 times
Joined: Mon Feb 27, 2012 7:33 am
Full Name: DW

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Tue Feb 28, 2012 9:34 am

Not sure what you mean by "yes and no". Regarding your test, you said your backup speed decreases with time, backing up very large files. So copying small 4.3GB ISO file won't really show anything?

Anyway, as far as I know your support case is being looked at.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby wahrheit2004 » Wed Feb 29, 2012 7:19 am

Hi Gostev,

Gostev wrote:So copying small 4.3GB ISO file won't really show anything?


I generated a 2,19 TB File in the volume of the Veeambackupfiles by using this command:
fsutil file createnew test.txt 2242880000000

Than I copied this file in same folder to "test - Kopie.txt" using windows Explorer.
Detail of copy process shows 2xxMB/s and Ressource manager shows 290MB/s in peeks up to 3xxMB/s.

Image

My 4,9 TB Veeam backup process is at 72% @ 8MB/s duration 69hrs 44 min...
Herefor I have a webex meeting tomorrow, hope we can find the issue.
wahrheit2004
Influencer
 
Posts: 10
Liked: 2 times
Joined: Mon Feb 27, 2012 7:33 am
Full Name: DW

Re: Full backup speed decreases as it runs

Veeam Logoby wahrheit2004 » Thu Mar 01, 2012 1:39 pm

Hi everyone,

I had the webex meeting today with veeam support. They identified it that is a issue deep in their B&R software.
Now developers should take it for deeper investigation and hopefully a fix soon.

What you can see ist, that the 1st HDDs are processed at a good speed, but than dramatically drop - see here:
Image

This is by running a full-backup. I hope that the reverse-incrmental will be processed a lot faster :D .

So waiting for fix 4 or 5....
wahrheit2004
Influencer
 
Posts: 10
Liked: 2 times
Joined: Mon Feb 27, 2012 7:33 am
Full Name: DW

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Fri Mar 02, 2012 9:30 am

wahrheit2004 wrote:They identified it that is a issue deep in their B&R software.

News to me! I am on the same email thread between support and devs regarding the webex with you (already saw this picture above). At this time, the opinion is nothing but speculation of a specific support engineer. Multiple tests on very large VMs in our own labs could not reproduce this behavior, however it is true however that developers will take it for deeper investigation ;)
Gostev
Veeam Software
 
Posts: 21396
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot], DGrinev and 29 guests