Full backup speed decreases as it runs

VMware specific discussions

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Sun Mar 04, 2012 8:15 pm

Just to provide an update - our QC had completed an incredible testing effort backing up VMs with multiple immense disks in the past few days, and we could not confirm the issue described above. My inbox is full of reports with graphs, and there is clearly no speed reduction at all. We are not giving up though, and are determined to find the explanation.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Mon Mar 05, 2012 2:23 pm

Gostev wrote:Just to provide an update - our QC had completed an incredible testing effort backing up VMs with multiple immense disks in the past few days, and we could not confirm the issue described above. My inbox is full of reports with graphs, and there is clearly no speed reduction at all. We are not giving up though, and are determined to find the explanation.


What were they using for a target? It only happened for me using a Windows target using the VeeamAgent (not CIFS). The problem went away when using a Linux target. By the way, I never did try the CBT fix and to see if that made a difference.

Thanks,
Wesley
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Mar 05, 2012 3:59 pm

We tested with the locally attached storage, because this is what the previous poster (who we had a webex with) had the issue with. Obviously we did not want to complicate the testing with more "variables"...

From the graphs you posted in the 1st post of this topic, it is clear that the issue has nothing to deal with CBT, so don't worry about testing that. Thanks!
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Mon Mar 05, 2012 4:07 pm

Gostev wrote:We tested with the locally attached storage, because this is what the previous poster (who we had a webex with) had the issue with. Obviously we did not want to complicate the testing with more "variables"...

From the graphs you posted in the 1st post of this topic, it is clear that the issue has nothing to deal with CBT, so don't worry about testing that. Thanks!

My storage was "local" to the Veeam server, remote from the source ESXi's.
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Mar 05, 2012 4:14 pm

Ah, got it. Then this is exactly what we did.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby npoIT » Mon Mar 05, 2012 5:15 pm

FWIW, we struggled with this exact issue for well over 6 months. I have many, many throughput plots that look just like yours. We had a very long running ticket open with Veeam, they prompted us to open tickets with our storage vendors and with VMware. We bought hardware HBA's, tested multiple sources and destinations (eventually focussing on local), wiped and reinstalled the Veeam server several times, and bought a new server with a lot of CPU and memory resources - all to no avail. In short, it was basically a nightmare. Just when I was ready to throw in the towel, we wiped the backup server again, installed Windows Server 2008, and everything cleared up.

I am discouraged to hear that you are having all of these problems on Server 2008 though. What service pack are you on? We are on R2 SP1.

Given the amount of time we have already put into this, I can't see us spending much more on it if it crops up here again.
npoIT
Influencer
 
Posts: 14
Liked: never
Joined: Fri Jan 27, 2012 1:29 pm
Full Name: npoIT

Re: Full backup speed decreases as it runs

Veeam Logoby wahrheit2004 » Fri Mar 16, 2012 4:08 pm

Hi everyone. I had a 4hr webex meeting today and the veeam support together with a developer did a lot of tests and they could see the decreasing speed.
I hope it helped and I should get a reply after investigation.

I am running 2008 SP2 on Dual Quad XEON 2,66 with 32GB RAM ECC.
wahrheit2004
Influencer
 
Posts: 10
Liked: 2 times
Joined: Mon Feb 27, 2012 7:33 am
Full Name: DW

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Apr 30, 2012 6:40 pm 1 person likes this post

Update and good news, we have finally tracked down the cause for this behavior. This issue plagued us for a long while, occasionally reported by different customers for a few version in a row. We spent so much effort researching this, and still could never reproduce this in our labs... needless to say, I am very happy that this one is finally resolved.

Long story short, this issue is caused by buggy Windows system cache behavior in Windows 2008 or later. I believe this was even stated once already in one of the topics regarding memory usage. We were able to reproduce this degradation with a simple file copies under certain conditions, doing some totally unrelated experiments and research (funny how things work out sometimes, talk about luck). To resolve this problem once and for all, starting from version 6.1 backup proxies will automatically modify some low level system cache settings upon installation, which resolves this performance degradation issue.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby npoIT » Mon Apr 30, 2012 6:45 pm

I appreciate the update and I feel your pain (so to speak), but we had this problem in Windows 2003 and it actually went away when we switched to 2008. Have you not been able to reproduce it in 2003 or do you only know for a fact that it is an issue in 2008?
npoIT
Influencer
 
Posts: 14
Liked: never
Joined: Fri Jan 27, 2012 1:29 pm
Full Name: npoIT

Re: Full backup speed decreases as it runs

Veeam Logoby tsightler » Mon Apr 30, 2012 6:58 pm

Windows 2003 64-bit had this problem horribly and it was well known and well documented, including various workarounds. Windows 2008 became "smarter" and the problem seemed to creep into "corner cases". Windows 2008R2 seemed to fix the problem for a majority of use cases, but then made it infinitely worse for others, however, all of the cache tools that 3rd parties had come up with for 2008 (and 2003) no longer worked in the same way for 2008R2.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Full backup speed decreases as it runs

Veeam Logoby npoIT » Mon Apr 30, 2012 7:13 pm

Windows 2003 64-bit had this problem horribly and it was well known and well documented

Are you referring to the Windows cache problem and/or the performance drop-off problem in Veeam? I believe we were using 64 bit Windows 2003, but we had a long running case with Veeam and were never told that they were aware of any such issue. It has been a while now so the fixes for the "well documented" problems could have easily been among the many dead ends we went down while attempting to resolve this issue. We tried everything we had found posted and everything Veeam had suggested. I am sure we rebuilt that server from scratch at least 4 or 5 times.

I am not trying to beat a dead horse here, I'd just like to be reassured that this is a bona fide fix for the issue that we struggled with for most of a year. It was a huge problem for us so I am still very reluctant to make any changes to that server. It would be nice to know that we can safely move or rebuild it if necessary.
npoIT
Influencer
 
Posts: 14
Liked: never
Joined: Fri Jan 27, 2012 1:29 pm
Full Name: npoIT

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Apr 30, 2012 7:21 pm

We will know this for sure as soon as the fix is tested on all affected customers, but in our testing it works flawlessly! The fix can be applied manually today, without having to wait for 6.1. I will get the details for you when I am back in the office, and post them here.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby PedroVDP » Tue May 08, 2012 9:42 am

I'm running Veeam on W2k8 R2 as well.

Is there any news about this testing or when we can expect v6.1 ?

If you could share the cache settings I'd like to see if they improve our performance.
PedroVDP
Service Provider
 
Posts: 18
Liked: 2 times
Joined: Wed Jan 18, 2012 9:10 am

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Tue May 08, 2012 10:43 am

We have just started the testing on customers, so I have not heard the results yet. Unfortunately these settings are not something you can set manually via registry, they can only be set through Windows API. The 6.1 version is about 1 month away.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby dweide » Thu Feb 07, 2013 11:22 am

Unfortunately this problem has not gone for good.
I am observing the same kind of performance drop from the original post. Our installation is version 6.5.0.128

I took the effort to try several configurations and the problem seems to be related to the repository server running Win2008 SP2 (HP Proliant ML350, 2x Xeon E5620. 24GB RAM).
The performance drop occurs on this physical machine with different storage setups:
- local RAID 5 with 8 drives
- iSCSI RAID10 array with 8 drives
- share on ths iSCSI array used directly as repository (veem agent on this server not used...)
Resetting CBT did not change things at all.

If I mount the same iSCSI array to a VM running WIN2008 R2 SP1 the performance is stable.
This is true for using this very VM as proxy (hotadd mode) or using a different proxy (san mode).
I also did tests with this VM using a different sample of the iSCSI array mentioned above mounted as VMDK and got stable performance.

Any ideas?
dweide
Enthusiast
 
Posts: 32
Liked: 4 times
Joined: Thu Mar 29, 2012 1:57 pm
Full Name: D. Weide

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 9 guests