Full backup speed decreases as it runs

[Gostev]

Just to provide an update - our QC had completed an incredible testing effort backing up VMs with multiple immense disks in the past few days, and we could not confirm the issue described above. My inbox is full of reports with graphs, and there is clearly no speed reduction at all. We are not giving up though, and are determined to find the explanation.

[jcsd]

Just to provide an update - our QC had completed an incredible testing effort backing up VMs with multiple immense disks in the past few days, and we could not confirm the issue described above. My inbox is full of reports with graphs, and there is clearly no speed reduction at all. We are not giving up though, and are determined to find the explanation.

What were they using for a target? It only happened for me using a Windows target using the VeeamAgent (not CIFS). The problem went away when using a Linux target. By the way, I never did try the CBT fix and to see if that made a difference.

Thanks,
Wesley

[Gostev]

We tested with the locally attached storage, because this is what the previous poster (who we had a webex with) had the issue with. Obviously we did not want to complicate the testing with more "variables"...

From the graphs you posted in the 1st post of this topic, it is clear that the issue has nothing to deal with CBT, so don't worry about testing that. Thanks!

[jcsd]

We tested with the locally attached storage, because this is what the previous poster (who we had a webex with) had the issue with. Obviously we did not want to complicate the testing with more "variables"...

From the graphs you posted in the 1st post of this topic, it is clear that the issue has nothing to deal with CBT, so don't worry about testing that. Thanks!

My storage was "local" to the Veeam server, remote from the source ESXi's.

[Gostev]

Ah, got it. Then this is exactly what we did.

[npoIT]

FWIW, we struggled with this exact issue for well over 6 months. I have many, many throughput plots that look just like yours. We had a very long running ticket open with Veeam, they prompted us to open tickets with our storage vendors and with VMware. We bought hardware HBA's, tested multiple sources and destinations (eventually focussing on local), wiped and reinstalled the Veeam server several times, and bought a new server with a lot of CPU and memory resources - all to no avail. In short, it was basically a nightmare. Just when I was ready to throw in the towel, we wiped the backup server again, installed Windows Server 2008, and everything cleared up.

I am discouraged to hear that you are having all of these problems on Server 2008 though. What service pack are you on? We are on R2 SP1.

Given the amount of time we have already put into this, I can't see us spending much more on it if it crops up here again.

[wahrheit2004]

Hi everyone. I had a 4hr webex meeting today and the veeam support together with a developer did a lot of tests and they could see the decreasing speed.
I hope it helped and I should get a reply after investigation.

I am running 2008 SP2 on Dual Quad XEON 2,66 with 32GB RAM ECC.

[Gostev]

Update and good news, we have finally tracked down the cause for this behavior. This issue plagued us for a long while, occasionally reported by different customers for a few version in a row. We spent so much effort researching this, and still could never reproduce this in our labs... needless to say, I am very happy that this one is finally resolved.

Long story short, this issue is caused by buggy Windows system cache behavior in Windows 2008 or later. I believe this was even stated once already in one of the topics regarding memory usage. We were able to reproduce this degradation with a simple file copies under certain conditions, doing some totally unrelated experiments and research (funny how things work out sometimes, talk about luck). To resolve this problem once and for all, starting from version 6.1 backup proxies will automatically modify some low level system cache settings upon installation, which resolves this performance degradation issue.

[npoIT]

I appreciate the update and I feel your pain (so to speak), but we had this problem in Windows 2003 and it actually went away when we switched to 2008. Have you not been able to reproduce it in 2003 or do you only know for a fact that it is an issue in 2008?

[tsightler]

Windows 2003 64-bit had this problem horribly and it was well known and well documented, including various workarounds. Windows 2008 became "smarter" and the problem seemed to creep into "corner cases". Windows 2008R2 seemed to fix the problem for a majority of use cases, but then made it infinitely worse for others, however, all of the cache tools that 3rd parties had come up with for 2008 (and 2003) no longer worked in the same way for 2008R2.

[npoIT]

Windows 2003 64-bit had this problem horribly and it was well known and well documented

Are you referring to the Windows cache problem and/or the performance drop-off problem in Veeam? I believe we were using 64 bit Windows 2003, but we had a long running case with Veeam and were never told that they were aware of any such issue. It has been a while now so the fixes for the "well documented" problems could have easily been among the many dead ends we went down while attempting to resolve this issue. We tried everything we had found posted and everything Veeam had suggested. I am sure we rebuilt that server from scratch at least 4 or 5 times.

I am not trying to beat a dead horse here, I'd just like to be reassured that this is a bona fide fix for the issue that we struggled with for most of a year. It was a huge problem for us so I am still very reluctant to make any changes to that server. It would be nice to know that we can safely move or rebuild it if necessary.

[Gostev]

We will know this for sure as soon as the fix is tested on all affected customers, but in our testing it works flawlessly! The fix can be applied manually today, without having to wait for 6.1. I will get the details for you when I am back in the office, and post them here.

[PedroVDP]

I'm running Veeam on W2k8 R2 as well.

Is there any news about this testing or when we can expect v6.1 ?

If you could share the cache settings I'd like to see if they improve our performance.

[Gostev]

We have just started the testing on customers, so I have not heard the results yet. Unfortunately these settings are not something you can set manually via registry, they can only be set through Windows API. The 6.1 version is about 1 month away.

[dweide]

Unfortunately this problem has not gone for good.
I am observing the same kind of performance drop from the original post. Our installation is version 6.5.0.128

I took the effort to try several configurations and the problem seems to be related to the repository server running Win2008 SP2 (HP Proliant ML350, 2x Xeon E5620. 24GB RAM).
The performance drop occurs on this physical machine with different storage setups:
- local RAID 5 with 8 drives
- iSCSI RAID10 array with 8 drives
- share on ths iSCSI array used directly as repository (veem agent on this server not used...)
Resetting CBT did not change things at all.

If I mount the same iSCSI array to a VM running WIN2008 R2 SP1 the performance is stable.
This is true for using this very VM as proxy (hotadd mode) or using a different proxy (san mode).
I also did tests with this VM using a different sample of the iSCSI array mentioned above mounted as VMDK and got stable performance.

Any ideas?

[Gostev]

Upgrade the repository to WIN2008 R2 SP1?

[Ratcha]

Upgrade the repository to WIN2008 R2 SP1?

Is that the only solution? We're also running Windows Server 2008 (not R2).

I've had this problem for years now as well, multiple support cases opened and threads here on the topic (other threads I started in the past have disappeared?), and was never provided with an explanation or resolution. I saw the earlier post in this thread stating the issue was fixed in version 6, but I'm on version 7 for a while now and the issue persists. During the latest support case I opened a couple of months ago, the support engineer insisted it was the SAN as the VM with the issue was on an array consisting of NLSAS disks instead of true SAS. I ran several I/O benchmarking utilities as instructed and all showed expected performance, so I gave up on the case as I ran out of time to ride Veeam on this. As the issue has become rather severe, I caved and purchased an expansion chassis for the SAN with 24 x 600GB 15k SAS disks configured in a single RAID 10 array, all to store this one file server with a 1.6TB disk. Ran the VMware I/O analyzer for a couple of days with excellent results for IOPS and throughput, both with a single worker and multiple workers. With everything looking good, I copied the VM to the new array last night and started running the backup this afternoon. It started at 90+MB/s, no different than when on the 6 disk RAID6 array, and is now down to 38MB/s 36% into the job. I'm expecting the job to be at 6MB/s or less in 2 or 3 days when it finally completes.

It will be nice to find out that a fresh install of 2008 R2 will resolve this. It would have been even nicer to know before spending $21,000 on a SAN upgrade.

Anton, do I need to open a new support case or is 2008 R2 the solution to this?

[foggy]

Actually a better recommendation is to upgrade to at least Windows 2012, as older versions were affected by system cache issues causing the observed behavior. There's a more recent discussion regarding similar problems with pretty much the same setup (Windows Server 2008 and Veeam B&R v7). And yes, opening the case is still recommended.

[Ratcha]

I don't believe we have 2012 licenses. Will 2008 R2 fix this?

[Gostev]

Well, last post on the previous page suggests that WIN2008 R2 SP1 does not exhibit said issue, so it's worth a try. And remember that you can always switch to Linux-based repository as another option, as long as you possess basic Linux skills.

[accesspc]

Hi to everyone,

I've been having this problem for a while and decided to search for sollution.

My Environment:
Veeam Backup & Repliction 6.5.0.144 (64bit, Standard)
VMware vSphere 5.1 (Standard)
VM to backup: Ubuntu Samba3 file server, containing 3 virtual disks: 8 GB, 1,9 TB and 1,7 TB.
Provisioned storage: 3,61 TB
Used (not-shared) storage: 3,29 TB

So the problem is, that full backup of this VM takes about 4 days and backup speed is degrading over all backup process. Veeam statistics for this backup job is showing, that:
1st disk average read speed is at ~66 MB/s [CBT], duration: 1-2 minutes
2nd disk average read speed is at ~27 MB/s [CBT], duration: 20+ hours
3rd disk average read speed is at ~10 MB/s [CBT], duration: 50-60+ hours

Sometimes backup fails on the 3rd disk with an error like this:
Timed out waiting for all VDDK disks to close
Error: Client error: VDDK error: 1.Unknown error Unable to retrieve next block transmission command. Number of already processed blocks: [1535853].

Busy: Source 39% > Proxy 28% > Network 32% > Target 41%

Is there any way to figure this out, why backup process takes so much time, why backup speed is degrading and why sometimes it fails?

If you need any additional info, please comment.

Waiting for some help, please

[Shestakov]

Hello and welcome to the forum!
The issue similar to yours has been discussed under this topic. Please get familiar.

What kind of OS is your repository running?
By the way, VBR v6.5 is not supported anymore. Upgrade to version 8 is free of charge for those on an active maintenance. Please upgrade.

Thanks.

[accesspc]

Forgot to mention, that repository is running on Windows Server 2008 SP2 (v6.0.6002).
Unfortunatelly our active mainenance licence has expired some time ago, and we cannot upgrade, but we are considering buying a new version in near future, but now it would be nice to find a sollution for this problem. I am willing to try the fix about CBT mentioned in one post, but some how I doubt it will give anything good. Any suggestions?

P.S. Thanks for putting my post in the right place.

[Shestakov]

As was written previously, this issue is caused by buggy Windows system cache behavior in some old OS versions. If upgrading to v8 of VBR is not an option for now, try upgrading Windows Server to Windows Server 2008 R2 (v6.1.x) or later.
Thank you!

[accesspc]

Apparently, OS upgrade to Windows Server 2012 R2 has helped. Although full backup has nors finished yet, there is significant increase in backup time of one of the partitions - instead of 20 hrs for 1,9 TB partition, now it has been backed up in 13 hrs. I AM HAPPY! and thank you for help.
P.S. could this "buggy Windows system cache behavior" impact on synthetic full backup time?

[Shestakov]

I`m glad it helped!
As far as I know, there were some issues in older versions of VBR.
The best you can do now is to upgrade to VBR v8 where the issue is solved and other valuable features are available.
Thank you.

[Dave338]

Also those with windows 2008, can try installing Windows Dynamic Cache Service.

Is a feature that, in theory, helps with the caché problems, giving the R2 performance level (or at least it seems to help). We installed some time ago in all of our SAP servers and at least it has not been bad

Regards.
Dave.

[accesspc]

Well, OS upgrade to WIndows Server 2012 R2 helped me a lot, backup time for this 3,3 TB VM is now at least twice faster then previously. Thank you for support and in near future we'll be upgrading Veeam to newest version

[modman]

Also those with windows 2008, can try installing Windows Dynamic Cache Service.

Is a feature that, in theory, helps with the caché problems, giving the R2 performance level (or at least it seems to help). We installed some time ago in all of our SAP servers and at least it has not been bad

Regards.
Dave.

Dave, when you say at least it has not been bad, is it when performing Veeam backups or did it help with SAP performance as well? I have interest in both of them.

[modman]

Like everyone here, I was seeing backup speed slow down gradually on files with larger virtual disks (500GB or above). My backup server/proxy/repository is on a Windows 2008 Storage Server with Veeam 8 Patch 1.

Got to a point where I am unable to finish a full backup of my file server which has a total of 7TB. I was able to backup other VMs fine. Even a file server with 2TB virtual disk backed up because I was able to let it crawl for 3+ days and finish the full pass.

Using the virtual appliance, it would start off at 50MB/s and then a couple of hours into the job, it would slow down to between 5MB and 10MB. Using the direct SAN method, it would start off at 95MB/s and slow down. The starting numbers gradually decreased as well. For example, on the 3rd attempt, the starting processed speed using VA might be 35MB/s and starting speed of direct SAN might be 75MB/s. If I reboot the backup server which is also a proxy and repository, the starting speed would go up again for some time. However, they can only be sustained for about two hours.

Veeam support blames my two EqualLogic PS6100E SANs with 7200 RPM drives in RAID10 although SANHQ shows that there was no load on the SAN (Case # 00731495). I told them I get the fact that I can't sustain 50MB/s. However, 5MB/s is just too slow even for a single 7200 RPM drive. When using BackupExec and writing to tape, I get about 30MB/s sustained speed for the same file server and the bottle neck is the LTO3 tape drive. So at the very least, 30MB/s is what I would expect.

I finally decided to install Veeam on another server running Windows 2008 R2 and was able to sustain the processed speed. This proves that Windows 2008 was the culprit.

I installed the Dynamic Cache Service on the original Windows 2008 backup server and tested again, but that did not help. When I setup a Repository on a Windows 2008 R2 server and used the original Windows 2008 backup server to backup the job and act as a proxy, I was able to get 98MB/s and sustain it throughout the job, using the direct SAN method.

To summarize, the latest version of Veeam (version 8 patch 1) does not fix the issue. The fix is to make sure your repository is not running Windows 2008 or earlier.