Full backup speed decreases as it runs

VMware specific discussions

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Sun Feb 10, 2013 3:24 pm

Upgrade the repository to WIN2008 R2 SP1? :D
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby Ratcha » Mon Dec 29, 2014 2:00 am

Gostev wrote:Upgrade the repository to WIN2008 R2 SP1? :D


Is that the only solution? We're also running Windows Server 2008 (not R2).

I've had this problem for years now as well, multiple support cases opened and threads here on the topic (other threads I started in the past have disappeared?), and was never provided with an explanation or resolution. I saw the earlier post in this thread stating the issue was fixed in version 6, but I'm on version 7 for a while now and the issue persists. During the latest support case I opened a couple of months ago, the support engineer insisted it was the SAN as the VM with the issue was on an array consisting of NLSAS disks instead of true SAS. I ran several I/O benchmarking utilities as instructed and all showed expected performance, so I gave up on the case as I ran out of time to ride Veeam on this. As the issue has become rather severe, I caved and purchased an expansion chassis for the SAN with 24 x 600GB 15k SAS disks configured in a single RAID 10 array, all to store this one file server with a 1.6TB disk. Ran the VMware I/O analyzer for a couple of days with excellent results for IOPS and throughput, both with a single worker and multiple workers. With everything looking good, I copied the VM to the new array last night and started running the backup this afternoon. It started at 90+MB/s, no different than when on the 6 disk RAID6 array, and is now down to 38MB/s 36% into the job. I'm expecting the job to be at 6MB/s or less in 2 or 3 days when it finally completes.

It will be nice to find out that a fresh install of 2008 R2 will resolve this. It would have been even nicer to know before spending $21,000 on a SAN upgrade.

Anton, do I need to open a new support case or is 2008 R2 the solution to this?


Image

Image
Ratcha
Influencer
 
Posts: 23
Liked: 7 times
Joined: Sun Jun 13, 2010 10:36 pm

Re: Full backup speed decreases as it runs

Veeam Logoby foggy » Mon Dec 29, 2014 11:44 am

Actually a better recommendation is to upgrade to at least Windows 2012, as older versions were affected by system cache issues causing the observed behavior. There's a more recent discussion regarding similar problems with pretty much the same setup (Windows Server 2008 and Veeam B&R v7). And yes, opening the case is still recommended.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Full backup speed decreases as it runs

Veeam Logoby Ratcha » Mon Dec 29, 2014 6:27 pm

I don't believe we have 2012 licenses. Will 2008 R2 fix this?
Ratcha
Influencer
 
Posts: 23
Liked: 7 times
Joined: Sun Jun 13, 2010 10:36 pm

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Mon Dec 29, 2014 11:53 pm 1 person likes this post

Well, last post on the previous page suggests that WIN2008 R2 SP1 does not exhibit said issue, so it's worth a try. And remember that you can always switch to Linux-based repository as another option, as long as you possess basic Linux skills.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

[MERGED] : v6.5 VM backup slowing down

Veeam Logoby accesspc » Mon Jan 26, 2015 10:53 am

Hi to everyone,

I've been having this problem for a while and decided to search for sollution.

My Environment:
Veeam Backup & Repliction 6.5.0.144 (64bit, Standard)
VMware vSphere 5.1 (Standard)
VM to backup: Ubuntu Samba3 file server, containing 3 virtual disks: 8 GB, 1,9 TB and 1,7 TB.
Provisioned storage: 3,61 TB
Used (not-shared) storage: 3,29 TB

So the problem is, that full backup of this VM takes about 4 days and backup speed is degrading over all backup process. Veeam statistics for this backup job is showing, that:
1st disk average read speed is at ~66 MB/s [CBT], duration: 1-2 minutes
2nd disk average read speed is at ~27 MB/s [CBT], duration: 20+ hours
3rd disk average read speed is at ~10 MB/s [CBT], duration: 50-60+ hours

Sometimes backup fails on the 3rd disk with an error like this:
Timed out waiting for all VDDK disks to close
Error: Client error: VDDK error: 1.Unknown error Unable to retrieve next block transmission command. Number of already processed blocks: [1535853].

Busy: Source 39% > Proxy 28% > Network 32% > Target 41%

Is there any way to figure this out, why backup process takes so much time, why backup speed is degrading and why sometimes it fails?

If you need any additional info, please comment.

Waiting for some help, please :)
accesspc
Novice
 
Posts: 4
Liked: never
Joined: Mon Jan 26, 2015 10:28 am
Full Name: Robertas

Re: Full backup speed decreases as it runs

Veeam Logoby Shestakov » Mon Jan 26, 2015 11:25 am

Hello and welcome to the forum!
The issue similar to yours has been discussed under this topic. Please get familiar.

What kind of OS is your repository running?
By the way, VBR v6.5 is not supported anymore. Upgrade to version 8 is free of charge for those on an active maintenance. Please upgrade.

Thanks.
Shestakov
Veeam Software
 
Posts: 4842
Liked: 393 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Full backup speed decreases as it runs

Veeam Logoby accesspc » Mon Jan 26, 2015 12:31 pm

Forgot to mention, that repository is running on Windows Server 2008 SP2 (v6.0.6002).
Unfortunatelly our active mainenance licence has expired some time ago, and we cannot upgrade, but we are considering buying a new version in near future, but now it would be nice to find a sollution for this problem. I am willing to try the fix about CBT mentioned in one post, but some how I doubt it will give anything good. Any suggestions?

P.S. Thanks for putting my post in the right place.
accesspc
Novice
 
Posts: 4
Liked: never
Joined: Mon Jan 26, 2015 10:28 am
Full Name: Robertas

Re: Full backup speed decreases as it runs

Veeam Logoby Shestakov » Mon Jan 26, 2015 12:58 pm

As was written previously, this issue is caused by buggy Windows system cache behavior in some old OS versions. If upgrading to v8 of VBR is not an option for now, try upgrading Windows Server to Windows Server 2008 R2 (v6.1.x) or later.
Thank you!
Shestakov
Veeam Software
 
Posts: 4842
Liked: 393 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Full backup speed decreases as it runs

Veeam Logoby accesspc » Thu Jan 29, 2015 5:51 am

Apparently, OS upgrade to Windows Server 2012 R2 has helped. Although full backup has nors finished yet, there is significant increase in backup time of one of the partitions - instead of 20 hrs for 1,9 TB partition, now it has been backed up in 13 hrs. I AM HAPPY! and thank you for help.
P.S. could this "buggy Windows system cache behavior" impact on synthetic full backup time?
accesspc
Novice
 
Posts: 4
Liked: never
Joined: Mon Jan 26, 2015 10:28 am
Full Name: Robertas

Re: Full backup speed decreases as it runs

Veeam Logoby Shestakov » Thu Jan 29, 2015 11:02 am

I`m glad it helped!
As far as I know, there were some issues in older versions of VBR.
The best you can do now is to upgrade to VBR v8 where the issue is solved and other valuable features are available.
Thank you.
Shestakov
Veeam Software
 
Posts: 4842
Liked: 393 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Full backup speed decreases as it runs

Veeam Logoby Dave338 » Thu Jan 29, 2015 1:33 pm 1 person likes this post

Also those with windows 2008, can try installing Windows Dynamic Cache Service.

Is a feature that, in theory, helps with the caché problems, giving the R2 performance level (or at least it seems to help). We installed some time ago in all of our SAP servers and at least it has not been bad :)

Regards.
Dave.
Dave338
Enthusiast
 
Posts: 35
Liked: 5 times
Joined: Tue Jan 27, 2015 12:21 pm
Full Name: David

Re: Full backup speed decreases as it runs

Veeam Logoby accesspc » Fri Jan 30, 2015 12:30 pm

Well, OS upgrade to WIndows Server 2012 R2 helped me a lot, backup time for this 3,3 TB VM is now at least twice faster then previously. Thank you for support and in near future we'll be upgrading Veeam to newest version ;)
accesspc
Novice
 
Posts: 4
Liked: never
Joined: Mon Jan 26, 2015 10:28 am
Full Name: Robertas

Re: Full backup speed decreases as it runs

Veeam Logoby modman » Wed Feb 04, 2015 4:38 pm

Dave338 wrote:Also those with windows 2008, can try installing Windows Dynamic Cache Service.

Is a feature that, in theory, helps with the caché problems, giving the R2 performance level (or at least it seems to help). We installed some time ago in all of our SAP servers and at least it has not been bad :)

Regards.
Dave.


Dave, when you say at least it has not been bad, is it when performing Veeam backups or did it help with SAP performance as well? I have interest in both of them.
modman
Novice
 
Posts: 3
Liked: 3 times
Joined: Mon Jan 26, 2015 4:46 pm

Re: Full backup speed decreases as it runs

Veeam Logoby modman » Wed Feb 04, 2015 5:01 pm 3 people like this post

Like everyone here, I was seeing backup speed slow down gradually on files with larger virtual disks (500GB or above). My backup server/proxy/repository is on a Windows 2008 Storage Server with Veeam 8 Patch 1.

Got to a point where I am unable to finish a full backup of my file server which has a total of 7TB. I was able to backup other VMs fine. Even a file server with 2TB virtual disk backed up because I was able to let it crawl for 3+ days and finish the full pass.

Using the virtual appliance, it would start off at 50MB/s and then a couple of hours into the job, it would slow down to between 5MB and 10MB. Using the direct SAN method, it would start off at 95MB/s and slow down. The starting numbers gradually decreased as well. For example, on the 3rd attempt, the starting processed speed using VA might be 35MB/s and starting speed of direct SAN might be 75MB/s. If I reboot the backup server which is also a proxy and repository, the starting speed would go up again for some time. However, they can only be sustained for about two hours.

Veeam support blames my two EqualLogic PS6100E SANs with 7200 RPM drives in RAID10 although SANHQ shows that there was no load on the SAN (Case # 00731495). I told them I get the fact that I can't sustain 50MB/s. However, 5MB/s is just too slow even for a single 7200 RPM drive. When using BackupExec and writing to tape, I get about 30MB/s sustained speed for the same file server and the bottle neck is the LTO3 tape drive. So at the very least, 30MB/s is what I would expect.

I finally decided to install Veeam on another server running Windows 2008 R2 and was able to sustain the processed speed. This proves that Windows 2008 was the culprit.

I installed the Dynamic Cache Service on the original Windows 2008 backup server and tested again, but that did not help. When I setup a Repository on a Windows 2008 R2 server and used the original Windows 2008 backup server to backup the job and act as a proxy, I was able to get 98MB/s and sustain it throughout the job, using the direct SAN method.

To summarize, the latest version of Veeam (version 8 patch 1) does not fix the issue. The fix is to make sure your repository is not running Windows 2008 or earlier.
modman
Novice
 
Posts: 3
Liked: 3 times
Joined: Mon Jan 26, 2015 4:46 pm

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 18 guests