Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Feb 10, 2013 3:24 pm

Upgrade the repository to WIN2008 R2 SP1? :D

Ratcha
Influencer
Posts: 23
Liked: 7 times
Joined: Jun 13, 2010 10:36 pm
Contact:

Re: Full backup speed decreases as it runs

Post by Ratcha » Dec 29, 2014 2:00 am

Gostev wrote:Upgrade the repository to WIN2008 R2 SP1? :D
Is that the only solution? We're also running Windows Server 2008 (not R2).

I've had this problem for years now as well, multiple support cases opened and threads here on the topic (other threads I started in the past have disappeared?), and was never provided with an explanation or resolution. I saw the earlier post in this thread stating the issue was fixed in version 6, but I'm on version 7 for a while now and the issue persists. During the latest support case I opened a couple of months ago, the support engineer insisted it was the SAN as the VM with the issue was on an array consisting of NLSAS disks instead of true SAS. I ran several I/O benchmarking utilities as instructed and all showed expected performance, so I gave up on the case as I ran out of time to ride Veeam on this. As the issue has become rather severe, I caved and purchased an expansion chassis for the SAN with 24 x 600GB 15k SAS disks configured in a single RAID 10 array, all to store this one file server with a 1.6TB disk. Ran the VMware I/O analyzer for a couple of days with excellent results for IOPS and throughput, both with a single worker and multiple workers. With everything looking good, I copied the VM to the new array last night and started running the backup this afternoon. It started at 90+MB/s, no different than when on the 6 disk RAID6 array, and is now down to 38MB/s 36% into the job. I'm expecting the job to be at 6MB/s or less in 2 or 3 days when it finally completes.

It will be nice to find out that a fresh install of 2008 R2 will resolve this. It would have been even nicer to know before spending $21,000 on a SAN upgrade.

Anton, do I need to open a new support case or is 2008 R2 the solution to this?


Image

Image

foggy
Veeam Software
Posts: 17712
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Full backup speed decreases as it runs

Post by foggy » Dec 29, 2014 11:44 am

Actually a better recommendation is to upgrade to at least Windows 2012, as older versions were affected by system cache issues causing the observed behavior. There's a more recent discussion regarding similar problems with pretty much the same setup (Windows Server 2008 and Veeam B&R v7). And yes, opening the case is still recommended.

Ratcha
Influencer
Posts: 23
Liked: 7 times
Joined: Jun 13, 2010 10:36 pm
Contact:

Re: Full backup speed decreases as it runs

Post by Ratcha » Dec 29, 2014 6:27 pm

I don't believe we have 2012 licenses. Will 2008 R2 fix this?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Dec 29, 2014 11:53 pm 1 person likes this post

Well, last post on the previous page suggests that WIN2008 R2 SP1 does not exhibit said issue, so it's worth a try. And remember that you can always switch to Linux-based repository as another option, as long as you possess basic Linux skills.

accesspc
Novice
Posts: 4
Liked: never
Joined: Jan 26, 2015 10:28 am
Full Name: Robertas
Contact:

[MERGED] : v6.5 VM backup slowing down

Post by accesspc » Jan 26, 2015 10:53 am

Hi to everyone,

I've been having this problem for a while and decided to search for sollution.

My Environment:
Veeam Backup & Repliction 6.5.0.144 (64bit, Standard)
VMware vSphere 5.1 (Standard)
VM to backup: Ubuntu Samba3 file server, containing 3 virtual disks: 8 GB, 1,9 TB and 1,7 TB.
Provisioned storage: 3,61 TB
Used (not-shared) storage: 3,29 TB

So the problem is, that full backup of this VM takes about 4 days and backup speed is degrading over all backup process. Veeam statistics for this backup job is showing, that:
1st disk average read speed is at ~66 MB/s [CBT], duration: 1-2 minutes
2nd disk average read speed is at ~27 MB/s [CBT], duration: 20+ hours
3rd disk average read speed is at ~10 MB/s [CBT], duration: 50-60+ hours

Sometimes backup fails on the 3rd disk with an error like this:
Timed out waiting for all VDDK disks to close
Error: Client error: VDDK error: 1.Unknown error Unable to retrieve next block transmission command. Number of already processed blocks: [1535853].

Busy: Source 39% > Proxy 28% > Network 32% > Target 41%

Is there any way to figure this out, why backup process takes so much time, why backup speed is degrading and why sometimes it fails?

If you need any additional info, please comment.

Waiting for some help, please :)

Shestakov
Veeam Software
Posts: 6499
Liked: 628 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Full backup speed decreases as it runs

Post by Shestakov » Jan 26, 2015 11:25 am

Hello and welcome to the forum!
The issue similar to yours has been discussed under this topic. Please get familiar.

What kind of OS is your repository running?
By the way, VBR v6.5 is not supported anymore. Upgrade to version 8 is free of charge for those on an active maintenance. Please upgrade.

Thanks.

accesspc
Novice
Posts: 4
Liked: never
Joined: Jan 26, 2015 10:28 am
Full Name: Robertas
Contact:

Re: Full backup speed decreases as it runs

Post by accesspc » Jan 26, 2015 12:31 pm

Forgot to mention, that repository is running on Windows Server 2008 SP2 (v6.0.6002).
Unfortunatelly our active mainenance licence has expired some time ago, and we cannot upgrade, but we are considering buying a new version in near future, but now it would be nice to find a sollution for this problem. I am willing to try the fix about CBT mentioned in one post, but some how I doubt it will give anything good. Any suggestions?

P.S. Thanks for putting my post in the right place.

Shestakov
Veeam Software
Posts: 6499
Liked: 628 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Full backup speed decreases as it runs

Post by Shestakov » Jan 26, 2015 12:58 pm

As was written previously, this issue is caused by buggy Windows system cache behavior in some old OS versions. If upgrading to v8 of VBR is not an option for now, try upgrading Windows Server to Windows Server 2008 R2 (v6.1.x) or later.
Thank you!

accesspc
Novice
Posts: 4
Liked: never
Joined: Jan 26, 2015 10:28 am
Full Name: Robertas
Contact:

Re: Full backup speed decreases as it runs

Post by accesspc » Jan 29, 2015 5:51 am

Apparently, OS upgrade to Windows Server 2012 R2 has helped. Although full backup has nors finished yet, there is significant increase in backup time of one of the partitions - instead of 20 hrs for 1,9 TB partition, now it has been backed up in 13 hrs. I AM HAPPY! and thank you for help.
P.S. could this "buggy Windows system cache behavior" impact on synthetic full backup time?

Shestakov
Veeam Software
Posts: 6499
Liked: 628 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Full backup speed decreases as it runs

Post by Shestakov » Jan 29, 2015 11:02 am

I`m glad it helped!
As far as I know, there were some issues in older versions of VBR.
The best you can do now is to upgrade to VBR v8 where the issue is solved and other valuable features are available.
Thank you.

Dave338
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jan 27, 2015 12:21 pm
Full Name: David
Contact:

Re: Full backup speed decreases as it runs

Post by Dave338 » Jan 29, 2015 1:33 pm 1 person likes this post

Also those with windows 2008, can try installing Windows Dynamic Cache Service.

Is a feature that, in theory, helps with the caché problems, giving the R2 performance level (or at least it seems to help). We installed some time ago in all of our SAP servers and at least it has not been bad :)

Regards.
Dave.

accesspc
Novice
Posts: 4
Liked: never
Joined: Jan 26, 2015 10:28 am
Full Name: Robertas
Contact:

Re: Full backup speed decreases as it runs

Post by accesspc » Jan 30, 2015 12:30 pm

Well, OS upgrade to WIndows Server 2012 R2 helped me a lot, backup time for this 3,3 TB VM is now at least twice faster then previously. Thank you for support and in near future we'll be upgrading Veeam to newest version ;)

modman
Novice
Posts: 3
Liked: 3 times
Joined: Jan 26, 2015 4:46 pm
Contact:

Re: Full backup speed decreases as it runs

Post by modman » Feb 04, 2015 4:38 pm

Dave338 wrote:Also those with windows 2008, can try installing Windows Dynamic Cache Service.

Is a feature that, in theory, helps with the caché problems, giving the R2 performance level (or at least it seems to help). We installed some time ago in all of our SAP servers and at least it has not been bad :)

Regards.
Dave.
Dave, when you say at least it has not been bad, is it when performing Veeam backups or did it help with SAP performance as well? I have interest in both of them.

modman
Novice
Posts: 3
Liked: 3 times
Joined: Jan 26, 2015 4:46 pm
Contact:

Re: Full backup speed decreases as it runs

Post by modman » Feb 04, 2015 5:01 pm 3 people like this post

Like everyone here, I was seeing backup speed slow down gradually on files with larger virtual disks (500GB or above). My backup server/proxy/repository is on a Windows 2008 Storage Server with Veeam 8 Patch 1.

Got to a point where I am unable to finish a full backup of my file server which has a total of 7TB. I was able to backup other VMs fine. Even a file server with 2TB virtual disk backed up because I was able to let it crawl for 3+ days and finish the full pass.

Using the virtual appliance, it would start off at 50MB/s and then a couple of hours into the job, it would slow down to between 5MB and 10MB. Using the direct SAN method, it would start off at 95MB/s and slow down. The starting numbers gradually decreased as well. For example, on the 3rd attempt, the starting processed speed using VA might be 35MB/s and starting speed of direct SAN might be 75MB/s. If I reboot the backup server which is also a proxy and repository, the starting speed would go up again for some time. However, they can only be sustained for about two hours.

Veeam support blames my two EqualLogic PS6100E SANs with 7200 RPM drives in RAID10 although SANHQ shows that there was no load on the SAN (Case # 00731495). I told them I get the fact that I can't sustain 50MB/s. However, 5MB/s is just too slow even for a single 7200 RPM drive. When using BackupExec and writing to tape, I get about 30MB/s sustained speed for the same file server and the bottle neck is the LTO3 tape drive. So at the very least, 30MB/s is what I would expect.

I finally decided to install Veeam on another server running Windows 2008 R2 and was able to sustain the processed speed. This proves that Windows 2008 was the culprit.

I installed the Dynamic Cache Service on the original Windows 2008 backup server and tested again, but that did not help. When I setup a Repository on a Windows 2008 R2 server and used the original Windows 2008 backup server to backup the job and act as a proxy, I was able to get 98MB/s and sustain it throughout the job, using the direct SAN method.

To summarize, the latest version of Veeam (version 8 patch 1) does not fix the issue. The fix is to make sure your repository is not running Windows 2008 or earlier.

Post Reply

Who is online

Users browsing this forum: Majestic-12 [Bot] and 7 guests