Discussions specific to the VMware vSphere hypervisor
Post Reply
bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Decreasing Backup Speed During Active Full

Post by bazzajok »

Hi,

After having to re-install our Veeam Server from scratch we are faced with an issue which we never had before!

During active full backups we see decreased performance through the time of the job which has taken our active full times from 16 hours to over 50 hours! Obviously this isn't good as we can't fit this into a weekend anymore.

I've seen other similair posts on here but the issues seemed to surround v6.5 and we are running v7 patch 4.

http://forums.veeam.com/vmware-vsphere- ... 04-30.html

This issue is a real pain and I'm struggling to see what the issue is. The issue seems to surround the Veeam server as it has happened during a backup to tape too.

I've linked a screenshot highlighting the issue. Any pointers would be great.

http://1drv.ms/1FNAhNp

The call has been logged with support Case # 00691513 but nothing sorted as of yet.

Thanks

Lee

nefes
Veeam Software
Posts: 642
Liked: 161 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: Decreasing Backup Speed During Active Full

Post by nefes »

Do you have latency control feature enabled by any chance?
It can be found in I/O Control tab in options.

Vitaliy S.
Product Manager
Posts: 25263
Liked: 2212 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Decreasing Backup Speed During Active Full

Post by Vitaliy S. »

Based on the screenshot I can see that most of the time is consumed by VM preparation task. Did it take the same amount of time before you decided to re-install Veeam backup server?

bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Re: Decreasing Backup Speed During Active Full

Post by bazzajok »

Thanks for the replies.

We don't have the I/O control option in version 7 do we?

Thanks

Lee

foggy
Veeam Software
Posts: 20451
Liked: 1949 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Decreasing Backup Speed During Active Full

Post by foggy »

No, it was first introduced in v8.

bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Re: Decreasing Backup Speed During Active Full

Post by bazzajok »

The long time spent preparing next VM for processing isn't something I've noticed it was more just the overall length of time.

Does this time include the time spent waiting for backup resources?

Lee

Vitaliy S.
Product Manager
Posts: 25263
Liked: 2212 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Decreasing Backup Speed During Active Full

Post by Vitaliy S. »

No, it does not include this time. Let's see what our engineer can say after reviewing your debug log files. BTW, did you notice any performance decrease during incremental job runs?

bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Re: Decreasing Backup Speed During Active Full

Post by bazzajok »

Hi,

They are OK, typically just over 1 hour it's the fulls which are killing me. I know we have some work to do on our infrastructure to increase the speeds but if we could at least keep a consistent speed it would help!

Thanks

Lee

Vitaliy S.
Product Manager
Posts: 25263
Liked: 2212 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Decreasing Backup Speed During Active Full

Post by Vitaliy S. »

Have you tried switching to different transport mode or creating job with less amount of VMs? Do you see the same issue? Just trying to isolate the problem.

bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Re: Decreasing Backup Speed During Active Full

Post by bazzajok »

Hi,

No, I think I'll have to try changing things slowly like you say to isolate the issue.

Thanks

Lee

Vitaliy S.
Product Manager
Posts: 25263
Liked: 2212 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Decreasing Backup Speed During Active Full

Post by Vitaliy S. »

Please keep us posted on your findings.

bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Re: Decreasing Backup Speed During Active Full

Post by bazzajok »

Hi,

After a server reboot the full backup over the weekend processed quicker but still poor with the same symptoms of the speed dropping off over time.

I am now running the full backup to tape which normally takes around 5-6 hours and this has done the same (see link). The Veeam server seems to have a high number of Pages/Sec and this is constant.

http://1drv.ms/1yyaULy

I am starting to wonder if I should rebuild the server with Server 2012 or 2008 R2.

Thanks

Lee

foggy
Veeam Software
Posts: 20451
Liked: 1949 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Decreasing Backup Speed During Active Full

Post by foggy »

The amount of cached memory makes me think this is very similar to that older issue. I will pass your case number to R&D for further investigation.

tsightler
VP, Product Management
Posts: 5905
Liked: 2759 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Decreasing Backup Speed During Active Full

Post by tsightler »

bazzajok wrote:I am starting to wonder if I should rebuild the server with Server 2012 or 2008 R2.
What version of Windows is the system currently and how much memory is in it? I'd certainly recommend 2012 or newer as old Windows versions had some serious cache issues that could cause the behavior you are seeing.

bazzajok
Influencer
Posts: 21
Liked: never
Joined: Nov 04, 2014 3:38 pm
Contact:

Re: Decreasing Backup Speed During Active Full

Post by bazzajok »

Foggy, thanks for that.

We are running Windows Server 2008 SP2 and the server has 72GB RAM (it was originally a VM host).

Thanks

Lee

Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests