Backup Job very slow

Hyper-V specific discussions

Backup Job very slow

Veeam Logoby jamie2497 » Wed Apr 27, 2016 9:40 am

Hi all,

We have one specific job (It is to a different cluster to all others) that is very slow (2-3MB/s), it looks like it is being throttled. It is a hyper-v cluster so using a onhost proxy, look at the logs for the job shows hundreds of the same error. Bottleneck is showing as target at 99%

Code: Select all
Failed to request latency data
[26.04.2016 22:05:40] <32> Error    [HvDiskLatencyCollector] No answer within 3000 ms was received from host Mail-HV2012R2. (System.Exception)
[26.04.2016 22:05:40] <32> Error    Server stack trace:
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.ResourceScheduler.SHvDiskLatencyHelper.GetLatency(CHvLatentObjectSpec hvObjectSpec)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[26.04.2016 22:05:40] <32> Error    Exception rethrown at [0]:
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Interaction.Management.ILatencyCollectionService.GetLatency(ILatentObjectSpec objectSpec)
[26.04.2016 22:05:40] <32> Error       at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[26.04.2016 22:05:40] <32> Error       at Veeam.Common.Remoting.CReconnectableMethod.Invoke[Ret](ActionDelegate`1 dlg)
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Interaction.Management.CLatencyCollectionServiceProxy.Veeam.Backup.Interaction.Management.ILatencyCollectionService.GetLatency(ILatentObjectSpec objectSpec)


There is a network throttle in place but it is set to 60MBPS at all times.

Code: Select all
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Core.CLatencyThrottler.ThrottlingProc()
[26.04.2016 22:07:01] <44> Info             [AP] (1687) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <20> Info             [AP] (c6ff) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <31> Info             [AP] (bdb4) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <41> Info             [AP] (3216) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'


mail-hv2012r2 is our cluster name.

Any ideas would be appreciated.

Support ID - 01780175

Many thanks

Jamie
jamie2497
Novice
 
Posts: 8
Liked: never
Joined: Thu Feb 11, 2016 9:10 am
Full Name: Jamie

Re: Backup Job very slow

Veeam Logoby foggy » Wed Apr 27, 2016 11:28 am

What kind of storage does this job use as backup repository?
foggy
Veeam Software
 
Posts: 14758
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup Job very slow

Veeam Logoby jamie2497 » Wed Apr 27, 2016 12:05 pm

Local RAID 5 storage. The rest of the jobs to other clusters are fine, just this mail cluster that seems to be slow. Weird that it says Target as the source of bottleneck though!
jamie2497
Novice
 
Posts: 8
Liked: never
Joined: Thu Feb 11, 2016 9:10 am
Full Name: Jamie

Re: Backup Job very slow

Veeam Logoby foggy » Wed Apr 27, 2016 2:43 pm

Is it a separate RAID or the one where backed up VMs are located?
foggy
Veeam Software
 
Posts: 14758
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup Job very slow

Veeam Logoby jamie2497 » Thu Apr 28, 2016 6:08 am

Separate RAID on a dedicated backup box.
jamie2497
Novice
 
Posts: 8
Liked: never
Joined: Thu Feb 11, 2016 9:10 am
Full Name: Jamie

Re: Backup Job very slow

Veeam Logoby foggy » Thu Apr 28, 2016 11:18 am

Got it. Let's let our engineer examine the logs you've provided.
foggy
Veeam Software
 
Posts: 14758
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Bing [Bot] and 10 guests