Host-based backup of Microsoft Hyper-V VMs.
Post Reply
jamie2497
Novice
Posts: 9
Liked: never
Joined: Feb 11, 2016 9:10 am
Full Name: Jamie
Contact:

Backup Job very slow

Post by jamie2497 »

Hi all,

We have one specific job (It is to a different cluster to all others) that is very slow (2-3MB/s), it looks like it is being throttled. It is a hyper-v cluster so using a onhost proxy, look at the logs for the job shows hundreds of the same error. Bottleneck is showing as target at 99%

Code: Select all

Failed to request latency data
[26.04.2016 22:05:40] <32> Error    [HvDiskLatencyCollector] No answer within 3000 ms was received from host Mail-HV2012R2. (System.Exception)
[26.04.2016 22:05:40] <32> Error    Server stack trace: 
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.ResourceScheduler.SHvDiskLatencyHelper.GetLatency(CHvLatentObjectSpec hvObjectSpec)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[26.04.2016 22:05:40] <32> Error    Exception rethrown at [0]: 
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Interaction.Management.ILatencyCollectionService.GetLatency(ILatentObjectSpec objectSpec)
[26.04.2016 22:05:40] <32> Error       at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[26.04.2016 22:05:40] <32> Error       at Veeam.Common.Remoting.CReconnectableMethod.Invoke[Ret](ActionDelegate`1 dlg)
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Interaction.Management.CLatencyCollectionServiceProxy.Veeam.Backup.Interaction.Management.ILatencyCollectionService.GetLatency(ILatentObjectSpec objectSpec)
There is a network throttle in place but it is set to 60MBPS at all times.

Code: Select all

[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Core.CLatencyThrottler.ThrottlingProc()
[26.04.2016 22:07:01] <44> Info             [AP] (1687) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <20> Info             [AP] (c6ff) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <31> Info             [AP] (bdb4) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <41> Info             [AP] (3216) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
mail-hv2012r2 is our cluster name.

Any ideas would be appreciated.

Support ID - 01780175

Many thanks

Jamie
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Job very slow

Post by foggy »

What kind of storage does this job use as backup repository?
jamie2497
Novice
Posts: 9
Liked: never
Joined: Feb 11, 2016 9:10 am
Full Name: Jamie
Contact:

Re: Backup Job very slow

Post by jamie2497 »

Local RAID 5 storage. The rest of the jobs to other clusters are fine, just this mail cluster that seems to be slow. Weird that it says Target as the source of bottleneck though!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Job very slow

Post by foggy »

Is it a separate RAID or the one where backed up VMs are located?
jamie2497
Novice
Posts: 9
Liked: never
Joined: Feb 11, 2016 9:10 am
Full Name: Jamie
Contact:

Re: Backup Job very slow

Post by jamie2497 »

Separate RAID on a dedicated backup box.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Job very slow

Post by foggy »

Got it. Let's let our engineer examine the logs you've provided.
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests