Backup Job very slow

[jamie2497]

Hi all,

We have one specific job (It is to a different cluster to all others) that is very slow (2-3MB/s), it looks like it is being throttled. It is a hyper-v cluster so using a onhost proxy, look at the logs for the job shows hundreds of the same error. Bottleneck is showing as target at 99%

Code: Select all

Failed to request latency data
[26.04.2016 22:05:40] <32> Error    [HvDiskLatencyCollector] No answer within 3000 ms was received from host Mail-HV2012R2. (System.Exception)
[26.04.2016 22:05:40] <32> Error    Server stack trace: 
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.ResourceScheduler.SHvDiskLatencyHelper.GetLatency(CHvLatentObjectSpec hvObjectSpec)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[26.04.2016 22:05:40] <32> Error    Exception rethrown at [0]: 
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[26.04.2016 22:05:40] <32> Error       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Interaction.Management.ILatencyCollectionService.GetLatency(ILatentObjectSpec objectSpec)
[26.04.2016 22:05:40] <32> Error       at Veeam.Common.Remoting.CRetryableMethod.DoInvoke[Ret](ActionDelegate`1 dlg)
[26.04.2016 22:05:40] <32> Error       at Veeam.Common.Remoting.CReconnectableMethod.Invoke[Ret](ActionDelegate`1 dlg)
[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Interaction.Management.CLatencyCollectionServiceProxy.Veeam.Backup.Interaction.Management.ILatencyCollectionService.GetLatency(ILatentObjectSpec objectSpec)

There is a network throttle in place but it is set to 60MBPS at all times.

Code: Select all

[26.04.2016 22:05:40] <32> Error       at Veeam.Backup.Core.CLatencyThrottler.ThrottlingProc()
[26.04.2016 22:07:01] <44> Info             [AP] (1687) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <20> Info             [AP] (c6ff) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <31> Info             [AP] (bdb4) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'
[26.04.2016 22:07:01] <41> Info             [AP] (3216) command: 'Invoke: Network.UpdateThrottling\n{\n  (EInt32) KbpsWriteBandwidth = 10240\n}\n'

mail-hv2012r2 is our cluster name.

Any ideas would be appreciated.

Support ID - 01780175

Many thanks

Jamie

[foggy]

What kind of storage does this job use as backup repository?

[jamie2497]

Local RAID 5 storage. The rest of the jobs to other clusters are fine, just this mail cluster that seems to be slow. Weird that it says Target as the source of bottleneck though!

[foggy]

Is it a separate RAID or the one where backed up VMs are located?

[jamie2497]

Separate RAID on a dedicated backup box.

[foggy]

Got it. Let's let our engineer examine the logs you've provided.