Is Storage Latency Control fully dynamic?

VMware specific discussions

Is Storage Latency Control fully dynamic?

Veeam Logoby Lewpy » Sun Jan 29, 2017 9:24 am

I have recently enabled Storage Latency Control for a customer, as we have consolidated down on to a few, large datastore and it seemed a sensible option to enable.
I have seen jobs get throttled (by having "[throttled]" appear next to the hard disk being backed up), so I know it is kicking in.
My question is: does it ever get turned off for a hard disk, after it has been turned on?
So, does the datastore latency keep being monitored, and when it drops below the monitor threshold again, does Veeam then "release" the throttle and allow the VM to be backed up at full speed?
The reason I ask is that the backups contain a mix of VMs: some small, some large (file servers, etc.). I have seen the file servers get throttled during the initial backup process, but when all the other VMs are backed up and the file servers are the only ones being backed up, they appear to stay throttled until they complete. If they are doing full backups (a few HA events due to power issues have disrupted CBT, for example), then they take a very long time to complete, even though looking at datastore latency values in VMware itself shows values of less than 5ms.
So I have the feeling that once storage latency control kicks in for a VM's hard disk, it doesn't ever get released.
The documentation only says that latency is monitored every 20 seconds through the hypervisor itself.
One side note: Veeam ONE 9.5 had an issue (fixed in Update 1) whereby it displayed the incorrect value for datastore latency (it's in the release notes: "Datastore latency performance counter is showing wrong measurement units"). Does Veeam B&R use the same codebase for this, and if so is the same issue present (or absent) in Veeam 9.5 B&R (with or without Update 1)? I ask because I have seen throttling kick in when the datastore isn't close to the latency value specified in the settings, and might explain the behaviour I am seeing. This customer was receiving floods of datastore latency alerts from Veeam ONE after upgrading to 9.5, so I had to turn that alarm off (as it was clearly not showing correct values); it's back one after upgrading to Update 1 and seeing the issue fixed in the release notes :)
Lewpy
Enthusiast
 
Posts: 39
Liked: 9 times
Joined: Tue Nov 27, 2012 1:00 pm
Location: Southern England
Full Name: Lewis Berrie

Re: Is Storage Latency Control fully dynamic?

Veeam Logoby Gostev » Sun Jan 29, 2017 5:55 pm

Yes, it's fully dynamic and so any applied throttling will be removed completely once production datastore latency allows.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Is Storage Latency Control fully dynamic?

Veeam Logoby tsightler » Mon Jan 30, 2017 4:59 pm

Indeed, we constantly monitor the latency throughput the job run and should return to normal performance if latency recovers, however, I believe I have, on occasion, also seen the behavior that the OP describes and it's always made me wonder if somehow the algorithm that is supposed to crank the throughput back up is not aggressive enough, but I have never attempted to actually research.. Perhaps I'll have to add that to my list to try to reproduce in the lab.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Is Storage Latency Control fully dynamic?

Veeam Logoby Lewpy » Mon Jan 30, 2017 6:41 pm

Anton/Tom,

Many thanks for the information :)
Tom: good to hear I may not be the only one to see this :wink: if I can provide any debug information, please let me know. Although I don't rate it very high on the criticality scale :)
Lewpy
Enthusiast
 
Posts: 39
Liked: 9 times
Joined: Tue Nov 27, 2012 1:00 pm
Location: Southern England
Full Name: Lewis Berrie

Re: Is Storage Latency Control fully dynamic?

Veeam Logoby IHeartCats » Mon Aug 07, 2017 2:57 pm

Hey,

I came to search the forums because of this issue too.

Basically we are backing up groups of file severs some in excess of 10TB

All of our backup jobs kick off at once and they queue until backup proxies have resources to process.

I note what with jobs on some of our biggest file servers that they start to back up with the the others and are throttled due to storage I/O control we have configured. These take much much longer to backup due to their size and they remain throttled right up until they complete despite the fact that every other backup has finished hours ago and the production storage it's reading from is quiet and veeam is doing nothing else.

A full backup took about 51hrs!
IHeartCats
Lurker
 
Posts: 2
Liked: never
Joined: Tue Jul 18, 2017 1:58 pm
Full Name: Helen

Re: Is Storage Latency Control fully dynamic?

Veeam Logoby foggy » Tue Aug 08, 2017 3:15 pm 2 people like this post

Helen, please allow our support engineers investigate this behavior. Thanks.
foggy
Veeam Software
 
Posts: 14736
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 17 guests