-
- Enthusiast
- Posts: 80
- Liked: 17 times
- Joined: Nov 27, 2012 1:00 pm
- Full Name: Lewis Berrie
- Location: Southern England
- Contact:
Is Storage Latency Control fully dynamic?
I have recently enabled Storage Latency Control for a customer, as we have consolidated down on to a few, large datastore and it seemed a sensible option to enable.
I have seen jobs get throttled (by having "[throttled]" appear next to the hard disk being backed up), so I know it is kicking in.
My question is: does it ever get turned off for a hard disk, after it has been turned on?
So, does the datastore latency keep being monitored, and when it drops below the monitor threshold again, does Veeam then "release" the throttle and allow the VM to be backed up at full speed?
The reason I ask is that the backups contain a mix of VMs: some small, some large (file servers, etc.). I have seen the file servers get throttled during the initial backup process, but when all the other VMs are backed up and the file servers are the only ones being backed up, they appear to stay throttled until they complete. If they are doing full backups (a few HA events due to power issues have disrupted CBT, for example), then they take a very long time to complete, even though looking at datastore latency values in VMware itself shows values of less than 5ms.
So I have the feeling that once storage latency control kicks in for a VM's hard disk, it doesn't ever get released.
The documentation only says that latency is monitored every 20 seconds through the hypervisor itself.
One side note: Veeam ONE 9.5 had an issue (fixed in Update 1) whereby it displayed the incorrect value for datastore latency (it's in the release notes: "Datastore latency performance counter is showing wrong measurement units"). Does Veeam B&R use the same codebase for this, and if so is the same issue present (or absent) in Veeam 9.5 B&R (with or without Update 1)? I ask because I have seen throttling kick in when the datastore isn't close to the latency value specified in the settings, and might explain the behaviour I am seeing. This customer was receiving floods of datastore latency alerts from Veeam ONE after upgrading to 9.5, so I had to turn that alarm off (as it was clearly not showing correct values); it's back one after upgrading to Update 1 and seeing the issue fixed in the release notes
I have seen jobs get throttled (by having "[throttled]" appear next to the hard disk being backed up), so I know it is kicking in.
My question is: does it ever get turned off for a hard disk, after it has been turned on?
So, does the datastore latency keep being monitored, and when it drops below the monitor threshold again, does Veeam then "release" the throttle and allow the VM to be backed up at full speed?
The reason I ask is that the backups contain a mix of VMs: some small, some large (file servers, etc.). I have seen the file servers get throttled during the initial backup process, but when all the other VMs are backed up and the file servers are the only ones being backed up, they appear to stay throttled until they complete. If they are doing full backups (a few HA events due to power issues have disrupted CBT, for example), then they take a very long time to complete, even though looking at datastore latency values in VMware itself shows values of less than 5ms.
So I have the feeling that once storage latency control kicks in for a VM's hard disk, it doesn't ever get released.
The documentation only says that latency is monitored every 20 seconds through the hypervisor itself.
One side note: Veeam ONE 9.5 had an issue (fixed in Update 1) whereby it displayed the incorrect value for datastore latency (it's in the release notes: "Datastore latency performance counter is showing wrong measurement units"). Does Veeam B&R use the same codebase for this, and if so is the same issue present (or absent) in Veeam 9.5 B&R (with or without Update 1)? I ask because I have seen throttling kick in when the datastore isn't close to the latency value specified in the settings, and might explain the behaviour I am seeing. This customer was receiving floods of datastore latency alerts from Veeam ONE after upgrading to 9.5, so I had to turn that alarm off (as it was clearly not showing correct values); it's back one after upgrading to Update 1 and seeing the issue fixed in the release notes
-
- Chief Product Officer
- Posts: 31538
- Liked: 6710 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Is Storage Latency Control fully dynamic?
Yes, it's fully dynamic and so any applied throttling will be removed completely once production datastore latency allows.
-
- VP, Product Management
- Posts: 6011
- Liked: 2843 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Is Storage Latency Control fully dynamic?
Indeed, we constantly monitor the latency throughput the job run and should return to normal performance if latency recovers, however, I believe I have, on occasion, also seen the behavior that the OP describes and it's always made me wonder if somehow the algorithm that is supposed to crank the throughput back up is not aggressive enough, but I have never attempted to actually research.. Perhaps I'll have to add that to my list to try to reproduce in the lab.
-
- Enthusiast
- Posts: 80
- Liked: 17 times
- Joined: Nov 27, 2012 1:00 pm
- Full Name: Lewis Berrie
- Location: Southern England
- Contact:
Re: Is Storage Latency Control fully dynamic?
Anton/Tom,
Many thanks for the information
Tom: good to hear I may not be the only one to see this if I can provide any debug information, please let me know. Although I don't rate it very high on the criticality scale
Many thanks for the information
Tom: good to hear I may not be the only one to see this if I can provide any debug information, please let me know. Although I don't rate it very high on the criticality scale
-
- Influencer
- Posts: 23
- Liked: never
- Joined: Jul 18, 2017 1:58 pm
- Full Name: Helen
- Contact:
Re: Is Storage Latency Control fully dynamic?
Hey,
I came to search the forums because of this issue too.
Basically we are backing up groups of file severs some in excess of 10TB
All of our backup jobs kick off at once and they queue until backup proxies have resources to process.
I note what with jobs on some of our biggest file servers that they start to back up with the the others and are throttled due to storage I/O control we have configured. These take much much longer to backup due to their size and they remain throttled right up until they complete despite the fact that every other backup has finished hours ago and the production storage it's reading from is quiet and veeam is doing nothing else.
A full backup took about 51hrs!
I came to search the forums because of this issue too.
Basically we are backing up groups of file severs some in excess of 10TB
All of our backup jobs kick off at once and they queue until backup proxies have resources to process.
I note what with jobs on some of our biggest file servers that they start to back up with the the others and are throttled due to storage I/O control we have configured. These take much much longer to backup due to their size and they remain throttled right up until they complete despite the fact that every other backup has finished hours ago and the production storage it's reading from is quiet and veeam is doing nothing else.
A full backup took about 51hrs!
-
- Veeam Software
- Posts: 21070
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Is Storage Latency Control fully dynamic?
Helen, please allow our support engineers investigate this behavior. Thanks.
Who is online
Users browsing this forum: Victor van Hoek SLTN and 65 guests