Host-based backup of VMware vSphere VMs.
Post Reply
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Is Storage Latency Control fully dynamic?

Post by Lewpy »

I have recently enabled Storage Latency Control for a customer, as we have consolidated down on to a few, large datastore and it seemed a sensible option to enable.
I have seen jobs get throttled (by having "[throttled]" appear next to the hard disk being backed up), so I know it is kicking in.
My question is: does it ever get turned off for a hard disk, after it has been turned on?
So, does the datastore latency keep being monitored, and when it drops below the monitor threshold again, does Veeam then "release" the throttle and allow the VM to be backed up at full speed?
The reason I ask is that the backups contain a mix of VMs: some small, some large (file servers, etc.). I have seen the file servers get throttled during the initial backup process, but when all the other VMs are backed up and the file servers are the only ones being backed up, they appear to stay throttled until they complete. If they are doing full backups (a few HA events due to power issues have disrupted CBT, for example), then they take a very long time to complete, even though looking at datastore latency values in VMware itself shows values of less than 5ms.
So I have the feeling that once storage latency control kicks in for a VM's hard disk, it doesn't ever get released.
The documentation only says that latency is monitored every 20 seconds through the hypervisor itself.
One side note: Veeam ONE 9.5 had an issue (fixed in Update 1) whereby it displayed the incorrect value for datastore latency (it's in the release notes: "Datastore latency performance counter is showing wrong measurement units"). Does Veeam B&R use the same codebase for this, and if so is the same issue present (or absent) in Veeam 9.5 B&R (with or without Update 1)? I ask because I have seen throttling kick in when the datastore isn't close to the latency value specified in the settings, and might explain the behaviour I am seeing. This customer was receiving floods of datastore latency alerts from Veeam ONE after upgrading to 9.5, so I had to turn that alarm off (as it was clearly not showing correct values); it's back one after upgrading to Update 1 and seeing the issue fixed in the release notes :)
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is Storage Latency Control fully dynamic?

Post by Gostev »

Yes, it's fully dynamic and so any applied throttling will be removed completely once production datastore latency allows.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Is Storage Latency Control fully dynamic?

Post by tsightler »

Indeed, we constantly monitor the latency throughput the job run and should return to normal performance if latency recovers, however, I believe I have, on occasion, also seen the behavior that the OP describes and it's always made me wonder if somehow the algorithm that is supposed to crank the throughput back up is not aggressive enough, but I have never attempted to actually research.. Perhaps I'll have to add that to my list to try to reproduce in the lab.
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Re: Is Storage Latency Control fully dynamic?

Post by Lewpy »

Anton/Tom,

Many thanks for the information :)
Tom: good to hear I may not be the only one to see this :wink: if I can provide any debug information, please let me know. Although I don't rate it very high on the criticality scale :)
IHeartCats
Influencer
Posts: 23
Liked: never
Joined: Jul 18, 2017 1:58 pm
Full Name: Helen
Contact:

Re: Is Storage Latency Control fully dynamic?

Post by IHeartCats »

Hey,

I came to search the forums because of this issue too.

Basically we are backing up groups of file severs some in excess of 10TB

All of our backup jobs kick off at once and they queue until backup proxies have resources to process.

I note what with jobs on some of our biggest file servers that they start to back up with the the others and are throttled due to storage I/O control we have configured. These take much much longer to backup due to their size and they remain throttled right up until they complete despite the fact that every other backup has finished hours ago and the production storage it's reading from is quiet and veeam is doing nothing else.

A full backup took about 51hrs!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Is Storage Latency Control fully dynamic?

Post by foggy » 2 people like this post

Helen, please allow our support engineers investigate this behavior. Thanks.
Post Reply

Who is online

Users browsing this forum: No registered users and 77 guests