Storage Latency Control

[dbr]

Hi,

We use storage latency control in order to decrease the load on our NetApp 8040 during backup window. Even though the read latency is almost all the time below the configured threshold the write latency isn't (we use Grafana to get performance data for our NetApp). The values are 20ms for new tasks and 30ms for throttling. We noticed write latency up to 300ms during active full backups. I know that Veeam uses VMware performance metrics but does Veeam also use write latency value for storage latency control calculation or only read latency? Must the current value be below the threshold for a couple of seconds or does Veeam check the average value? Could anyone please clarify how storage latency control works in detail? I could neither find some more detailed information in help center nor in forum.

Thanks,
Daniel.

[PTide]

Hi,

does Veeam also use write latency value for storage latency control calculation

No, it does not.

Must the current value be below the threshold for a couple of seconds or does Veeam check the average value?

Veeam monitors the storage read latency on the production datastores using real-time metrics from the hypervisor. By default, metrics from the hypervisor are collected every 20 seconds. These settings are inherited from vSphere. Also Veeam starts processing VM disks residing on the same datastore with a 40-60 second time offset. This offset helps Veeam evaluate the current I/O load on the datastore.

Please check "Storage Latency Control" section of this BP guide, and feel free to post additional questions should you have any.

Thanks

[dbr]

Thanks for your quick reply. I already noticed the section in BP guide. It's good to know that Veeam does not respect write latency at the moment but would it be an option for future versions? In my opinion checking write latency should at least be done before triggering snapshot commit because snapshot removal is a load intensive operation and as far as I know it cannot be throttled through VMware.

[PTide]

In my opinion checking write latency should at least be done before triggering snapshot commit because snapshot removal is a load intensive operation and as far as I know it cannot be throttled through VMware

What actions would you expect from VBR if write latency was above the threshold? Should it throttle snapshot commitment, or postpone it for X minutes/until latency is back to normal? Or should it just give you a warning?

Thanks

[dbr]

Good question. A warning would be a good start but would also be annoying because it doesn't change anything. Easiest thing is maybe to throttle read throughput for running tasks even more when write latency is too high. I would assume that write latency will decrease once the throughput is throttled more. Another Idea is to reduce concurrent snapshot commitment based on latency. Postpone commitment too long is neither a good option. Snapshots could get too large when write latency doesn't get back to normal.

I think that storage latency control is the more dynamical way versus limiting concurrent tasks or data rates on repositories. I guess it's difficult to implement respect of write latency without reducing overall throughput too much but maybe it's enough food for thought for your developers.

As workaround I will set latency control to a lower value and see what changes.

[dellock6]

The main difference, however, would be that, while read latency can be directly influenced by Veeam via BIOC and either throttling or pausing our backups, write latency can just be checked, but once we issue a snapshot removal command to vSphere there's no control anymore, the snapshot commit is executed. So, I'm not sure how much it would be helpful; we already have solutions in place to lower the load for snapshot commit by taking into account how many snapshots can be concurrently committed, plus other solutions from VMware can be implemented (upgrade to vsphere 6.x to leverage the new mirror driver, vsan/vvol...).