Feature Enhancement - Storage Latency

[ekisner]

Would it be possible to add an option to pause already-running jobs (not end them, just set their read speed to zero) when IO latency limits are triggered? Something tiered.

For example, a three tier implementation:
- [Source|Target|Any] Latency exceeds [50] ms, do not start new jobs
- [Source|Target|Any] latency exceeds [75]ms, throttle existing jobs
- [Source|Target|Any] latency exceeds [100]ms, reduce concurrent tasks by [2] on target repository and reduce throughput to zero on excess jobs

The third layer being new, would reduce the concurrency bit by bit until the latency got under control, even if it ends up down to 1 concurrent task.

I ask, because even with IO throttling enabled, I frequently see latency spikes up to 500ms on my production storage during backups. I can most effectively resolve this by reducing the level of parallel processing. But that requires some babysitting as I generally need to turn it back up again to meet backup windows.

It's mostly only a problem when I have significant quantities of data to back up such as a file server migration, or in my exact case, rebuilding full backups of file servers after my backup storage failed.

[BartP]

I ask, because even with IO throttling enabled, I frequently see latency spikes up to 500ms on my production storage during backups. I can most effectively resolve this by reducing the level of parallel processing. But that requires some babysitting as I generally need to turn it back up again to meet backup windows.

The core solution would be Tuning your Veeam infra, but I do think the ability to pause a backup job (not cancel) could be very handy for Ad-Hoc and troubleshooting performance.

[serkantek]

I think it will be a good idea to be able to throttle storage latency at certain times.
For example, if a file server backup job runs into operating hours you wouldn't want the storage to be getting hit by backup and also users.
It will be good if you can set latency for example between 7am - 6pm to Xms and between 6pm - 7am to Xms

[Gostev]

Would it be possible to add an option to pause already-running jobs (not end them, just set their read speed to zero) when IO latency limits are triggered?

We've made the conscious decision to limit throttling (not throttle each job to zero) to let the already running jobs still progress. Otherwise, it is easy to run into much worse issues due to the VM snapshot size. The spikes you're seeing are likely due to VM snapshot commit anyway - and the longer snapshots will be around, the worse these will be.