I know I have been just throwing lots of ideas out here lately, sorry for the flood! I wanted to put things I think of to paper for the engineers to see specific field use cases.
Storage IO control has been wonderful since implementation but I think it could be more powerful. Also, I don't think machine datastores are really given enough configuration options.
Let's say Storage IO control is configured to allow faster backups without stunning machines. Unfortunately that means too much load for production hours.
If backups run in the normal backup window, but there were extra changes & they need to run a bit longer the next day this impacts production.
If an out of band backup is required for a specific task, this impacts production.
Why not allow sIO to be scheduled similar to a backup window?
If I could do that it would eliminate the allowable windows for me per job. I don't care if backups are running if my machines remain fast and responsive, but I don't want to set SIO to be lax enough to throttle my nightlies on a regular basis.
Perhaps it makes sense to merge into the regular backup window per job and add a third state: Permitted/Denied/restricted. When a job hits the restricted window it may continue to run but with enhanced throttling if so configured.
There are per repository maximum load control settings, but why not per source datastore?
I feel datastores would be a good configurable.
I don't group my jobs by datastore, I group by type/similarity for compression/dedupe. Some jobs reside on different datastores. It would be nice to take into account which datastores are allowed to be hit harder than others. Example being that I don't care if many concurrent jobs hit the dev datastore.
These configurables would basically lend flexibility whereas I can configure for best case scenario and push things when I can while not having to consider or be concerned about worst case scenario.