Control virtual disk processing priority to improve performance

[PHBG]

With every VBR release Veeam invest much to improving performance. I am thankful that Veeam listens their customer and have such good place as this forum -for direct communication between Veeam and end users
So I think if user can control and somehow prioritize virtual disk order during backup processing can achieve better speed/backup time.

Example 1.
You have a VM with 3 disks (1TB +1TB +2TB) but can process only 2 in parallel. If you backup 1 TB per 1 hour, 2 TB per 2 hour. They are two scenarios.
- Start Backup 1+1 TB disks in parallel, and after finish first one, process 2 TB disk – total backup time 3 hour
- Start Backup 2 TB disk and 1 TB in parallel. After finish 1 TB, start processing second 1 TB disk – total backup time 2 hour
It’s obviously that you will want to start bigger disks first. Many of us have experience when backup perform very well in the beginning (many disks processing in parallel) and very bad in the end, when one big virtual disk is processing only.

Example 2
Most of the Storages are ALUA and every lun/datastore have owning storage controller (SP). I’m pretty sure that many of us place bigger VM at least on 2 datastores on different SP owning controllers to balance storage load.
It’s normal to want during backup to load all storage controller equally to improve backup performance. And this can happen if you can control which virtual disks are processing in parallel = > more SPs, more HBAs, more throughput > better backup speed

[PetrM]

Hi Petar,

Thanks for the interesting idea!
Just wondering to know what is the reason to limit the number of disks to be processed in parallel? Why don't add more proxies or task slots on a proxy?

And one more thought about the example 2: storage I/O control allows you to specify I/O thresholds for individual datastores. This approach is even easier because you control I/O load at the level of datastore instead of configuring disk priorities based on their location: it might be challenging for large-scale environments with a huge number of VMs.

Thanks!

[PHBG]

I’m primary vSphere Admin and most of the challenges for VI/Backup is not the quantity of VMs but big/monster VMs.
We have 2 VMs that make more storage load (average 4-5 GB/s) than all other VMs (>1000). One of them is 150 TB.

Example 1 was just to simplify the situation. In the real world you can have a VM with 22 virtual disk, but can process 20 in parallel and according Murphy’s law the last 2 will be the bigger. We have a VM with 60 virtual disk. You know vmware now support 256 virtual disk…. Of course at your backup time there may be other backup/restore jobs consuming proxies tasks slot. So there will be always situation when you can’t process all virtual disk in one VM in parallel.
I think limitation one virtual disk to be process by one proxy will be valid in the near future. So I supposed that most of the people will want big disks to be with highest priority than small one from parallel process perspective in backup/restore job, because they are somehow “bottleneck” of the whole job. One more use case from me – a VM with multiple virtual disk which are not equal from storage perspective (located on different tiers, raid group, pools, performance….) so again you will want to have some control of processing order.

Example 2 – Storage I/O control is a great. But what I wrote is just adding extra value in processing speed, not alternative of IO control.
One real example – We have 1 VM with 1 OS disk + 6 Data disks, split on two datastores (DS) - two LUNs of different owning by different storage controller/processor (SP)
- Backup Job starting reading 3 disks (Hard Disk2,3,4) from first DS, there was storage saturation and StorageIO is activated.
- After 15-20min for the first time is staring reading disks from second DS (Hard Disk 5,6,7) – this LUN is owned by less utilize SP, so never activated throttling and latency keep in normal.
So you can see from the screenshot how the speed in the beginning is low (not processing hard disk from different Datastores), rise after that, and in the end also is low.



Storage IO control is reactive mechanism to control backup impact. There will be always balance between backup/restore speed (RPO/RTO) and impact of production. If some components from storage perspective (one SP, HBAs/NICs/Adapters, Links) are heavily load and Storage IO Control reduce the load, but another components (another SP, another Adapters, Links) still have extra load capacity – why not proactively load them more.

[PetrM]

Hi Petar,

Thank you so much for the detailed post and the clear explanation!
Usually, we decide on new features based on number of similar requests so let's consider this topic as +1 for prioritizing disks to be processed.

Thanks!