Feature Request - VM anti-affinity within a job

[glamic26]

I would like to raise a request for a feature that would allow us to keep two or more VMs in a single job but specify that snapshot operations should not occur on the selected VMs at the same time. Some kind of VM back anti-affinity within a job.

We are getting an increasing number of customers requiring us to split their VMs into different jobs because when they are in the same job they can potentially be snapshotted or snapshot removed at the same time. This causes the VMs to stun at the same time and causes problems with various applications like Exchange, MS clusters, SQL, and even AD. The customers complain that even though they have a clustered SQL environment, for example, their application scripts disconnect overnight when the backup runs and our investigations find that both VMs were performing snapshot operations at the same time which causes the cluster to shutdown (we already have the highest recommended cluster timeout settings configured).

Our current solution is to remove one of the VMs from the existing job and put it into a job of its own to run after successful completion of the first job. Whilst this resolves the application timeout issue the increasing number of jobs we now have to do this for is significantly increasing the number of jobs that we have to look after. It also means we don't get the benefits of deduplication in a larger job (particularly frustrating with replicated data applications like Exchange) and means additional snapshots on the storage array and more API calls etc meaning an increased backup window. Veeam also recommend not using the job chaining scheduler as well.

The solution we would like to see would be a setting in the backup job to allow us to keep these VMs in the same job but will prevent snapshot operations from occurring on specific VMs in the job at the same time. This would allow us to still benefit from the deduplication not only with each other but also all of the other VMs in the same job, and would give us less jobs to manage and would reduce the number of jobs that we have to chain together.

[foggy]

Wouldn't I/O control settings allow to achieve what you're after?

[glamic26]

I don't see how it possibly could. You might need to explain a bit further. We have I/O control turned on to limit the tasks on the datastores when they hit a certain latency but I don't know of any way to use that to prevent two VMs from performing snapshot tasks at the same time.

[foggy]

It wouldn't prevent two VMs from performing snapshot tasks at the same time, however, will prevent assignment of new tasks if the datastore is overloaded, which will allow to avoid excessive parallel snapshot commits. However, if even two concurrent snapshot operations are not desired, then VMs can be processed sequentially to avoid them.

[veremin]

Have you thought about limiting maximum number of concurrent tasks proxy or repository can process? In this case, VMs will not be processed in parallel and there will be only one snapshot at any given moment. Thanks.

[glamic26]

Thanks for the suggestions. I just don't see them being a scalable solution. We backup 1000+ VMs on the Veeam Backup Server and it wouldn't be feasible to reduce the number of concurrent tasks proxies or repositories can process just to prevent two VMs from being snapshotted at the same time. The knock on effect to the performance of the rest of the environment would surely be catastrophic?

[foggy]

Yes, it will reduce the overall backup performance. Btw, if speaking about your suggestion, do you realize that this might result in the fact that the time the VM runs on the snapshot will be sometimes artificially extended (due to the fact that it is waiting while another snapshot commit is being performed)?

[glamic26]

Yes I would expect that but it would be more than acceptable for us if it allowed us to stop having to create multiple jobs.

My idea would be:
We have a job with 26 VMs called VM-A through to VM-Z
VM-D and VM-E are two nodes of a window cluster that should not have snapshot operations at the same time
Veeam initiates the VMware snapshots on all of the VMs in the job but making sure that the "take snapshot" on VM-E does not start until the same task has been completed on VM-D (or vice versa)
It doesn't matter if snapshots are present on both VM-D and VM-E at the same time, I am not trying to avoid that. I just don't want the "take snapshot" task happening at the same time as this is what causes the stun that causes cluster timeouts
The job will then backup the data in the normal way (in our case either nbd or storage snapshot)
When the time comes to remove the snapshots (either after successful storage snapshot in the case of this feature, or once the backup is complete in the case of nbd) Veeam sends the API for VMware to remove the snapshots but will not allow the API to be sent to either VM-D or VM-E if either are still removing the snapshot (again to avoid stuns at the same time).
I would only expect a slight extension to the backup time over a normal job, and a reduced backup time versus splitting in a second job (as well as dedupe improvements)

[foggy]

Yep, your request is pretty clear. Thanks for the detailed feedback.