[Feature Request] Process big VMs first

[Seve CH]

Dear Veeam team

Here is the business case:

We use vSphere tags to map VMs to jobs. We have several tags like this: "Backup-Repo1-24H-2330-STD" (backup to repo 1, every 24h, starting at 23:30, no app processing/logs). All VMs with that tag are mapped to the corresponding Veeam Backup Job.

The problem is that with that kind of tags, there is no VM ordering possible: In Veeam job properties, there is only 1 tag and not the full list of VMs. For us it is easy to reach jobs with more than 150 VMs. Veeam start processing the VMs in an undefined order, which could lead to process the 140 VMs 15GB big first and then backup the 2-3 multiTB servers last which may exceed our backup window during an active full.

With parallelism we are able to achieve 2-3GB/s typical in aggregated backup speed. But with 1 big disk alone (i.e. 2TB disk), it is much slower (no parallelism, 130MB/s approx. best case, 2h10 min/TB).

I think it will be interesting to have the option to "start processing VMs by size, big VMs first" (and the option for small VMs first too) which I think should be easy to implement because you need to query vCenter's infrastructure anyway to get the VMs with a specific tag. Sorting them by their expected read size would be better, but much more complicated to implement IMHO because you will need to read the CBT for each VM and it will not be not worth it in most scenarios.

With that option, the big VM will start doing it's 4 hours backup and in parallel the rest of VMs will also be processed.

There are some workarounds (several tags to create several groups for the same job) but it complicates the system and makes the deployment and reporting less clean. We do not want to map VMs directly to jobs because we do not want to give vCenter administrators access to administer Veeam servers (role segmentation)

Thanks and best regards.

[dellock6]

Hi,
this is not to put down your idea, but to discuss about it.

Have you considered the fact that the bigger vm may not be the vm with more daily change rate? Processing vm by size would help for an active full backup, but not in any other situation where only changed blocks are tracked.

Indeed CBT may hep, but the problem is, CBT information are written to the CBT file only upon snapshot creation (our colleague Timo wrote an amazing post explaining this, I suggest everyone to read it: http://www.veeamhub.io/2017/01/backup-c ... explained/) so there's no way to know how much data a VM will consume, without first snapshot it. We may use the historical data of previous run, but if any unplanned change in data happens, this can ruin the entire idea. Not to say again there's no value in the idea, but there are many more consideration than simple VM size.

Luca

[Seve CH]

Have you considered the fact that the bigger vm may not be the vm with more daily change rate? Processing vm by size would help for an active full backup, but not in any other situation where only changed blocks are tracked.

Whe find the problem while processing active fulls (Every 15 days, 7 days or 2 days depending of the job) and the copy jobs (much worse) which read from source instead of synthesizing (we use always that setting). In many cases, we have the backup window overrun because the job is waiting for a big VM to finish.

So, for instance, real data right now: at this moment I have a job running. It is 9.2TB in total but one of the VMs is 7.8TB big. It is making an active full. Veeam has processed 600GB from other VMs before starting with the big one. All the other VMs of the job have finished by now but the job is waiting for the big one to finish. Had it started with the big one first and running the smaller ones later in parallel, it would finish almost 1 hour earlier.

Indeed CBT may hep, but the problem is, CBT information are written to the CBT file only upon snapshot creation (our colleague Timo wrote an amazing post explaining this, I suggest everyone to read it: http://www.veeamhub.io/2017/01/backup-c ... explained/) so there's no way to know how much data a VM will consume, without first snapshot it.

Thanks for the link. It is very instructive and very appreciated. So it is clear to me that unless guessing from historical data, there is no way to know in advance how many changes Veeam needs to process. No CBT then and not worth the complexity IMHO. But active fulls are easier to deal with

Not to say again there's no value in the idea, but there are many more consideration than simple VM size.

That's why it's the administrator who should have the choice instead (and that's why I have a manual gearbox in my car instead of an automatic one). I know my specific workload and requirements while you can't. At this moment it is not clear to me even the order Veeam will follow while using tags. I like well defined and simple configurations

This is what is happening on the example job:
Backup all VMs with X tag every 6 hours. There is only 1 tag introduced in this job.
Order followed this run: SZBNT092, SNPV-ICU-060, SZBNT039_HA, szbnt301, szbnt159 (7.8TB, still running after 2h 16' processing time, 55%), szbnt300, SZBNT161_HA, szbnt302. Nice ordering

Regards.
Seve.

[dellock6]

A quick and dirty workaround, have you considered to simply process that large 8TB vm with a dedicated job? At that point you would just need to schedule the job to start a few seconds before the other to have the priority you need. I know it's not elegant, but while you wait for a feature that may or not come in v10 or even later, this could solve your problem. Tags are aimed at managing batches of VMs as "cattle", all with the same options. If you have a "pet" like that large VM, it's better in my opinion to manage it like a pet and give it a dedicated job.

[itfnb]

A quick and dirty workaround, have you considered to simply process that large 8TB vm with a dedicated job? At that point you would just need to schedule the job to start a few seconds before the other to have the priority you need.

That's exactly what I do on our replication jobs:

1. Exchange
2. Primary file server
3. Everything else

[Spex]

If you use one tag to describe the complete processing I understand your concern about complicated job handling.
In our environment each tag has only one meaning. we have tags for schedule time, guest processing, retention, useraccount for guest agent.
So you can combine them all to your need. For each backuptime I have a backupjob - the rest is done by tags within the job.
If you have the need for a new backuptime (e.g. large vm) copy a backupjob, create a new tag and change the starttime+tag assignment that's all.

[ITP-Stan]

If you use TAGS you do not get a list of VM's in the JOB but you can actually manually add those 2 big VM's to the job and set it to process them first. Veeam will be smart enough not to process them twice.
However this does remove some of the flexibility the tags system offers. Because when you change the tags on those VM's they will still be processed by that job you added them manually too.

[Seve CH]

A quick and dirty workaround, have you considered to simply process that large 8TB vm with a dedicated job?

It is possible, but then we may have too many things running in parallel or the small VMs from the other job tmay steal all repository slots. Again, we prefer "well defined" settings than "random selection".

Also, moving these VMs to a different job will mean new full + chains and housekeeping the former jobs to clear manually old unneeded versions.

but while you wait for a feature that may or not come in v10 or even later, this could solve your problem

Well, we will need to adapt and live with it. But if it was implemented (just ordering should be not a big deal IMHO), it would be a plus

If you use one tag to describe the complete processing I understand your concern about complicated job handling.
In our environment each tag has only one meaning. we have tags for schedule time, guest processing, retention, useraccount for guest agent.
So you can combine them all to your need. For each backuptime I have a backupjob - the rest is done by tags within the job.
Yes, we use several tags for different kinds of jobs (repo, app processing or not, scheduling, retention, etc.).

We have different tags to manage for a VM the kind of retention, backup frequency (from every 1h to every 7 days), off-site copyjob settings, the target repository, app processing or not, etc. But we would like to avoid "thinking" as much as possible as to where to put a VM and doubling all tags to add a suffix "-bigVM" or "-1stGroup" to them complicates much more the infrastructure.

If you use TAGS you do not get a list of VM's in the JOB but you can actually manually add those 2 big VM's to the job and set it to process them first. Veeam will be smart enough not to process them twice.
However this does remove some of the flexibility the tags system offers. Because when you change the tags on those VM's they will still be processed by that job you added them manually too.

Exactly. We can do that but we lose the flexibility and role separation. It also breaks our automatic reporting system:

APP X uses VM1, VM2, VM3, VM4, which have tags A,B,C,and D which are well defined backup policies. There is no need to query Veeam to discover if somebody added a VM by hand here or there or if it lands in a job because it was nested from somewhere else. Our reporting just needs to query vCenter.

Thanks you all for this discussion and ideas