Advice on best practises for scheduling backups

[mattiasd]

Hello all,

Im running Veeam 5 with the vSphere hotfix and backing up about 150 VMs.

My problem is with the scheduling, the sum of it is that certain jobs keep taking longer time then expected resulting in a 2nd job starting and consuming resources, eventually i have 3-4 jobs running simultaneously and the Veeam server basically grinds to a complete halt due to lack of CPU/bandwith/memory etc.

I've been experimenting with a few different setups to resolv this which i will describe below. Basically im looking to hear how other people backing up 100+ VMs are scheduling and grouping their machines to get a reliable schedule.
As a note i'm currently forced to starting one job manually every day when i get in to work and start a 2nd once its finished etc which is very tedious and prone to human error.

Currently i've set it up as follows:
* 2 jobs with 5 each of our largest VMs - These jobs are the ones consuming the most time (often 24h+) - i've tried scheduling these 2 with 24h space between them but they keep colliding at times when they take extra time.
* 5 jobs with 10 VMs each which are hosted customer servers. Ideally i'd like to back these up nightly but that simply isnt feasible since the jobs take at a minimum 8 hours or so to run, so i'm trying to schedule them one each night during the work week, this hasnt really worked either since they sometimes take longer and sometimes shorter to run so its very hard to predict.
* 1 job containing about 10 small internal servers (DNS, DHCP etc) - This one is fairly manageable and usually only take about 6 hours to run fairly predictably (probably since its pretty static servers that dont change much).
Besides this there is quite a few jobs with single servers that need more frequent backups (nightly) which doesnt really cause any issues since they run pretty quickly.

I'm wondering if i would get better performance/better ability to time box the backups if i would make each job a single VM? It would be a large initial job but if i got it working unattended it would definitely be worth it.
Is it normal to have such varying backup speeds? Also im using reverse incremental backups but i dont get the feeling that it takes "less and less" time to do the backups like you would expect from an incremental backup.
The 2 big jobs can take anywhere from 18h - 3 days+ sometimes. Could it be a misconfiguration of my SAN?

Also, is there any downside to doing backups during daytime (when the servers are actively being used)? And is there anyway to avoid this if a job takes 24h+ to run?

Basically im looking for input on how other people are managing this amount of VMs and what i can do to finally be able to fully automate my backups!

Best Regards and thanks in advance,
Mattias

[foggy]

Mattias, I would strongly recommend to upgrade to Veeam Backup & Replication version 6 as, among other cool features, it introduces a lot of stuff right in the area of jobs concurrency. No need to babysit backups now as VBR handles jobs overlapping automatically for you dynamically assigning them to available proxies (servers doing actual backups) with smart load balancing between them.

[Vitaliy S.]

Moreover, v6 will give you a brief summary on possible bottlenecks for the job. These stats are quite handy when you start analyzing performance issues.