Large amount of backup clients

[jcorbin121]

Does anyone have any tips for dealing with 225 (and growing weekly) backup clients? I have 4 ESXi clusters and only so many hours in the day to backup... anyone else doing this successfully? I am getting 20-25mb/s on our 1gb network, soon we will be on 10gbE which I presume will help. I have been scheduling 3to4 VM's every 30 minutes which has been working most nights, but seems like when I get into the middle of the night things start to slow and backups bunch up and hang. Have had a couple tickets open with support and no real solution as of yet. Would love to chat with someone who is having success on multiple clusters with large amount of VM's


John

[Vitaliy S.]

How many proxy servers do you have installed in your environment? In situation like yours I would recommend distributing all the load among multiple proxy servers (VM data movers), that would allow you to have smaller backup window.

Also please search these forums for existing discussions, as there are plenty of them. Here is one of them: Are you a large customer backing 300+ VMs?

[tsightler]

I work with customers every day that are backing up 1000+ VMs. Are you getting 20-25MB/s on incremental runs? That's pretty bad for incremental. What is your backup target? Are you using V6?

[jcorbin121]

Yes 20-25mb/s on reversed incrementals. Yes using V6, My backup target is a CIFS share on a Quantum DXI8500, We write at over 100gb per hour using Commvault on same VM to same target (DXI) which would be completely tcp/ip transfer. So far not seeing the benefit, but maybe I have someting wrong. I have a Veeam backup server and 5 proxies (all VM) which have 8 CPU and 32GB RAM. I have multiple jobs with 2-4 VM's per job that are scheduled so no more than 4-5 are running at any one time.

[tsightler]

What backup mode are you attempting to use? Reverse incremental mode is very likely to perform poorly with the DXI since it is a deduplication solution and will not be able to push the required I/Os. Typically to see the best performance with a deduplicating device you must use forward incremental mode.

Also, you state that you are writing at 100GB/hr with Commvault, but that's only about 28MB/s, so that's effectively in the same ballpark. If your seeing 20-25MB/s * 4 jobs, that's much faster, but perhaps I'm not understanding your exact setup or the comparisons you are drawing.

[jcorbin121]

Tom et al - I applied hotfix 3 today and more improvement. Monday & Tuesday nights backups ran smooth as silk, I bundled up between 10-20 VM's per job, removed the guest file system indexing, kept the Reverse incremental job type. All 225 VMs had success. Today the GUI is way better in responsiveness - feeling a lot better about everything after your visit. Thanks again!

John

[tsightler]

Great news John! It was good to be able to get onsite and discuss your environment. Please keep us posted as you continue to grow out your environment.