Slow backup performance of Microsoft Teams

[Pagrossman]

Hello guys,

I have opened service ticket 04771512.

The problem at the beginning was that Veeam Backup for Microsoft Office 365 told us that the bottleneck was Target, but our disk array, CPU, RAM and network weren't performed.
Of course we want to use Veeam Backup for Microsoft Office 365 in maximum performance, so when VBO tell us that the problem is CPU, we will increase count of CPU, if the problem will be RAM, we increase RAM and the same for disk array.
But unfortunately VBO says Target and all components weren't performed.
During resolving the ticket we tested a lot of things, defragmentation of Windows disk, defragmentation of repository, changing disk array, creating new server, RAM disk, etc... and the performance was still same.

Few days ago, during another test I found that the core problem is in resolving backup job objects, because, resolving backup job objects use only one proxy thread and it seems that it is not enough.

Whole backup (incremental backup) of our Teams (approximately 800 objects) is on 30-35 minutes.
If I set only one thread on proxy and started the backup job, this thread was of course assigned to resolving job objects. The process of resolving (without data backup) was on 34 minutes, which is the same time as backup job with 64 threads. After the last object was resolved, Backup job assigned the thread to backup process. In this time I changed the number of proxy threads from 1 to 64 and I waited what will happen. The backup job started backup resolved objects as a shot (by the very similar speed as backup of SharePoint or Exchange, where we don't have performance problem) and the backup job finished after 7 minutes with 800 objects... at the same time, now the disk array was performed between 60-90%, which is perfect. The backup job increase the speed by 400%!!!
Here it is possible to see that the bottleneck is on resolving backup job objects.

Unfortunately I have a feeling that this information was thrashed by solver and he told me that I need to do another defragmentation and add another Veeam proxy, but why?

Please is it possible to assign multiple threads to resolve backup job objects that will increase the performance of the entire backup by hundreds of percent?

Thank you

[Polina]

Hi Jan,

It's sad to know that no one from Veeam was able to explain to you how threads allocation works and that you had to perform these tests on your own.

Your findings are correct - VBO uses just 1 thread for resolving objects. This thread reads objects, splits them into smaller batches of data, and sends them to other threads for processing.

Why do we use just one thread? Mainly because with multiple threads, backup would be quickly throttled by Microsoft, and instead of the performance increase, it'd result in backup stuck for hours.

Thanks!

[Pagrossman]

Hi Polina,

thank you for your message.

Why is for example 5 threads for resolving job objects problem?
I still can create 5 separated jobs with 1/5 of all teams each, but each job has own 1 thread for resolving.

I don't know why, but resolving of 657 sites in SharePoint backup job is on 9 minutes and resolving of 940 mailboxes is on 13 minutes. These are the good times.
But resolving of 278 teams is on 30-35 minutes?

In she SharePoint and Exchange backup job, there is possible to see that resolving is very fast, and then the bottleneck is disk array, but in case of Teams the absolutely bottleneck is resolving (only of 278 Teams).

[Polina]

If there were 5 resolving threads you still could have 5 separate jobs and, thus, 25 threads would be instantly wasted on resolving objects (and get throttled quickly as we saw in our multiple tests).

Nevertheless, your points are valid and make sense. Building a queue of Teams objects takes more time because each team is a 3-in-1: mailbox, site, and metadata. And VBO gets information about a team first and then resolves its mailbox and site. It seems like there's some room for improvement, and I'll certainly discuss it with the RnD team.

Thanks!

[yusaku]

Hi Polina,

I have the same issue with this problem. Is there any plan to improve the process of resolving objects?

Thanks,
Yusaku

[Mike Resseler]

Hi Yusaku,

There are always plans to resolve the performance, but keep in mind that in the backend of M365 a lot of changes are happening there. One of the items we are looking into is making sure as much graph API's as possible are being used. Those changes will change the performance but will need to be taken step-by-step