Select mailboxes takes long time

[tomhkr]

I have about 60,000 mailboxes in my Offce 365 tenant. Trying out the Veeam Office 365 backup, it stucks with selecting mailboxes to backup. "Loading mailboxes.." stays on for very long time (30+ minutes) and doesn't display any boxes.

How long should I wait?

Process "Veeam.Archiver.Service.exe" generates incoming network traffic with about 150-300 kByte/s but stop occasionally.

[tomhkr]

Just noticed the process has allocated all RAM and swapping heavily. Need a beefier server for this.. it has 8 GB RAM.

[Mike Resseler]

Hi,

This is interesting. So you are trying to select some mailboxes but it can't enumerate the list of mailboxes. Did the process finally succeeded or never ended?
In the meantime I will try to get more information on the RAM and the swapping

Thanks
Mike

[tomhkr]

To clarify, these are the steps taken:

• Add backup job
• At "Select mailboxes to backup" i choose "Backup the following mailboxes:" and press "Select.."
• The "Select mailboxes" box displays "Loading mailboxes...." forever. The process consumes all memory there is and nothing happened for about an hour, so I killed it.

I'm giving it a new try at a machine with 32 GB of RAM instead, and will get back in this thread with the results.

[Dima P.]

tomhkr,

Please let us know how it goes. We might need get access to the application debug logs in case of a failure.

[tomhkr]

So it finally succeeded in loading all the mailboxes with a beefier server.
The problem was, I think, that with 60.000 mailboxes the first server ran out of memory and began swapping. A swapping server seldomly does any good.

Next problem I ran into is the "An entry with the same key already exists" which I see there is a thread on already and fix coming.

It is still slow, though. Loading all the boxes takes 15-30 MINUTES. This on a server with 32 GB RAM and a 10 GbE internet connection.

There really ought to be some sort of filtering option when selecting mailboxes or creating jobs dynamically based on filter criterias.

This could be accomplished with one of the following:

• Mailbox membership (possibly problematic?; mailbox can be member of multiple groups)
• Retention policy
• Custom attributes

Look, it became a feature request!

[Dima P.]

tomhkr,

Thanks - good catch! Is it possible to share the logs anyway? I bet DEV team would like to investigate your case.

[tomhkr]

Certainly. Where can I send them?

[Dima P.]

If you have a dropbox or googledrive account (any other cloud hosting works too), just send me the link via PM. You can also upload the logs to the google drive folder I've sent you via PM.

[Kostya]

Confirmed. Will be improved in the GA build.

[Kostya]

It is still slow, though. Loading all the boxes takes 15-30 MINUTES. This on a server with 32 GB RAM and a 10 GbE internet connection.

Tom, sent you a link via PM to the experimental performance improvements and fixes.

[JaySt]

Tom, just curious, could you also share some findings when you're able to run a actual back-up? Performance for example?
interesting to see what the software is able to do with that volume of mailboxes.

[jchillman]

I would also be interested in the performance data. We have a couple of Large O365 Domains that we will be looking to backup.

[Mike Resseler]

Hi James, without being able to give you performance data for now... When you say large... What # are you talking about?

[jchillman]

Hi Mike,

We have one around the 25k Seat mark and a couple around the 5-6k mark.

[tomhkr]

I have applied the experimental fix and tried creating a new backup job.
It is significantly faster, it now took "only" 11 minutes to arrive to mailbox selection.

It would still be a very welcomed feature to be able to select mailboxes with some criteria, like aforementioned group or list membership, custom attributes et.c. , to search for and browser through tens of thousands of mailboxes to find 15 of them is cumbersome.

Regarding performance on running backup job, the read rate from Office 365 appears to be about 1.2-1.5 Mbyte/s
This is quite low, considering large organizations. But I beliieve this is due to rate limiting at Microsoft, cannot blame Veeam for the low transfer rate. I did a few mass migrations to Office 365 and they all exhibited the same speeds.

[foggy]

Regarding performance on running backup job, the read rate from Office 365 appears to be about 1.2-1.5 Mbyte/s

With the purpose of getting some stats, what is the total number of mailboxes and what was the entire job duration/repository size?

[tomhkr]

I did a test backup with only 2 mailboxes from the IT department since I know some of them contains some meat.

Here are the stats:

First full backup:
xxxx.onmicrosoft.com; 2016-10-19 15:55; 19 minutes, 26 seconds; Items: 37866 (1,59 GB); Processing rate: 32 items/s (1,40 MB/s)

Incremental 8 hours later:
xxxx.onmicrosoft.com; 2016-10-20 00:01; 1 minute, 40 seconds; Items: 16 (642,66 KB); Processing rate: 0 items/s (6,39 KB/s)

Please let me know if I can run more test jobs.

Btw just for kicks I did
(Get-Mailbox -ResultSize Unlimited).Count
59279


I did not measure it accurately but I believe it took about 10-15 minutes. So the issue with listing mailboxes at large tenants is probably a limit with Office 365. This would make it even more interesting to get the ability to filter on properties.

[Mike Resseler]

Hi Tom,

Interesting what you did. So just using PS fron your workstation (I assume?) you did a simple count to verify how much mailboxes there are in your environment and it took that long to get the results? Did I understand that correctly?

Mike

[tomhkr]

Yes that's correct.
It takes huge amounts of time, cpu and memory to list all mailboxes in a large tenant like this.
The local Powershell-process bloats to about 7 GB used RAM while fetching the boxes.
I ran it again, this time measuring properly.

Measure-Command {(Get-Mailbox -ResultSize Unlimited).Count}
Days : 0
Hours : 0
Minutes : 27
Seconds : 16
Milliseconds : 657

Almost 30 minutes.

[Mike Resseler]

Tom,

Thanks for doing this! We really appreciate it. It learns us a lot

Brgds,

Mike

[tomhkr]

Send a PM if you need additional tests done, I'm not at the forum daily.

[foggy]

The fix for this and several other issues reported for Veeam Backup for Office 365 beta 1 is available here.