Maintain control of your Microsoft 365 data
Post Reply
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Select mailboxes takes long time

Post by tomhkr » 1 person likes this post

I have about 60,000 mailboxes in my Offce 365 tenant. Trying out the Veeam Office 365 backup, it stucks with selecting mailboxes to backup. "Loading mailboxes.." stays on for very long time (30+ minutes) and doesn't display any boxes.

How long should I wait?

Process "Veeam.Archiver.Service.exe" generates incoming network traffic with about 150-300 kByte/s but stop occasionally.
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr »

Just noticed the process has allocated all RAM and swapping heavily. Need a beefier server for this.. it has 8 GB RAM.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Select mailboxes takes long time

Post by Mike Resseler » 1 person likes this post

Hi,

This is interesting. So you are trying to select some mailboxes but it can't enumerate the list of mailboxes. Did the process finally succeeded or never ended?
In the meantime I will try to get more information on the RAM and the swapping

Thanks
Mike
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr »

To clarify, these are the steps taken:
  • Add backup job
  • At "Select mailboxes to backup" i choose "Backup the following mailboxes:" and press "Select.."
  • The "Select mailboxes" box displays "Loading mailboxes...." forever. The process consumes all memory there is and nothing happened for about an hour, so I killed it.
I'm giving it a new try at a machine with 32 GB of RAM instead, and will get back in this thread with the results.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Select mailboxes takes long time

Post by Dima P. » 1 person likes this post

tomhkr,

Please let us know how it goes. We might need get access to the application debug logs in case of a failure.
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr »

So it finally succeeded in loading all the mailboxes with a beefier server.
The problem was, I think, that with 60.000 mailboxes the first server ran out of memory and began swapping. A swapping server seldomly does any good.

Next problem I ran into is the "An entry with the same key already exists" which I see there is a thread on already and fix coming.

It is still slow, though. Loading all the boxes takes 15-30 MINUTES. This on a server with 32 GB RAM and a 10 GbE internet connection.

There really ought to be some sort of filtering option when selecting mailboxes or creating jobs dynamically based on filter criterias.

This could be accomplished with one of the following:
  • Mailbox membership (possibly problematic?; mailbox can be member of multiple groups)
  • Retention policy
  • Custom attributes
Look, it became a feature request! ;)
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Select mailboxes takes long time

Post by Dima P. »

tomhkr,

Thanks - good catch! Is it possible to share the logs anyway? I bet DEV team would like to investigate your case.
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr »

Certainly. Where can I send them?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Select mailboxes takes long time

Post by Dima P. »

If you have a dropbox or googledrive account (any other cloud hosting works too), just send me the link via PM. You can also upload the logs to the google drive folder I've sent you via PM.
Kostya
Veeam Software
Posts: 104
Liked: 28 times
Joined: Jun 18, 2012 9:38 am
Full Name: Kostya Yasyuk
Contact:

Re: Select mailboxes takes long time

Post by Kostya »

Confirmed. Will be improved in the GA build.
Kostya
Veeam Software
Posts: 104
Liked: 28 times
Joined: Jun 18, 2012 9:38 am
Full Name: Kostya Yasyuk
Contact:

Re: Select mailboxes takes long time

Post by Kostya »

tomhkr wrote:It is still slow, though. Loading all the boxes takes 15-30 MINUTES. This on a server with 32 GB RAM and a 10 GbE internet connection.
Tom, sent you a link via PM to the experimental performance improvements and fixes.
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Select mailboxes takes long time

Post by JaySt »

Tom, just curious, could you also share some findings when you're able to run a actual back-up? Performance for example?
interesting to see what the software is able to do with that volume of mailboxes.
Veeam Certified Engineer
jchillman
Service Provider
Posts: 6
Liked: never
Joined: Jan 13, 2016 8:49 pm
Full Name: James Chillman
Contact:

Re: Select mailboxes takes long time

Post by jchillman »

I would also be interested in the performance data. We have a couple of Large O365 Domains that we will be looking to backup.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Select mailboxes takes long time

Post by Mike Resseler »

Hi James, without being able to give you performance data for now... When you say large... What # are you talking about?
jchillman
Service Provider
Posts: 6
Liked: never
Joined: Jan 13, 2016 8:49 pm
Full Name: James Chillman
Contact:

Re: Select mailboxes takes long time

Post by jchillman »

Hi Mike,

We have one around the 25k Seat mark and a couple around the 5-6k mark.
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr » 1 person likes this post

I have applied the experimental fix and tried creating a new backup job.
It is significantly faster, it now took "only" 11 minutes to arrive to mailbox selection.

It would still be a very welcomed feature to be able to select mailboxes with some criteria, like aforementioned group or list membership, custom attributes et.c. , to search for and browser through tens of thousands of mailboxes to find 15 of them is cumbersome.

Regarding performance on running backup job, the read rate from Office 365 appears to be about 1.2-1.5 Mbyte/s
This is quite low, considering large organizations. But I beliieve this is due to rate limiting at Microsoft, cannot blame Veeam for the low transfer rate. I did a few mass migrations to Office 365 and they all exhibited the same speeds.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Select mailboxes takes long time

Post by foggy »

tomhkr wrote:Regarding performance on running backup job, the read rate from Office 365 appears to be about 1.2-1.5 Mbyte/s
With the purpose of getting some stats, what is the total number of mailboxes and what was the entire job duration/repository size?
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr » 1 person likes this post

I did a test backup with only 2 mailboxes from the IT department since I know some of them contains some meat.

Here are the stats:

First full backup:
xxxx.onmicrosoft.com; 2016-10-19 15:55; 19 minutes, 26 seconds; Items: 37866 (1,59 GB); Processing rate: 32 items/s (1,40 MB/s)

Incremental 8 hours later:
xxxx.onmicrosoft.com; 2016-10-20 00:01; 1 minute, 40 seconds; Items: 16 (642,66 KB); Processing rate: 0 items/s (6,39 KB/s)

Please let me know if I can run more test jobs.

Btw just for kicks I did
(Get-Mailbox -ResultSize Unlimited).Count
59279

:shock:
I did not measure it accurately but I believe it took about 10-15 minutes. So the issue with listing mailboxes at large tenants is probably a limit with Office 365. This would make it even more interesting to get the ability to filter on properties.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Select mailboxes takes long time

Post by Mike Resseler »

Hi Tom,

Interesting what you did. So just using PS fron your workstation (I assume?) you did a simple count to verify how much mailboxes there are in your environment and it took that long to get the results? Did I understand that correctly?

Mike
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr »

Yes that's correct.
It takes huge amounts of time, cpu and memory to list all mailboxes in a large tenant like this.
The local Powershell-process bloats to about 7 GB used RAM while fetching the boxes.
I ran it again, this time measuring properly.

Measure-Command {(Get-Mailbox -ResultSize Unlimited).Count}
Days : 0
Hours : 0
Minutes : 27
Seconds : 16
Milliseconds : 657

Almost 30 minutes.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Select mailboxes takes long time

Post by Mike Resseler »

Tom,

Thanks for doing this! We really appreciate it. It learns us a lot

Brgds,

Mike
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Select mailboxes takes long time

Post by tomhkr » 1 person likes this post

Send a PM if you need additional tests done, I'm not at the forum daily.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Select mailboxes takes long time

Post by foggy »

The fix for this and several other issues reported for Veeam Backup for Office 365 beta 1 is available here.
Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests