Maintain control of your Microsoft 365 data
Post Reply
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Really slow backup speeds

Post by jbcap »

Hi,

I have an old support case (04018118, closed in March) that kind of went nowhere, I've mostly been ignoring this problem but I can't any longer.

Back then we were doing about 500 mailboxes, speed ~10 items/s & 1MB/s processing rate, 20GB transferred @ 5 hours.

We are now backing up Onedrive & Sharepoint (very minimal use) and these rates are now worse, ~3 items/s & 500KB/s, 20GB transferred @ 10+ hours.

My understanding of the answer given by support is that our disk i/o is the bottleneck, they highlighted that during backup time the disk highest active time is around 90% even tho disk transfer speeds are really low.

I was asked to run diskspd with the following options: diskspd.exe -t16 -r -b4K -d600 -L -o1 -w80 -Sh E:\testfile.dat, results: 360.09 MiB/s total, 72.01/288.09 MiB/s Read/Write.

This is on a physical server, specs:
Xeon E5-2609 v4 @ 1.7GHz
Windows Server 2019 1809
Using tiered storage spaces with 4x 860 EVO 4TB with 4x Seagate IronWolf NAS 8TB.
We have a gigabit fibre link

I know server specs aren't the greatest but Backup & Recovery runs fine on this box, there's got to be something wrong here to cause these slow speeds.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Really slow backup speeds

Post by Mike Resseler »

Hi Jim,

Are you using S2D?
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

Hi Mike,

It is single host, so technically only Storage Spaces, not sure if that makes a difference?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Really slow backup speeds

Post by Mike Resseler »

We have seen another customer with really bad experience on storage spaces. We did a support call with Microsoft to try to figure out what is happening, but they couldn't figure out the issue. It seems that a Jet DB on storage spaces is not a good idea and it causes a big bottleneck. The specific customer moved away (or better, is moving away) from storage spaces to on-premises object storage and is seeing completely different results.

RE: storage spaces / S2D... If it is S2D with performance and capacity tier, then it certainly won't work as a database is a very bad thing to run on such an implementation. That's why I asked to be sure
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

Regarding storage spaces & databases in general being a no no, do you have any references for this?

Every test metric shows storage performance far exceeding what we're seeing in VBO, I haven't seen a good explanation for that so far & hope that one exists.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Really slow backup speeds

Post by Mike Resseler »

Hi Jim,

Unfortunately no. That is where we worked together with MSFT to figure out what is going on. Our database is a jet DB, the same type as used in Exchange. They also couldn't figure out why the performance was so bad. We do know a few more things now obviously, but still not everything we would like. What version are you on at this moment?
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

4.0.0.1345
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

Is that it then? Bad luck, I can't use Storage spaces with VBO365?
popjls
Enthusiast
Posts: 52
Liked: 5 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: Really slow backup speeds

Post by popjls »

I also have really poor backup speeds from Office 365 so youre not alone, it's not network(gig internet), it's not disk (All flash). I'm about to test update 5...
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Really slow backup speeds

Post by Mike Resseler »

Hi @popjls What version are you using at the moment?
popjls
Enthusiast
Posts: 52
Liked: 5 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: Really slow backup speeds

Post by popjls »

Hi Mike,

I'm now using the latest update (4.0.1.531) and it's maybe marginally faster but still pretty slow imo. Might be where we are pulling the data from to our location but either way, not a huge deal as we don't have thousands of people to backup.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Really slow backup speeds

Post by Mike Resseler »

@popjls Sorry to keep asking questions :-)

Is it only slow for SharePoint/ OneDrive? Or also for Exchange?
Are you using multiple service accounts to backup SharePoint?
popjls
Enthusiast
Posts: 52
Liked: 5 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: Really slow backup speeds

Post by popjls »

All good :)

OneDrive is the quickest (with the most retries), Sharepoint is excruciatingly slow (again not a huge deal), email is ok. Only a single account. Not sure what everyone else is seeing but we see anything from 1mbit to 150mbit at best.
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

Can I confirm that Service accounts only help for Sharepoint & Onedrive speeds?
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Really slow backup speeds

Post by Polina »

Hi Jim,

Correct, additional service accounts are only used for SPO/ODFB
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

So... what can I do here?

We are using storage spaces because it is ideal for B&R.
VBO365 apparently doesn't like storage spaces for some unknown reason.
I don't have resources to separate these products on their own hardware.

We seem stuck, can we get this performance issue looked into further?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Really slow backup speeds

Post by Mike Resseler »

Jim,

Can you use a VM with a disk attached to it?
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Really slow backup speeds

Post by jbcap »

So I set up minio to test with, I can't see how to use object storage only though? Looks like it can only extend existing storage.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Really slow backup speeds

Post by Polina »

Hi Jim,

That's correct, extending a local repository with object storage is the way to go. Backups will go to directly your minio, and the metadata will be stored locally in a Jet database.
Check out the documentation to learn more.
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests