Really slow backup speeds

[jbcap]

Hi,

I have an old support case (04018118, closed in March) that kind of went nowhere, I've mostly been ignoring this problem but I can't any longer.

Back then we were doing about 500 mailboxes, speed ~10 items/s & 1MB/s processing rate, 20GB transferred @ 5 hours.

We are now backing up Onedrive & Sharepoint (very minimal use) and these rates are now worse, ~3 items/s & 500KB/s, 20GB transferred @ 10+ hours.

My understanding of the answer given by support is that our disk i/o is the bottleneck, they highlighted that during backup time the disk highest active time is around 90% even tho disk transfer speeds are really low.

I was asked to run diskspd with the following options: diskspd.exe -t16 -r -b4K -d600 -L -o1 -w80 -Sh E:\testfile.dat, results: 360.09 MiB/s total, 72.01/288.09 MiB/s Read/Write.

This is on a physical server, specs:
Xeon E5-2609 v4 @ 1.7GHz
Windows Server 2019 1809
Using tiered storage spaces with 4x 860 EVO 4TB with 4x Seagate IronWolf NAS 8TB.
We have a gigabit fibre link

I know server specs aren't the greatest but Backup & Recovery runs fine on this box, there's got to be something wrong here to cause these slow speeds.

[Mike Resseler]

Hi Jim,

Are you using S2D?

[jbcap]

Hi Mike,

It is single host, so technically only Storage Spaces, not sure if that makes a difference?

[Mike Resseler]

We have seen another customer with really bad experience on storage spaces. We did a support call with Microsoft to try to figure out what is happening, but they couldn't figure out the issue. It seems that a Jet DB on storage spaces is not a good idea and it causes a big bottleneck. The specific customer moved away (or better, is moving away) from storage spaces to on-premises object storage and is seeing completely different results.

RE: storage spaces / S2D... If it is S2D with performance and capacity tier, then it certainly won't work as a database is a very bad thing to run on such an implementation. That's why I asked to be sure

[jbcap]

Regarding storage spaces & databases in general being a no no, do you have any references for this?

Every test metric shows storage performance far exceeding what we're seeing in VBO, I haven't seen a good explanation for that so far & hope that one exists.

[Mike Resseler]

Hi Jim,

Unfortunately no. That is where we worked together with MSFT to figure out what is going on. Our database is a jet DB, the same type as used in Exchange. They also couldn't figure out why the performance was so bad. We do know a few more things now obviously, but still not everything we would like. What version are you on at this moment?

[jbcap]

4.0.0.1345

[jbcap]

Is that it then? Bad luck, I can't use Storage spaces with VBO365?

[popjls]

I also have really poor backup speeds from Office 365 so youre not alone, it's not network(gig internet), it's not disk (All flash). I'm about to test update 5...

[Mike Resseler]

Hi @popjls What version are you using at the moment?

[popjls]

Hi Mike,

I'm now using the latest update (4.0.1.531) and it's maybe marginally faster but still pretty slow imo. Might be where we are pulling the data from to our location but either way, not a huge deal as we don't have thousands of people to backup.

[Mike Resseler]

@popjls Sorry to keep asking questions

Is it only slow for SharePoint/ OneDrive? Or also for Exchange?
Are you using multiple service accounts to backup SharePoint?

[popjls]

All good

OneDrive is the quickest (with the most retries), Sharepoint is excruciatingly slow (again not a huge deal), email is ok. Only a single account. Not sure what everyone else is seeing but we see anything from 1mbit to 150mbit at best.

[jbcap]

Can I confirm that Service accounts only help for Sharepoint & Onedrive speeds?

[Polina]

Hi Jim,

Correct, additional service accounts are only used for SPO/ODFB

[jbcap]

So... what can I do here?

We are using storage spaces because it is ideal for B&R.
VBO365 apparently doesn't like storage spaces for some unknown reason.
I don't have resources to separate these products on their own hardware.

We seem stuck, can we get this performance issue looked into further?

[Mike Resseler]

Jim,

Can you use a VM with a disk attached to it?

[jbcap]

So I set up minio to test with, I can't see how to use object storage only though? Looks like it can only extend existing storage.

[Polina]

Hi Jim,

That's correct, extending a local repository with object storage is the way to go. Backups will go to directly your minio, and the metadata will be stored locally in a Jet database.
Check out the documentation to learn more.