-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Really slow backup speeds
Hi,
I have an old support case (04018118, closed in March) that kind of went nowhere, I've mostly been ignoring this problem but I can't any longer.
Back then we were doing about 500 mailboxes, speed ~10 items/s & 1MB/s processing rate, 20GB transferred @ 5 hours.
We are now backing up Onedrive & Sharepoint (very minimal use) and these rates are now worse, ~3 items/s & 500KB/s, 20GB transferred @ 10+ hours.
My understanding of the answer given by support is that our disk i/o is the bottleneck, they highlighted that during backup time the disk highest active time is around 90% even tho disk transfer speeds are really low.
I was asked to run diskspd with the following options: diskspd.exe -t16 -r -b4K -d600 -L -o1 -w80 -Sh E:\testfile.dat, results: 360.09 MiB/s total, 72.01/288.09 MiB/s Read/Write.
This is on a physical server, specs:
Xeon E5-2609 v4 @ 1.7GHz
Windows Server 2019 1809
Using tiered storage spaces with 4x 860 EVO 4TB with 4x Seagate IronWolf NAS 8TB.
We have a gigabit fibre link
I know server specs aren't the greatest but Backup & Recovery runs fine on this box, there's got to be something wrong here to cause these slow speeds.
I have an old support case (04018118, closed in March) that kind of went nowhere, I've mostly been ignoring this problem but I can't any longer.
Back then we were doing about 500 mailboxes, speed ~10 items/s & 1MB/s processing rate, 20GB transferred @ 5 hours.
We are now backing up Onedrive & Sharepoint (very minimal use) and these rates are now worse, ~3 items/s & 500KB/s, 20GB transferred @ 10+ hours.
My understanding of the answer given by support is that our disk i/o is the bottleneck, they highlighted that during backup time the disk highest active time is around 90% even tho disk transfer speeds are really low.
I was asked to run diskspd with the following options: diskspd.exe -t16 -r -b4K -d600 -L -o1 -w80 -Sh E:\testfile.dat, results: 360.09 MiB/s total, 72.01/288.09 MiB/s Read/Write.
This is on a physical server, specs:
Xeon E5-2609 v4 @ 1.7GHz
Windows Server 2019 1809
Using tiered storage spaces with 4x 860 EVO 4TB with 4x Seagate IronWolf NAS 8TB.
We have a gigabit fibre link
I know server specs aren't the greatest but Backup & Recovery runs fine on this box, there's got to be something wrong here to cause these slow speeds.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Really slow backup speeds
Hi Jim,
Are you using S2D?
Are you using S2D?
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
Hi Mike,
It is single host, so technically only Storage Spaces, not sure if that makes a difference?
It is single host, so technically only Storage Spaces, not sure if that makes a difference?
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Really slow backup speeds
We have seen another customer with really bad experience on storage spaces. We did a support call with Microsoft to try to figure out what is happening, but they couldn't figure out the issue. It seems that a Jet DB on storage spaces is not a good idea and it causes a big bottleneck. The specific customer moved away (or better, is moving away) from storage spaces to on-premises object storage and is seeing completely different results.
RE: storage spaces / S2D... If it is S2D with performance and capacity tier, then it certainly won't work as a database is a very bad thing to run on such an implementation. That's why I asked to be sure
RE: storage spaces / S2D... If it is S2D with performance and capacity tier, then it certainly won't work as a database is a very bad thing to run on such an implementation. That's why I asked to be sure
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
Regarding storage spaces & databases in general being a no no, do you have any references for this?
Every test metric shows storage performance far exceeding what we're seeing in VBO, I haven't seen a good explanation for that so far & hope that one exists.
Every test metric shows storage performance far exceeding what we're seeing in VBO, I haven't seen a good explanation for that so far & hope that one exists.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Really slow backup speeds
Hi Jim,
Unfortunately no. That is where we worked together with MSFT to figure out what is going on. Our database is a jet DB, the same type as used in Exchange. They also couldn't figure out why the performance was so bad. We do know a few more things now obviously, but still not everything we would like. What version are you on at this moment?
Unfortunately no. That is where we worked together with MSFT to figure out what is going on. Our database is a jet DB, the same type as used in Exchange. They also couldn't figure out why the performance was so bad. We do know a few more things now obviously, but still not everything we would like. What version are you on at this moment?
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
4.0.0.1345
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
Is that it then? Bad luck, I can't use Storage spaces with VBO365?
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: Really slow backup speeds
I also have really poor backup speeds from Office 365 so youre not alone, it's not network(gig internet), it's not disk (All flash). I'm about to test update 5...
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Really slow backup speeds
Hi @popjls What version are you using at the moment?
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: Really slow backup speeds
Hi Mike,
I'm now using the latest update (4.0.1.531) and it's maybe marginally faster but still pretty slow imo. Might be where we are pulling the data from to our location but either way, not a huge deal as we don't have thousands of people to backup.
I'm now using the latest update (4.0.1.531) and it's maybe marginally faster but still pretty slow imo. Might be where we are pulling the data from to our location but either way, not a huge deal as we don't have thousands of people to backup.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Really slow backup speeds
@popjls Sorry to keep asking questions
Is it only slow for SharePoint/ OneDrive? Or also for Exchange?
Are you using multiple service accounts to backup SharePoint?
Is it only slow for SharePoint/ OneDrive? Or also for Exchange?
Are you using multiple service accounts to backup SharePoint?
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: Really slow backup speeds
All good
OneDrive is the quickest (with the most retries), Sharepoint is excruciatingly slow (again not a huge deal), email is ok. Only a single account. Not sure what everyone else is seeing but we see anything from 1mbit to 150mbit at best.
OneDrive is the quickest (with the most retries), Sharepoint is excruciatingly slow (again not a huge deal), email is ok. Only a single account. Not sure what everyone else is seeing but we see anything from 1mbit to 150mbit at best.
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
Can I confirm that Service accounts only help for Sharepoint & Onedrive speeds?
-
- Veeam Software
- Posts: 3195
- Liked: 774 times
- Joined: Oct 21, 2011 11:22 am
- Full Name: Polina Vasileva
- Contact:
Re: Really slow backup speeds
Hi Jim,
Correct, additional service accounts are only used for SPO/ODFB
Correct, additional service accounts are only used for SPO/ODFB
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
So... what can I do here?
We are using storage spaces because it is ideal for B&R.
VBO365 apparently doesn't like storage spaces for some unknown reason.
I don't have resources to separate these products on their own hardware.
We seem stuck, can we get this performance issue looked into further?
We are using storage spaces because it is ideal for B&R.
VBO365 apparently doesn't like storage spaces for some unknown reason.
I don't have resources to separate these products on their own hardware.
We seem stuck, can we get this performance issue looked into further?
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Really slow backup speeds
Jim,
Can you use a VM with a disk attached to it?
Can you use a VM with a disk attached to it?
-
- Influencer
- Posts: 18
- Liked: never
- Joined: May 24, 2018 4:12 am
- Full Name: Jim Bell
- Contact:
Re: Really slow backup speeds
So I set up minio to test with, I can't see how to use object storage only though? Looks like it can only extend existing storage.
-
- Veeam Software
- Posts: 3195
- Liked: 774 times
- Joined: Oct 21, 2011 11:22 am
- Full Name: Polina Vasileva
- Contact:
Re: Really slow backup speeds
Hi Jim,
That's correct, extending a local repository with object storage is the way to go. Backups will go to directly your minio, and the metadata will be stored locally in a Jet database.
Check out the documentation to learn more.
That's correct, extending a local repository with object storage is the way to go. Backups will go to directly your minio, and the metadata will be stored locally in a Jet database.
Check out the documentation to learn more.
Who is online
Users browsing this forum: No registered users and 4 guests