Slow restore performance

[dasfliege]

We performed some restore testing for a customers audit and were quite surprise about how slow they are.

Our infrastructure consists of the following components:
Server: 16 Core, 64GB Memory
Storage: Synology RS3617xs+ with 12 Disks
Network: 2x10Gbit from Server to Storage
Internet: 500Mbit up/down

We did the following two tests:

Restore a complete Sharepoint site to Sharepoint online:
- 63GB were restored in 8 hours
- Awerage restore rate of 2.2MB/s or 17Mbit/s

Restore a bunch of mailboxes as .PST to another local storage:
- 200GB were restored in 16hours
- Awerage restore rate of 3.5MB/s or 28Mbit/s


As these values are way below of what our infrastructure is capable of, i would like to find out what the bottleneck could be. Are there any people who already performed such tests and have some reference values?

[Mildur]

Hello

Could you maybe be open a support case and upload the debug logs?
That would help to analyze the bottleneck.
Please provide the case number with us for future reference.

How is the Synology connected to the VB365 server? iSCSI or SMB 3.0 (experimental support)?

Thanks
Fabian

[dasfliege]

Case 05696422

The Syno is connected via iSCSI MPIO. Here's a diskspd log on the source filesystem with 64k blocksize. Don't know how representative this test is, as i don't know what blocksize VBM365 is operating with:

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 6043795456 | 92221 | 96.06 | 1537.02 | 0.532 | 0.570 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
1 | 6086721536 | 92876 | 96.75 | 1547.93 | 0.530 | 0.536 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
2 | 6081085440 | 92790 | 96.66 | 1546.50 | 0.532 | 0.499 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
3 | 6080888832 | 92787 | 96.65 | 1546.45 | 0.532 | 0.528 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
-----------------------------------------------------------------------------------------------------
total: 24292491264 | 370674 | 386.12 | 6177.90 | 0.532 | 0.534

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 6079250432 | 92762 | 96.63 | 1546.03 | 0.762 | 6.547 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
1 | 6105137152 | 93157 | 97.04 | 1552.62 | 0.757 | 5.817 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
2 | 6081806336 | 92801 | 96.67 | 1546.68 | 0.758 | 5.871 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
3 | 6082527232 | 92812 | 96.68 | 1546.87 | 0.758 | 5.864 | C:\MountPoints\xxxx\ioB64K.dat (2048MiB)
-----------------------------------------------------------------------------------------------------
total: 24348721152 | 371532 | 387.01 | 6192.20 | 0.759 | 6.032

[Polina]

I'm afraid that the speed of SharePoint restore was heavily affected by throttling.
At the same time, I'd expect to see faster times of the export to PST. Thanks for providing your support case ID, I'll follow up on the details.

Thanks!

[dasfliege]

We're still struggling finding the root cause for this issue. Case number is 05696422

Last test i did was:
- Created another Repository on local SSD Disks
- Performed a fresh backup of my own mailbox to the new repository for testing purpose
- Performed a restore of my backed up mailbox to PST

It took roughly 30min to export 16GB of Data to the PST, which corresponds to a restore rate of 8.8MB/s.
The disks where the new repository is located is an enterprise SSD. It is capable to deliver several thousand IOPS and a few hundred MB/s throughput. So only 8.8MB/s restore rate still seems way to less.

Does anyone has any values to compare? I wonder if there is anyone who is able to restore to PST faster or if it just is a limitation of veeam oder the explorer or jetDB...

[dasfliege]

I catch up on this with just a general question to other service providers:
Don't know if we are the only ones who have customers asking us how long it would take to perform a full restore of their M365 organization. In switzerland most of the companys do have audits where they have to proof that they have a working business continuity concept and part of that is the information, how long it would take to be up and running in a desaster scenario.
The lack of responses here make me think that eigther we are the only ones having such slow restore rates, or others just don't care about or have not tested it?

Regarding our testing, we are at a point where it seems we nailed down the slowlyness of local PST export to a limitation in the Outlook Engine. Direct restores to M365 are slow anyway because of their throttling. If some could confirm that they are in the same boat, i may feel better to tell our customers that there is no way to speed up thing in case of a desaster (MS explodes ) and that an offsite backup like VBM365 should primarly been used for selectiv restores of the most critical items and not for restores of a whole organization, as it just would take too long.