Top restore performance?

[JRRW]

What is the expected top restore performance when not using rollback.

For reference, when restoring from an all flash (24x3.84tb SSD Raid-5) backup copy job (so does have dedupe) writing TO a PureStorage C60, I'm maxing at 500MB/s - and it's not bottlenecking anywhere that I can see. Storage wise the source can pull easily at 1.5GB/s and the destination I've seen write at upwards of 4GB/s.

Xeon CPUs have a ton of leg room available on both the repository (16c/32t @ 2.99ghz Silver 4215) and the destination (32c/64t @ 3.27Ghz Gold 6326).

My only theory i can figure out at this point is that it's a single CPU threaded operation on the Destination as I do see a single CPU core/thread at 100% constantly... but the overall host is ranging from 5% to 11% so there's a ton of available resources. I'm basing this purely on having done two restores at the same time and it went to about 1GB/s.

Is there any way to improve a single large disk restore? For reference, this is only a 5TB VHDX which is not even in the top 10 in terms of size of VHDXs I backup... so I'm somewhat nervous about having to restore one of my 30TB-50TB volumes.

Note: Also tested a restore from my all-flash CEPH with nearly identical results.

Thanks in advance!

[HannesK]

Hello,
if you try to write a single stream to your PureStorage... how fast is that? I guess it's around 500MByte/s. At least, that's a common value around many vendors since years.

You should also see 1GByte/s if you restore two disks in parallel.

Best regards,
Hannes

[JRRW]

Interesting. I supposed thinking through it that makes a sort of sense - I was under the impression that Veeam somehow split those restore operations even of a single drive into multiple threads - but if it's a single thread then yes, even an all NVMe array will only have a single thread write speed of around 550-600MB/s. And as Microsoft still hasn't released NVMe-oF or NVMe-TCP we can't see if leveraging an end-to-end NVMe storage path would improve that.

That's disappointing though. In comparison, if I'm doing a single file copy - of a VHDX or any large file - over the network using something like robocopy, I'm able to get a solid 800MB/s-1.2GB/s despite it being a 'single file'. That seems to indicate there's something more at play to the task that is the true bottleneck.

[HannesK]

just to be sure... you are doing full-VM restore, right?

The biggest difference between robocopy and our restore is probably the compression. Do you maybe have a chance to do a backup and restore that and tell us how fast it is?

[JRRW]

Considering it's a 10TB volume I'm looking at, I'd rather not run a new full backup/restore, but running a restore from the same exact file but to the same server it's on (meaning the source repository but on a different RAID) it's closer to 1GB/s. So it has something to do with how your product is handling the translation over the network and/or writing to a FC HBA. As noted above, the source repository is 'slower' when it comes to actual drive types, and has slower/lower end processors for handling any decompression... The only variable then is either the network (which is a non-starter as these are end to end 2xSFP28 connections through a Cisco C9500 and running layer-2 not layer-3) and that the 'slower' is a SAN (which I would assume would be 'optimized' by Veeam given that most Enterprise' are going to have a SAN as a production storage)

If needed I can just open a case - I was hoping there was a simple check mark or functional element that would be a 'known' limitation. We've had our setup reviewed by Veeam as recently as a year ago and there were no configuration errors or settings 'wrong' etc.

[HannesK]

well, any network has some kind of impact. I have many customers with stretched storage clusters between a few hundred meters / a few kilometers distance. Restoring to a synchronously mirrored volume cuts restore speed half vs. unmirrored volume restore. To be fair, that restore test was with VMware, where things work a bit different, but just to clarify that network has an impact.

In general, there is a recommendation to grow disks not too large because of restore speed. One can work with quick rollback (CBT restore) or instant recovery as alternative sometimes, but for plain disk restore, it might be the physical limit.

Feel free to check with support. Please post the support case number, so that I can follow it.