restore speed from Wasabi S3

[babzik25]

What are your restore speeds from S3?

I did a restore test from wasabi S3 and v10. I got restore rate at 15 MB/s. I have 1Gb internet and nvme drives, so no bottlenecks here. Files are encrypted. Block size is Local Target. Am I getting a normal restore rate here or should it be higher?

[Gostev]

Hello!

Kindly use search before creating new topics, especially when you have basic questions. I mean, you can never be the first to wonder about performance, ever this applies to any IT forum in existence!

You can find some results for Wasabi in this topic for example. And two posts down in the same topic, for Amazon S3.

But of course, because we're talking public cloud, your mileage may vary even depending on the Moon's phase! OK, at least depending on the time of the day - for sure!

Thanks!

[babzik25]

I did search and did not find any. The post you mentioned is a year old and was done before v10, not same conditions. New feature to copy all data to S3 is awesome. And what I am trying to figure out is how fast I can restore a VM from S3 bucket if all local data is lost. I have internet download speed 1Gb/s. Wasabi speed test or a single large file download speed is close to 1Gb/s as well. But restore rate was a lot less than I expected, 15MB/s. If download and write speeds are not a bottleneck, then what? Encryption? How many blocks does it download at the same time? Is there something I can tweak?

[Gostev]

v10 uses the same object storage integration engine as before, so conditions should be largely the same. Encryption can never be a bottleneck at data rates we're talking about with public cloud, and there's nothing to tweak as we already use massive concurrency (as is the only way to get a meaningful throughput with object storage).

What Wasabi data center are you using? I think the easiest would be for us to try the same test in our own QC lab.

[babzik25]

It is s3.us-east-2.wasabisys.com.

[Gostev]

I can confirm we too are seeing a few times slower performance with s3.us-east-2.wasabisys.com, comparing to s3.eu-central-1.wasabisys.com or with Amazon S3 (the latter two are about the same). Thanks!

[babzik25]

Thank you for testing this.

[CofIAlan]

FWIW, we're seeing oddly slow restores out of Wasabi's US-EAST-2 as well (here in June).
@babzik25 did you ever see a speed increase or has it remained consistently slow these last few months? Our offloads run fast but this test restore is surprising. It would be a real challenge if we actually had to download all of our backups from the cloud in a disaster.

[veremin]

We've decided to involve a Wasabi representative, as they might have a better understanding of environment-specific cases. Thanks!

[CofIAlan]

Cool, thank you. I was actually going to open a support inquiry depending on what I heard back here (if anything). On a test run I have going now I'm getting about 8 Mbps download, which translates to about 15 MB/s in actual Veeam processing throughput (restore rate). We've got plenty of bandwidth to get more than 8 Mbps at the network level and it doesn't appear that any resources on this test host are strained and causing a bottleneck.

[babzik25]

FWIW, we're seeing oddly slow restores out of Wasabi's US-EAST-2 as well (here in June).
@babzik25 did you ever see a speed increase or has it remained consistently slow these last few months? Our offloads run fast but this test restore is surprising. It would be a real challenge if we actually had to download all of our backups from the cloud in a disaster.

No. Speed test - full speeds up and down, upload speed - great, all the way up, restore speed - crawling. Nothing changed after talking to wasabi support. At the moment I gave up on wasabi and s3 all together. Too many problems to deal with.

[vbelagodu]

Hi Babzik25,

My name is Vilas Belagodu, I am part of the technology onboarding team at Wasabi Technologies. Could you provide more info about the restore action you are trying, Are you downloading a backup or are you restoring a VM in a new location while the backup data is sourced on Wasabi? If we peel the onion on that, the steps involved for both operations are different. Also, did you perform speedtest against wasabi customized speedtest server? Wasabi support should have provided you with a password for that. I can work with you on getting to the bottom of the issue and request appropriate Veeam assistance as we need to resolve the issue. Let me know if you would like to connect via a conf call and then troubleshoot the performance issue live. If yes, please updated the Wasabi ticket and add vilas@wasabi.com to the support email - i will take over the ticket and coordinate with you thru our support channel.

Regards,
-Vilas

[babzik25]

Hi @vbelagodu.

I used customized wasabi speed test server. I was getting a full gigabit down on that speed test.

I tried both restoring a file from moved backup chain (no local blocks) and worst case scenario tried to restore a full vm. Restore speed was ~15MB/s on same gigabit connection. Server used was us-east-2. I don't know if this is related to wasabi itself of the way s3 is handled by veeam, but this is too slow to rely on for worst case scenario. Just restoring a few files from time to time should be fine.

Other problems I mentioned are related to veeam, not wasabi. My last straw with s3 was when I did a simple backups evacuation from extent to a new one and it broke capacity tier. Nothing could upload any more to capacity tier. I assume index got broken, and this was not the only time it happened. First time similar happened it took several weeks and escalations to fix. At the end I still lost several restore points. And don't get me wrong here, I like veeam tech support, but techs answering the phone can't fix these problems, escalating takes time and I can't have clients without offsite backups for long time.

Thank you for offering to help, I may come back to using wasabi once veeam improves it's s3 handling.

[veremin]

Can you provide us with the support ticket numbers for the issues mentioned - we would like to double check them? I recall you had an issue with NFS extent, but support team were quite responsive on that ticket and the case was supervised by one of product managers. Thanks!

[babzik25]

case #04110902
recent one #04189036. I did not follow this one through to the end.

[babzik25]

I recall you had an issue with NFS extent, but support team were quite responsive on that ticket and the case was supervised by one of product managers.

Yes, they were. I have no complaints about support quality. Everyone was great! They provided registry entry that fixed issue. But I ran into same issue again on other clients and registry entry did not help that time around. I did not create tickets for those, I moved storage back to SMB. I try to follow Gostev's recommendations and wanted to use NFS instead of SMB. SMB + cloud connect has been rock stable for years. NFS + S3 not so much.

[veremin]

OK, we will take a look at the provided ticket next week. Will write to you back, if anything interesting pops up. Thanks!