Veeam & Object Storage: Real World Review

[LEWISF]

Hi -

We're looking at possibly using Wasabi for offsite storage. We're comparing Azure, AWS, and Wasabi. I would be interested in any real world issues anyone can share with these 3 services? I had an Veeam support person who expressed that Wasabi does have some issues with Veeam and I believe Veeam prefers to use S3.

Any info would be appreciated.

Thanks.

Lewis

[micoolpaul]

Good Morning Lewis,

I hope you’re well. Thought I’d weigh in here as someone that has used them all across different customers. The comment you make regarding Wasabi, yes I have seen this in the past, but with very specific circumstance, tiny objects and attempting to delete basically all of it at once stressed out their API. This was a long time ago and I know from conversations with Wasabi they’ve put a lot of effort into ensuring that Wasabi plays nice with Veeam, and I haven’t seen this issue since. On the comment of S3, Wasabi is “S3 compatible” and it does it really well!

Onto the wider topic at hand:
All of these solutions will scale extremely well for Veeam backups, but you will hit a point where performance degrades across all of them. That’s why in v12 you can add multiple object storage buckets to a SOBR tier. But depending on how much data you need to backup this might not be a problem, I’ve not seen official numbers on “limits” beyond each vendor’s own “bucket recommendations” for generic use, each vendor mentioned I’ve seen data spanning between hundreds of TBs towards the PBs mark with the customer’s WAN a connection still being the bottleneck for performance.


I would therefore look at this from the optics of your DR strategy, where do you intend to restore these backups to if you need them?

If it’s an offsite copy, what happens if the primary site disappears, burns down for example? Wasabi holds the advantage with pricing in no small part due to no egress fees and no API charges, but it does hold a disadvantage for recovery vs AWS & Azure because Wasabi doesn’t have any compute options available. If your DR strategy is to run VMs in the cloud, then you could leverage AWS S3 with AWS EC2 or Azure storage with Azure VMs and perform your recovery in-cloud, something that Wasabi doesn’t offer. You wouldn’t have any bandwidth egress in such a scenario, and similarly if you’re heavily into a particular cloud and have a Microsoft ExpressRoute or AWS DirectConnect, you might have lesser egress fees or an unmetered connection to mitigate those costs assuming there was a site to restore to.

If your plan is to restore massive volumes of data from object storage, on the one hand Wasabi makes sense because of the lack of API calls and egress, but on the drawback, will your connectivity to Wasabi be able to pull the data back in a timely manner, some features such as instant recovery can mitigate this by running the VM “from” Wasabi as the source to reduce RTO, but if you’ve got a large amount of data to read, performance will inevitably suffer and this might not be suitable.

Appreciate the outcome of this is “it depends” but yeah, all the services you mentioned are great, and I recommend them to my enterprise customers daily, but which one is best I would focus on which one makes the recovery process the smoothest and quickest

[Gostev]

Neither AWS S3, Wasabi or Azure would be a mistake. These are top 3 vendors by object storage consumption with tens of thousands Veeam deployments behind each one, so all are quite proven.

Wasabi tends to be the choice of smaller customers due to the price so for sure they can seem "noisier" to our Support Engineer or even on this very forum. But that's only because they reach their comparable consumption with many more of much smaller customers.

[pirx]

Hi -

We're looking at possibly using Wasabi for offsite storage. We're comparing Azure, AWS, and Wasabi. I would be interested in any real world issues anyone can share with these 3 services? I had an Veeam support person who expressed that Wasabi does have some issues with Veeam and I believe Veeam prefers to use S3.

Any info would be appreciated.

Thanks.

Lewis

We just switched from AWS to Wasabi and it's a mess. Not only offloads are taking much longer and failing, even backup jobs take >24h. The backup itself is done pretty quick to performance tier but then the retention/cleanup takes ages. TCP connection limit on Wasabi side was already removed. I'm currently spending hours in calls and on cases with Veeam support to clean up the mess. The latest changes to offloading in Veeam also added new issues. I think we have now the 3rd hotfix for latest versions with more to come. Even reducing concurrent tasks to 8 did not solve the issues. The opposite, we learned the hard way that offloading breaks in cases where too many tasks are fighting for slots (result: endless loop).

Wasabi promised that the situation will change as they have changed something in the background. We will see.

But it's also Veeam offloading that is still kind of experimental. We use this now for 4 years and I still spent a lot time reporting issues/bugs.

[LEWISF]

Thank you for your information and sharing your experiences with these vendors. It is greatly appreciated.

Just a note, I have had issues with our current S3 provider, some of which are similar to issues I have read in this thread. Our performance tier backups work very well now that I've cleaned up a lot of things, but we have an on-going issue with our capacity tier using S3 where objects don't delete on during the offload process or when I submit jobs for deletion where we no longer want restore/date outside of our organization. This has been going on for over a year and I thought I was the only person, but I see that is not the case.

Since we're looking at possibly changing vendors, and we have a presence in Azure and AWS, the information that all of you shared has been very helpful. If you know anyone else with a perspective, please feel free to share this post with them.

Thanks!!

[pirx]

Wasabi solved most of our issues in FRA region, so performance is much better now.

[LEWISF]

Has anyone done any performance testing for backup comparing backup/restore times for Wasabi vs. Azure vs. AWS or do you have any links that can be shared?

[harbinger]

If you want to use health checks for large backups with Wasabi, it's better to use it in a SOBR capacity tier instead of direct Backup to S3 or Backup Copy to S3.
S3 repositories in the capacity tier are able to use the "health check lite" whereas direct backup to S3 is running the full health check, which will be performed by your Wasabi gateway server because at the moment it isn't possible to use computing resources in Wasabi to execute these health checks. This means the gateway server is downloading the full backup and executing the health checks, which impacts the bandwidth of your connection (throttling limits do not apply for health checks, it takes what it can get) and also your IP will be throttled by Wasabi due to all the API requests from these health checks. I don't know if or when Veeam will deliver a workaround for this issue.

[cptkommin]

Just bumping in here, what about MinioS3, or looking at other options looked ootbi or zadara?

[cgsm]

We are small, with under a dozen TB of backup data. We originally used Azure but then moved to Wasabi. Cost was the primary reason (we cut our storage costs by half). We never had a performance issue (backup or restore) from Azure. Likewise, performance to Wasabi has been good. I would recommend either service.

We are using Wasabi as a "backup copy" location, so sadly we cannot use Health Check, as has been noted, this is a full check and requires downloading ALL the VM data to perform the health check....we are hoping Veeam releases the "lite" Health Check for non-SOBR S3-compatible repos some day.

[Gostev]

@veremin I was under impression that we covered all the bases:
- In V12 health check lite was added for regular object storage repositories
- In V12.1 health check lite was extended to SOBR

Are there some scenarios left where we lite mode is still not available?

[CaliMSP]

I've moved several clients to Wasabi, as mentioned they are on the smaller side compared to large enterprises. I have not seen any issues with performance or reliability. Everything runs well, provided you have enough bandwidth to push the data within the desired off-load window.
Wasabi is super simple and quick to deploy, yes it doesn't have compute option behind it, but the price is something you can't ignore either. AWS and others are often just out of reach price wise to smaller organizations.

[tgx]

Backblaze works well.

[veremin]

@veremin I was under impression that we covered all the bases:
- In V12 health check lite was added for regular object storage repositories
- In V12.1 health check lite was extended to SOBR

Are there some scenarios left where we lite mode is still not available?

Currently, the lite health check functionality is available for Capacity Tier and agent backup jobs that are directed to standalone object storage repositories. Enabling lite health check for other backup jobs that target standalone object storage is considered an enhancement for future product versions.

Thanks!

[mcz]

Has anyone done any performance testing for backup comparing backup/restore times for Wasabi vs. Azure vs. AWS or do you have any links that can be shared?

I don't have the comparison you're asking for but a wasabi-guy told me some years ago that you can restore with 1 Gbit/s. So if you're deploying a proxy on e.g. azure/aws, you can write the data back with that 1 Gbit/s. Depending on the size of your vm this might be enough to restore in an appropriate time.

When I did restores or better said copies from the object storage to the SOBR for further local restores, the internet-link is always the bottleneck, so I can't complain about wasabi's performance. But as always, it depends on the environment you have...

[MCH_helferlein]

Using Backblaze since some years.
Cheaper as AWS/Azure and it works well.

[18436572]

We've been on Wasabi for about 1 year. Currently Wasabi has 75.3 TB of our data. Initial fulls of "large" servers (say 3-6TB) did take over 24 hours. Now increments take maybe 20 min. The best value for concurrent tasks seems to be 6 for us. Wasabi is our main target for most jobs. We have a 1GB internet connection. I did wait until v12 was out before implementing Wasabi. Now I am on VBR v12.1.1.56. I did contact Wasabi and told them we are using the immutable bucket for Veeam only. They adjusted the delete time for this need. The only issues really are the health checks, mentioned above. I've put a bandwidth restrictions to any web destination, this seems to help. I've scheduled large VM health checks to run Saturdays at 3PM and they are mostly done by Monday at 8AM. This is a minor pain point. We intend to put our VBR server in Azure this year so that moves the network traffic from onprem-Wasabi to Azure-Wasabi. No MS Express routes are available in our area. I would like the health check lite whatever that is... As long as it doesn't compromise the health check and our backup data. That said I use Backblaze for my home network.

[mcz]

It was very cool if veeam could give us a little appliance which you'd then deploy "somewhere really close to wasabi" or any other S3 provider (maybe even in the same datacenter) to be able to do health checks... I know, you could argue that you already have it, but there the synchronization is a little bit an issue since there are (luckily) no metadata local anymore and also it probably was a manual task. Happy to talk about that feature request. Thanks!