Wasabi vs Backblaze B2 vs iDrive E2

[hrai]

I have been using Azure cool storage as our Capacity Tier for several months, however I wanted to reduce costs since our data keeps growing and API transaction costs represented almost 50% of our bill. First, I tried changing all our backup jobs to use Large Blocks (4MB) to reduce the API calls (which were very high using the default block size of 1MB) which helped reducing our costs by 25%, but increased the incrementals size considerably (using more precious space of our Performance Tier storage).

This prompted me into looking at other object storage providers that offered cheaper storage and no API (PUT) transaction costs. There are many object storage providers out there but I could not find any comparisons on these forums. So I decided to write this post for other Veeam customers out there doing the same research, and hopefully it helps them in their decision. I started using and testing 3 object storage providers in the past month: Wasabi, Backblaze B2 and iDrive E2 and here are my findings.


Storage cost:
iDrive: $4.09/TB/month ($40/TB/year; $4/TB/year for first year Promo)
Backblaze: $5.12/TB/month
Wasabi: $5.99/TB/month (Discounts over 50TB on 1, 3 or 5 years upfront)

Egress cost and/or free limit:
iDrive: Free up to 3X the total storage in a month
Wasabi: Free up to 1X the total storage in a month
Backblaze: $10.24/TB

API Transactions cost:
iDrive & Wasabi: Free
Backblaze: $0.004 to $0.040 per 10,000 with 2,500 free per day

Early deletion policy:
iDrive and Backblaze: None
Wasabi: Minimum 30 days for Veeam customers plan (90 days for regular customers)

Storage Regions:
Wasabi: 12
iDrive: 9
Backblaze: 2

Immutability:
iDrive and Wasabi: Compatible and objects cannot be deleted on their website.
Wasabi: Compatible but objects could be deleted on their website. Maybe I need to set compliance in their UI to match Veeam immutability settings but seems cumbersome.

Upload speeds (averaged out of 3 tests per region using 30GB vbk and large block):
Azure (Canada - Toronto): 105 MB/s
Wasabi (USA - Virginia): 102 MB/s
Wasabi (Canada - Toronto): 98 MB/s
iDrive (USA - Virginia): 74 MB/s
iDrive (USA - Chicago): 62 MB/s
Backblaze (USA - West): 60 MB/s * could be Sacramento or Phoenix 21 MB/s latest
Wasabi (USA - Oregon): 56 MB/s
iDrive (USA - Oregon): 23 MB/s

Upload speed to closest region (to Toronto using 1Gbps fiber connection):
Wasabi: 102 MB/s
iDrive: 74 MB/s
Backblaze: 60 MB/s

Search matches in Veeam Object Storage forums (as of 28-Jun-2022):
Wasabi: 478
Backblaze: 233
iDrive: 0

UI Ease of use:
iDrive and Wasabi: Snappy and friendly
Backblaze: Slow to show bucket contents when it contains large amounts of blocks. Not that friendly.

Final thoughts:

• iDrive: It's without a doubt the cheapest of all 3 options. The upload speeds (second place) and lack of data center in Canada is a minus for me. The immutability worked out of the box and their UI is possibly the best one (Wasabi is very close in this area). However the fact that they did not showed up in any of the Veeam forums is a concern and they are also the smallest company out of the 3 in terms of revenue and number of employees so their existence in this space is the questionable.

• Backblaze: It's the slowest of the 3 options and their high egress fees are a big issue for me. The lack of data centers is also a concern. They are a well established company but since they went public their stock value has been dropping a lot. Their founders are selling stocks for the next year so the future of this company is uncertain and makes me nervous picking them up for my company data.

• Wasabi: They are the most expensive for storage fees and also the only ones that impose a minimum retention period of 30 days (to Veeam customers, 90 days to others). They are the newest company but they are growing fast and the only dedicated object storage company out of the 3 (the other two do PC backup sand other stuff). They are the most renowned in Veeam forums and my only concern is that I could delete objects in their website that I marked as immutable in Veeam. They do have a compliance setting which is extra to Object Lock so maybe they both need to be set up to the same timeframe.

Conclusion:
For now I have decided to go with Wasabi for 3 reasons: upload speed, data center in Toronto (where my company is) and usage by Veeam customers as it can be seen in the forums.

I would love to hear feedback from other Veeam customers...

PS: The only place on the internet I found with a comparison of these 3 companies was at BOBcloud website here: https://www.bobcloud.net/reviews/micros ... reloaded=1

[chrisWasabi]

Hi hrai, thanks for your consideration! At the time of writing this, we actually have 13 regions and are continuing our region-per-month rollout for 2022! I believe Bobs site has some older data on it. As far as the immutability goes, we have direct integration with Veeam Immutability and are Veeam vReady Object with Immutability certified. It is not possible to delete the data if it is locked by Veeam through the website, however, it may look like you can, due to the Versioning required on a bucket for object lock s3 compatibility. If you click delete on an object, a DeleteMarker will be placed on the object so it is not shown in the console, you can turn on the toggle to see object versions, and your object will still be there. If you attempt to delete it, you will get an error.

We have an integration guide for Veeam here: https://wasabi-support.zendesk.com/hc/e ... ntegration

If you need anything, don't hesitate to reach out!

Cheers

[Andreas Neufert]

Thanks for sharing.
Chris is correct on how deletes are handeled on all object storage vendors that support object lock immutability. You can go and delete an immutable object over the API, the UI or elsewere it will only create an additional version of the object with a delete marker.

[raghava]

Hi harai,

Thanks for considering and trying out IDrive e2.!

Yes, we are new to object storage, but we are not new to cloud backup.

IDrive is an established cloud backup service provider in the consumer/SMB/MSPs market and we have been in the business more than two decades. IDrive currently serves millions of users for their backup needs and stores over 1000 PB of data across the globe.

Regarding e2 region in Canada; right now we are in the process of expanding our regions to Canada, EU, Asia and Australia.

If you need anything in the future, please don't hesitate to reach out!

[hrai]

It's great to hear that iDrive will be expanding to Canada in the future. I really like your offering and I had a tough choice picking up a provider, if you had a data center in Canada and was listed in Veeam's "Unofficial object storage compatibility list for Veeam Backup & Replication" then I would have picked iDrive.

[hrai]

I cannot edit my original post but today I made several tests and I am sad to report Wasabi immutability is questionable. Anyways this is only related to my findings, which I made some videos to document my issues. But in summary I decided to go with iDrive, since their immutability works out of the box. So I change my original pick and I will take slower upload speeds but to me immutability is key... Going with iDrive for now...

[Andreas Neufert]

Why do you think that Wasabi Immutability is questionable? Is it because they offer an option in the UI to delete objects while iDrive do not have this option. Then there is no real difference here. See my statement above. Please share the video with me or document what test you have done so that we can analyse what you did.

[hrai]

Hi Andreas:

Unless I am doing something wrong I think I discovered a bug in Wasabi and is not only in their web UI but also occurs when using S3 Browser.
Here are the steps I took:

1. Create new bucket in Wasabi website (select Bucket Versioning and Enable Object Locking)
2. Create new Object Storage Repository in Veeam (S3 Compatible, in the Bucket section Make recent backups immutable for 1 day)
3. Create a new SOBR using the new Wasabi Object Storage repository
4. Create a new Backup Job (in the Storage section use the new SOBR as the backup repository, set retention policy at 2 days)
5. Run the backup job and wait until the local backup is created and then offloaded to the Wasabi bucket
6. Go to Wasabi website and drill down the bucket and go to the objs subfolder (there will be 4 files in there)
7. Delete one of the files and look it disappear.
8. Change the toggle button "Show Versions" to ON and then you can see the deleted file (with a type icon of version).
9. Delete a second file and all is good to this point.
10. Delete a third file and this is where things get weird. After deleting the 3rd file the entire subfolder objs gets deleted.
11. I tried doing the same in the checkpoints subfolder and since there are only 2 files there as soon as I delete 1 file the entire subfolder gets deleted too.

If I did something wrong let me know. But I tried these exact same steps with Backblaze and iDrive and I could not delete anything in there.

[dalbertson]

Are you sure they are being deleted or are they having a delete marker being placed on them? You can try to remove the delete markers and it should return.

https://wasabi-support.zendesk.com/hc/e ... -a-bucket-

[chrisWasabi]

Hi Hrai,

I was unable to duplicate where only removing some of the files caused the removal of the whole path, if you place a DeleteMarker on all the objects inside a path, that path will disappear from the GUI, as all the objects were "deleted". But rest assured, that is the way versioning works. If you follow the guide that Dustin posted, the DeleteMarkers can be removed, and your path and objects will be back, or you can use something like S3 Browsers to view DeletedVersions selecting the Versions tab, and the filter for DeletedVersions on the bottom right.

https://imgur.com/a/1JBS16Y

Either way, we are happy to look into this further with you if you open a support ticket by email support@wasabi.com

Cheers!

[hrai]

Hi daltbertson:

I checked again and even though the Wasabi web file browser does not show anything inside the bucket, the objects are actually there if I look at versioning in S3 Browser. I tried restoring the backups from that bucket and Veeam could not see the objects either. So even though the objects are still there Veeam will not be able to see them until the DELETE MARKER is removed from all objects.

So to clarify the bug is with Wasabi not deleting objects but setting the DELETE MARKERS in all objects within a subfolder one there is one left.

[dalbertson]

This is expected behavior due to how object lock / versioning works. When you try to delete an object locked item it creates whats called a delete marker and that marker becomes the new version. The object is removed from the console and view of veeam because of the delete marker. This happens on other cloud providers as well like AWS. The items are not deleted until object lock expires. The link I provided before is a way to remove the delete markers and restore the original object so that it appears back in the console and back to veeam.

Again this is not a bug of wasabi or veeam. This is how delete markers on S3 object storage work.

[Andreas Neufert]

This is unrelated to Veeam, just how the S3 versioning standard works.

[hrai]

Hi dalbertson:

So before trying to restore from Veeam I would need to remove all the DELETE MARKERS from all the objects? I just deleted 4 files so only 4 files should have DELETE MARKERS not 43,592, correct? Is this normal or is it a bug?

[dalbertson]

Correct, you would only need to remove the delete markers from the 4 files. The script in the link just makes the process much easier.

[matt_778]

Cool writeup - another one to possibly try is Cloudflare E2
https://blog.cloudflare.com/introducing ... t-storage/

[OrangeWing]

Thanks for the great thread. I am a small shop and not using cloud backup although I want to. We have been using Veeam for 6 years though.
How easy is it to set up and since we are in US, is the interface at Wasabi or iDrive easy to use/understand?
Thanks!!!

[BradM2]

Another provider to consider is OVHCloud. Currently the only place they offer S3 is in France, but they have no ingress fees of any kind, only egress. I have a Performance Tier in their Quebec location and the Capacity Tier in their France location. Their Object Lock functionality is in beta so I haven't tested that functionality yet. I've been testing the Capacity Tier with BnR and it seems to work well, and if you're not in the US, your data doesn't go to the US, so you're not subject to the US Cloud Act.

[hrai]

UPDATE: After getting errors in my backup jobs 2 days in a row last week using iDrive I decided to switch back to Wasabi. Now that I understand how the DELETE MARKERS work with immutability in Wasabi I feel confident my backups are safe there. My backup jobs have been working smoothly in the past 3 days with Wasabi. These were the errors I got with iDrive during SOBR Offload:

REST API error: 'Service Temporarily Unavailable', error code: 503
REST API error: 'S3 error: Server down for maintenance Code: Service Unavailable', error code: 503
HTTP exception: WinHttpReceiveResponse: 12002: The operation timed out, error code: 12002

[dloseke]

I did find a bug with the Wasabi UI when editing policies and support was quick to respond to gather more info and replicate the issue. Their support has been excellent to work with as well. The only other vendor I've really considered was BackBlaze B2 but the egress fees were what got me. As a MSP, the one thing I don't like is that with a standard account, I have to create a bucket with each tenant and setup some strange policies so that it doesn't list all buckets for my clients....if I stepped up to using their WACM (I believe that's the correct acronym), I can setup a sub account for each tenant and then I don't have those same limitations, and I can also then give clients a login to access their own sub and buckets. I also can't place limits on how much data can be stored at the bucket level but can at the sub-account level. But that's an extra $100 per month that is harder to absorb for a smaller MSP, and there's not migration path when I last spoke to them about this from a standard account to the WACM accounts. As an end user, they're great....as partner to a MSP, I think they're getting there.....

[raghava]

Hi hrai, There were some resource limit that got hit which we resolved.

You may try now.

[hrai]

Hi raghava:

I actually cancelled our iDrive account. I had pre-paid 10TB for a full year taking advantage of the great first year offer, but as per iDrive policies cancelling within 15 days and I could get a refund (which I did btw). I think I will let iDrive E2 mature a little before giving it another try. I know iDrive has been in the Cloud biz a long time but the E2 offering barely started in April this year. Once Veeam adds it to the "Unofficial object storage compatibility list for Veeam Backup & Replication" and you have a data centre in Canada, I might reconsider. For now is a risk I am not willing to take. Beside the fact I could not Offload SOBR for 2 days I also did 2 tests recovering VMs from iDrive and the download speed is much lower than the actual upload speed. I did not tested download speed in my original set of tests.

[hrai]

I did find a bug with the Wasabi UI when editing policies and support was quick to respond to gather more info and replicate the issue. Their support has been excellent to work with as well. The only other vendor I've really considered was BackBlaze B2 but the egress fees were what got me. As a MSP, the one thing I don't like is that with a standard account, I have to create a bucket with each tenant and setup some strange policies so that it doesn't list all buckets for my clients....if I stepped up to using their WACM (I believe that's the correct acronym), I can setup a sub account for each tenant and then I don't have those same limitations, and I can also then give clients a login to access their own sub and buckets. I also can't place limits on how much data can be stored at the bucket level but can at the sub-account level. But that's an extra $100 per month that is harder to absorb for a smaller MSP, and there's not migration path when I last spoke to them about this from a standard account to the WACM accounts. As an end user, they're great....as partner to a MSP, I think they're getting there.....

Thank you for the feedback from a MSP point of view

[raghava]

Hi hrai,

FYI.. we just added Montreal (Canada) region to IDrive e2, and it is also added to "Unofficial object storage compatibility list for Veeam Backup & Replication"

You may give it a try.

Thanks

[hrai]

That is great news raghava! I will totally look into this for archiving purposes.

[TonioRoffo]

Hi dalbertson:

So before trying to restore from Veeam I would need to remove all the DELETE MARKERS from all the objects? I just deleted 4 files so only 4 files should have DELETE MARKERS not 43,592, correct? Is this normal or is it a bug?

Is this 100% safe?

I'm thinking of a scenario where a potential insider tried to delete objects -> they get "deleted" with a marker. The insider does a few more backups, syncing to S3 storage and calls it a day. There's now new objects in the S3 storage - if I remove the delete markers at this moment, is there a possibility the backups are bad (because of new objects replacing the ones that were marked as deleted?) or am I panicking here?

Thanks.

[dalbertson]

This is safe. We track the versions of objects that we place, objects are unique. So when someone tries to delete them it just places these markers on the objects as they are locked till the lock date. even if more backups are run, they would be NEW objects and versions. So removing delete markers does not impact the "new" objects....only the ones that have the markers on them to delete.

[jim.lowry]

Echoing @dalbertson - when the blocks are marked for the delete process, they will not be deleted until after the immutability time for the object has been met. Keep in mind though, that as soon as they are no longer immutable, when the next delete cycle is run it will delete and purge the objects without notifying you. This is precisely because they are marked for deletion. So make sure that your object retention and immutability settings/RPO match up as best you can. I highly suggest testing the scenario in a lab environment.

When v12 comes out, we will also have some options added with object and tape arrays for immutable airgapping if you are concerned about object storage delete flags and immutability retention.

[jvl]

Cool writeup - another one to possibly try is Cloudflare E2
https://blog.cloudflare.com/introducing ... t-storage/

Did you test it? it seems to be GA now https://blog.cloudflare.com/r2-ga/ they are not charging egress fees at all

[raghava]

Hello everyone, Just wanted to keep the community current: we recently released a major performance update to IDrive e2 which will boost the performance of all S3 calls.

And we have expanded more regions in the EU and the next expansion is going to be the Asia Pacific and Australia regions. Will keep you all posted.!

Thanks