Discussions related to using object storage as a backup target.
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai » 12 people like this post

I have been using Azure cool storage as our Capacity Tier for several months, however I wanted to reduce costs since our data keeps growing and API transaction costs represented almost 50% of our bill. First, I tried changing all our backup jobs to use Large Blocks (4MB) to reduce the API calls (which were very high using the default block size of 1MB) which helped reducing our costs by 25%, but increased the incrementals size considerably (using more precious space of our Performance Tier storage).

This prompted me into looking at other object storage providers that offered cheaper storage and no API (PUT) transaction costs. There are many object storage providers out there but I could not find any comparisons on these forums. So I decided to write this post for other Veeam customers out there doing the same research, and hopefully it helps them in their decision. I started using and testing 3 object storage providers in the past month: Wasabi, Backblaze B2 and iDrive E2 and here are my findings.


Storage cost:
iDrive: $4.09/TB/month ($40/TB/year; $4/TB/year for first year Promo)
Backblaze: $5.12/TB/month
Wasabi: $5.99/TB/month (Discounts over 50TB on 1, 3 or 5 years upfront)

Egress cost and/or free limit:
iDrive: Free up to 3X the total storage in a month
Wasabi: Free up to 1X the total storage in a month
Backblaze: $10.24/TB

API Transactions cost:
iDrive & Wasabi: Free
Backblaze: $0.004 to $0.040 per 10,000 with 2,500 free per day

Early deletion policy:
iDrive and Backblaze: None
Wasabi: Minimum 30 days for Veeam customers plan (90 days for regular customers)

Storage Regions:
Wasabi: 12
iDrive: 9
Backblaze: 2

Immutability:
iDrive and Wasabi: Compatible and objects cannot be deleted on their website.
Wasabi: Compatible but objects could be deleted on their website. Maybe I need to set compliance in their UI to match Veeam immutability settings but seems cumbersome.

Upload speeds (averaged out of 3 tests per region using 30GB vbk and large block):
Azure (Canada - Toronto): 105 MB/s
Wasabi (USA - Virginia): 102 MB/s
Wasabi (Canada - Toronto): 98 MB/s
iDrive (USA - Virginia): 74 MB/s
iDrive (USA - Chicago): 62 MB/s
Backblaze (USA - West): 60 MB/s * could be Sacramento or Phoenix 21 MB/s latest
Wasabi (USA - Oregon): 56 MB/s
iDrive (USA - Oregon): 23 MB/s

Upload speed to closest region (to Toronto using 1Gbps fiber connection):
Wasabi: 102 MB/s
iDrive: 74 MB/s
Backblaze: 60 MB/s

Search matches in Veeam Object Storage forums (as of 28-Jun-2022):
Wasabi: 478
Backblaze: 233
iDrive: 0

UI Ease of use:
iDrive and Wasabi: Snappy and friendly
Backblaze: Slow to show bucket contents when it contains large amounts of blocks. Not that friendly.

Final thoughts:
  • iDrive: It's without a doubt the cheapest of all 3 options. The upload speeds (second place) and lack of data center in Canada is a minus for me. The immutability worked out of the box and their UI is possibly the best one (Wasabi is very close in this area). However the fact that they did not showed up in any of the Veeam forums is a concern and they are also the smallest company out of the 3 in terms of revenue and number of employees so their existence in this space is the questionable.
  • Backblaze: It's the slowest of the 3 options and their high egress fees are a big issue for me. The lack of data centers is also a concern. They are a well established company but since they went public their stock value has been dropping a lot. Their founders are selling stocks for the next year so the future of this company is uncertain and makes me nervous picking them up for my company data.
  • Wasabi: They are the most expensive for storage fees and also the only ones that impose a minimum retention period of 30 days (to Veeam customers, 90 days to others). They are the newest company but they are growing fast and the only dedicated object storage company out of the 3 (the other two do PC backup sand other stuff). They are the most renowned in Veeam forums and my only concern is that I could delete objects in their website that I marked as immutable in Veeam. They do have a compliance setting which is extra to Object Lock so maybe they both need to be set up to the same timeframe.
Conclusion:
For now I have decided to go with Wasabi for 3 reasons: upload speed, data center in Toronto (where my company is) and usage by Veeam customers as it can be seen in the forums.

I would love to hear feedback from other Veeam customers...

PS: The only place on the internet I found with a comparison of these 3 companies was at BOBcloud website here: https://www.bobcloud.net/reviews/micros ... reloaded=1
chrisWasabi
Technology Partner
Posts: 19
Liked: 33 times
Joined: Feb 23, 2021 3:42 pm
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by chrisWasabi » 2 people like this post

Hi hrai, thanks for your consideration! At the time of writing this, we actually have 13 regions and are continuing our region-per-month rollout for 2022! I believe Bobs site has some older data on it. As far as the immutability goes, we have direct integration with Veeam Immutability and are Veeam vReady Object with Immutability certified. It is not possible to delete the data if it is locked by Veeam through the website, however, it may look like you can, due to the Versioning required on a bucket for object lock s3 compatibility. If you click delete on an object, a DeleteMarker will be placed on the object so it is not shown in the console, you can turn on the toggle to see object versions, and your object will still be there. If you attempt to delete it, you will get an error.

We have an integration guide for Veeam here: https://wasabi-support.zendesk.com/hc/e ... ntegration

If you need anything, don't hesitate to reach out!

Cheers
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by Andreas Neufert » 1 person likes this post

Thanks for sharing.
Chris is correct on how deletes are handeled on all object storage vendors that support object lock immutability. You can go and delete an immutable object over the API, the UI or elsewere it will only create an additional version of the object with a delete marker.
raghava
Novice
Posts: 5
Liked: 4 times
Joined: Jun 29, 2022 8:47 pm
Full Name: Raghavendra Meeniga
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by raghava » 2 people like this post

Hi harai,

Thanks for considering and trying out IDrive e2.!

Yes, we are new to object storage, but we are not new to cloud backup.

IDrive is an established cloud backup service provider in the consumer/SMB/MSPs market and we have been in the business more than two decades. IDrive currently serves millions of users for their backup needs and stores over 1000 PB of data across the globe.

Regarding e2 region in Canada; right now we are in the process of expanding our regions to Canada, EU, Asia and Australia.

If you need anything in the future, please don't hesitate to reach out!
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

It's great to hear that iDrive will be expanding to Canada in the future. I really like your offering and I had a tough choice picking up a provider, if you had a data center in Canada and was listed in Veeam's "Unofficial object storage compatibility list for Veeam Backup & Replication" then I would have picked iDrive.
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

I cannot edit my original post but today I made several tests and I am sad to report Wasabi immutability is questionable. Anyways this is only related to my findings, which I made some videos to document my issues. But in summary I decided to go with iDrive, since their immutability works out of the box. So I change my original pick and I will take slower upload speeds but to me immutability is key... Going with iDrive for now...
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by Andreas Neufert » 1 person likes this post

Why do you think that Wasabi Immutability is questionable? Is it because they offer an option in the UI to delete objects while iDrive do not have this option. Then there is no real difference here. See my statement above. Please share the video with me or document what test you have done so that we can analyse what you did.
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

Hi Andreas:

Unless I am doing something wrong I think I discovered a bug in Wasabi and is not only in their web UI but also occurs when using S3 Browser.
Here are the steps I took:

1. Create new bucket in Wasabi website (select Bucket Versioning and Enable Object Locking)
2. Create new Object Storage Repository in Veeam (S3 Compatible, in the Bucket section Make recent backups immutable for 1 day)
3. Create a new SOBR using the new Wasabi Object Storage repository
4. Create a new Backup Job (in the Storage section use the new SOBR as the backup repository, set retention policy at 2 days)
5. Run the backup job and wait until the local backup is created and then offloaded to the Wasabi bucket
6. Go to Wasabi website and drill down the bucket and go to the objs subfolder (there will be 4 files in there)
7. Delete one of the files and look it disappear.
8. Change the toggle button "Show Versions" to ON and then you can see the deleted file (with a type icon of version).
9. Delete a second file and all is good to this point.
10. Delete a third file and this is where things get weird. After deleting the 3rd file the entire subfolder objs gets deleted.
11. I tried doing the same in the checkpoints subfolder and since there are only 2 files there as soon as I delete 1 file the entire subfolder gets deleted too.

If I did something wrong let me know. But I tried these exact same steps with Backblaze and iDrive and I could not delete anything in there.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by dalbertson »

Are you sure they are being deleted or are they having a delete marker being placed on them? You can try to remove the delete markers and it should return.

https://wasabi-support.zendesk.com/hc/e ... -a-bucket-
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
chrisWasabi
Technology Partner
Posts: 19
Liked: 33 times
Joined: Feb 23, 2021 3:42 pm
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by chrisWasabi » 1 person likes this post

Hi Hrai,

I was unable to duplicate where only removing some of the files caused the removal of the whole path, if you place a DeleteMarker on all the objects inside a path, that path will disappear from the GUI, as all the objects were "deleted". But rest assured, that is the way versioning works. If you follow the guide that Dustin posted, the DeleteMarkers can be removed, and your path and objects will be back, or you can use something like S3 Browsers to view DeletedVersions selecting the Versions tab, and the filter for DeletedVersions on the bottom right.

https://imgur.com/a/1JBS16Y

Either way, we are happy to look into this further with you if you open a support ticket by email support@wasabi.com

Cheers!
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

Hi daltbertson:

I checked again and even though the Wasabi web file browser does not show anything inside the bucket, the objects are actually there if I look at versioning in S3 Browser. I tried restoring the backups from that bucket and Veeam could not see the objects either. So even though the objects are still there Veeam will not be able to see them until the DELETE MARKER is removed from all objects.

So to clarify the bug is with Wasabi not deleting objects but setting the DELETE MARKERS in all objects within a subfolder one there is one left.

Image

Image
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by dalbertson » 3 people like this post

This is expected behavior due to how object lock / versioning works. When you try to delete an object locked item it creates whats called a delete marker and that marker becomes the new version. The object is removed from the console and view of veeam because of the delete marker. This happens on other cloud providers as well like AWS. The items are not deleted until object lock expires. The link I provided before is a way to remove the delete markers and restore the original object so that it appears back in the console and back to veeam.

Again this is not a bug of wasabi or veeam. This is how delete markers on S3 object storage work.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by Andreas Neufert » 1 person likes this post

This is unrelated to Veeam, just how the S3 versioning standard works.
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

Hi dalbertson:

So before trying to restore from Veeam I would need to remove all the DELETE MARKERS from all the objects? I just deleted 4 files so only 4 files should have DELETE MARKERS not 43,592, correct? Is this normal or is it a bug?
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by dalbertson »

Correct, you would only need to remove the delete markers from the 4 files. The script in the link just makes the process much easier.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
matt_778
Enthusiast
Posts: 25
Liked: 2 times
Joined: Feb 08, 2010 9:25 am
Full Name: Matt
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by matt_778 »

Cool writeup - another one to possibly try is Cloudflare E2
https://blog.cloudflare.com/introducing ... t-storage/
OrangeWing
Novice
Posts: 7
Liked: 1 time
Joined: Jan 12, 2015 5:14 pm
Full Name: Stafford Fields
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by OrangeWing »

Thanks for the great thread. I am a small shop and not using cloud backup although I want to. We have been using Veeam for 6 years though.
How easy is it to set up and since we are in US, is the interface at Wasabi or iDrive easy to use/understand?
Thanks!!!
BradM2
Novice
Posts: 3
Liked: 1 time
Joined: Mar 12, 2021 7:51 pm
Full Name: Brad Melrose
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by BradM2 »

Another provider to consider is OVHCloud. Currently the only place they offer S3 is in France, but they have no ingress fees of any kind, only egress. I have a Performance Tier in their Quebec location and the Capacity Tier in their France location. Their Object Lock functionality is in beta so I haven't tested that functionality yet. I've been testing the Capacity Tier with BnR and it seems to work well, and if you're not in the US, your data doesn't go to the US, so you're not subject to the US Cloud Act.
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

UPDATE: After getting errors in my backup jobs 2 days in a row last week using iDrive I decided to switch back to Wasabi. Now that I understand how the DELETE MARKERS work with immutability in Wasabi I feel confident my backups are safe there. My backup jobs have been working smoothly in the past 3 days with Wasabi. These were the errors I got with iDrive during SOBR Offload:

REST API error: 'Service Temporarily Unavailable', error code: 503
REST API error: 'S3 error: Server down for maintenance Code: Service Unavailable', error code: 503
HTTP exception: WinHttpReceiveResponse: 12002: The operation timed out, error code: 12002
dloseke
Service Provider
Posts: 60
Liked: 28 times
Joined: Jul 13, 2018 3:33 pm
Full Name: Derek M. Loseke
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by dloseke » 1 person likes this post

I did find a bug with the Wasabi UI when editing policies and support was quick to respond to gather more info and replicate the issue. Their support has been excellent to work with as well. The only other vendor I've really considered was BackBlaze B2 but the egress fees were what got me. As a MSP, the one thing I don't like is that with a standard account, I have to create a bucket with each tenant and setup some strange policies so that it doesn't list all buckets for my clients....if I stepped up to using their WACM (I believe that's the correct acronym), I can setup a sub account for each tenant and then I don't have those same limitations, and I can also then give clients a login to access their own sub and buckets. I also can't place limits on how much data can be stored at the bucket level but can at the sub-account level. But that's an extra $100 per month that is harder to absorb for a smaller MSP, and there's not migration path when I last spoke to them about this from a standard account to the WACM accounts. As an end user, they're great....as partner to a MSP, I think they're getting there.....
Derek M. Loseke, Senior Systems Engineer | Veeam Legend 2022-2023 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
raghava
Novice
Posts: 5
Liked: 4 times
Joined: Jun 29, 2022 8:47 pm
Full Name: Raghavendra Meeniga
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by raghava »

Hi hrai, There were some resource limit that got hit which we resolved.

You may try now.
Raghava
VP, Technologies
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

Hi raghava:

I actually cancelled our iDrive account. I had pre-paid 10TB for a full year taking advantage of the great first year offer, but as per iDrive policies cancelling within 15 days and I could get a refund (which I did btw). I think I will let iDrive E2 mature a little before giving it another try. I know iDrive has been in the Cloud biz a long time but the E2 offering barely started in April this year. Once Veeam adds it to the "Unofficial object storage compatibility list for Veeam Backup & Replication" and you have a data centre in Canada, I might reconsider. For now is a risk I am not willing to take. Beside the fact I could not Offload SOBR for 2 days I also did 2 tests recovering VMs from iDrive and the download speed is much lower than the actual upload speed. I did not tested download speed in my original set of tests.
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

dloseke wrote: Jul 05, 2022 4:54 pm I did find a bug with the Wasabi UI when editing policies and support was quick to respond to gather more info and replicate the issue. Their support has been excellent to work with as well. The only other vendor I've really considered was BackBlaze B2 but the egress fees were what got me. As a MSP, the one thing I don't like is that with a standard account, I have to create a bucket with each tenant and setup some strange policies so that it doesn't list all buckets for my clients....if I stepped up to using their WACM (I believe that's the correct acronym), I can setup a sub account for each tenant and then I don't have those same limitations, and I can also then give clients a login to access their own sub and buckets. I also can't place limits on how much data can be stored at the bucket level but can at the sub-account level. But that's an extra $100 per month that is harder to absorb for a smaller MSP, and there's not migration path when I last spoke to them about this from a standard account to the WACM accounts. As an end user, they're great....as partner to a MSP, I think they're getting there.....
Thank you for the feedback from a MSP point of view :)
raghava
Novice
Posts: 5
Liked: 4 times
Joined: Jun 29, 2022 8:47 pm
Full Name: Raghavendra Meeniga
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by raghava » 1 person likes this post

Hi hrai,

FYI.. we just added Montreal (Canada) region to IDrive e2, and it is also added to "Unofficial object storage compatibility list for Veeam Backup & Replication"

You may give it a try.

Thanks
Raghava
VP, Technologies
hrai
Influencer
Posts: 16
Liked: 14 times
Joined: Jun 26, 2022 11:43 pm
Full Name: HRAI Admin
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by hrai »

That is great news raghava! I will totally look into this for archiving purposes.
TonioRoffo
Enthusiast
Posts: 51
Liked: 5 times
Joined: Jun 18, 2009 2:27 pm
Full Name: Yves Smolders
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by TonioRoffo »

hrai wrote: Jun 30, 2022 8:13 pm Hi dalbertson:

So before trying to restore from Veeam I would need to remove all the DELETE MARKERS from all the objects? I just deleted 4 files so only 4 files should have DELETE MARKERS not 43,592, correct? Is this normal or is it a bug?
Is this 100% safe?

I'm thinking of a scenario where a potential insider tried to delete objects -> they get "deleted" with a marker. The insider does a few more backups, syncing to S3 storage and calls it a day. There's now new objects in the S3 storage - if I remove the delete markers at this moment, is there a possibility the backups are bad (because of new objects replacing the ones that were marked as deleted?) or am I panicking here?

Thanks.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by dalbertson » 1 person likes this post

This is safe. We track the versions of objects that we place, objects are unique. So when someone tries to delete them it just places these markers on the objects as they are locked till the lock date. even if more backups are run, they would be NEW objects and versions. So removing delete markers does not impact the "new" objects....only the ones that have the markers on them to delete.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
jim.lowry
Veeam Software
Posts: 228
Liked: 62 times
Joined: Jul 12, 2018 4:45 pm
Full Name: Jim Lowry
Location: California
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by jim.lowry »

Echoing @dalbertson - when the blocks are marked for the delete process, they will not be deleted until after the immutability time for the object has been met. Keep in mind though, that as soon as they are no longer immutable, when the next delete cycle is run it will delete and purge the objects without notifying you. This is precisely because they are marked for deletion. So make sure that your object retention and immutability settings/RPO match up as best you can. I highly suggest testing the scenario in a lab environment.

When v12 comes out, we will also have some options added with object and tape arrays for immutable airgapping if you are concerned about object storage delete flags and immutability retention.
Jim Lowry
Sr. Systems Engineer
VCSP North America West
VMCE, VMCA, VCP-DC
jvl
Enthusiast
Posts: 44
Liked: 1 time
Joined: Sep 09, 2014 1:12 pm
Full Name: jvl
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by jvl »

matt_778 wrote: Jul 04, 2022 5:53 am Cool writeup - another one to possibly try is Cloudflare E2
https://blog.cloudflare.com/introducing ... t-storage/
Did you test it? it seems to be GA now https://blog.cloudflare.com/r2-ga/ they are not charging egress fees at all
raghava
Novice
Posts: 5
Liked: 4 times
Joined: Jun 29, 2022 8:47 pm
Full Name: Raghavendra Meeniga
Contact:

Re: Wasabi vs Backblaze B2 vs iDrive E2

Post by raghava » 1 person likes this post

Hello everyone, Just wanted to keep the community current: we recently released a major performance update to IDrive e2 which will boost the performance of all S3 calls.

And we have expanded more regions in the EU and the next expansion is going to be the Asia Pacific and Australia regions. Will keep you all posted.!

Thanks
Raghava
VP, Technologies
Post Reply

Who is online

Users browsing this forum: MarkBoothmaa and 16 guests