Wasabi immutability - Proof of concept question

[18436572]

We recently upgraded to v12, so far so good. We want to start using Wasabi. I am performing a POC with a free 30 day Wasabi account. I am pretty pleased so far. I see that object storage is working because I've run many active full backups (and incrementals). These add up to over 100GB but the storage is less than 50GB. I also tested restores, they work as expected. Great.

As a POC I want to test immutability:
- From the Veeam console I cannot delete any backups. Perfect.
- From the Wasabi console I was able to delete random pieces of the bucket. LOL. The whole bucket was then removed from the Wasabi console. Uh oh.

After the above, I ran a new test backup and it worked with no issues! In Veeam B&R, under Backups, Object Storage I see the entire chain with fulls and increments, some with monthly and some with weekly retention. All is good in immutability land. I also ran a successful restore test.

So it seems immutability works. But now I cannot see the bucket in Wasabi. I assume I need to contact Wasabi and that this bucket must be recovered by them?

[18436572]

After around an hour, the bucket re-appeared in-tact. So I ran a test backup which worked. Then from the Wasabi console I tried to deleted the whole bucket and was denied. I'm calling it a win! Thanks for reading.

[jeremyrogers]

So... "cloud lag" in that second part of your testing made it appear that you'd successfully deleted the bucket via WC at first, but then after a polling interval had passed it was back?

Those kinds of things drive me crazy when dealing w/ various vendors webUIs. Feels like pulling a Neil Peart drumstick toss while using them.

[AlexHeylin]

But now I cannot see the bucket in Wasabi. I assume I need to contact Wasabi and that this bucket must be recovered by them?

As you've found there's often lag between different parts of systems, especially in cloud systems.
There's a whole thing about different types of consistency in distributed systems - Google "Strong vs. Eventual Consistency" and read up on what it means and the tradeoffs if you're having trouble sleeping one night...
You've also found that one part might "lie" by - for example - saying it deleted objects / buckets, which it could not have deleted due to immutability.
Wasabi uses the eventual consistency model - so eventually all the parts agree (become consistent) on what the state of the system / data is.

TL;DR - wait a bit and it'll sort itself out

[sfirmes]

Wasabi states they use stong consistency which they call "immediate consistency": What data consistency model does Wasabi employ?

The key fact being that when you make an api call to Wasabi and immediately issue another api call, the results will be the same.