S3 error: The specified version does not exist (Wasabi)

[deduplicat3d]

I think I'm having a related issue. Veeam v11. SOBR with a Capacity Tier using Wasabi. Immutability is enabled (from the beginning) following the Wasabi guide (versioning and object locking enabled). In Veeam set to immutable for 30 days. I set this up about 2 weeks ago, my jobs are finally almost done copying to the Capacity Tier, and then out of the 9 jobs that are being offloading, I am getting this same error (as above) on just 2 jobs.
10/7/2021 9:08:26 PM :: Processing <jobname> Error: REST API error: 'S3 error: The specified version does not exist. Code: NoSuchVersion', error code: 404
Other: Key: 'Veeam/Archive/veeam/<fullpath>.00000000000000000000000000000000.blk', VersionId: '<versionid>', HostId: '<hostid>'

Should I contact support? It's strange that the object wouldn't exist since it hasn't even been the 30 days.

[deduplicat3d]

Thanks. I submitted a support request to Wasabi. If they can't figure it out, I probably will evaluate B2.

Would be interested to hear if Veeam is familiar with this on Wasabi specifically.

Not looking forward to having to push all of the data again.

[Gostev]

I've split your posts into the new topic. That other topic you picked was specific to Backblaze due to some confirmed bugs on their side, which are now fixed.

No, I don't remember seeing such an issue reported before for Wasabi.

[deduplicat3d]

Thanks. I opened tickets for Veeam and Wasabi

[deduplicat3d]

FYI Wasabi support admitted that they "lost" the data. Working with Veeam support to fix the issue. It's pretty concerning that this can happen.

""""Thanks for your patience while we completed the investigation for the two objects. I coordinated with our operations team to find out complete details on this matter. During the short period of time within minutes [2021-10-07 1:56:14 UTC - 2021-10-07 1:57:25 UTC], few buckets were being migrated from the old database structure to the new database during the upgrade. We successfully migrated the entire bucket to the new architecture database at 2021-10-07 1:57:25 UTC, while this was taking place for your bucket the below objects attempted to get uploaded at 2021-10-07 01:56:38 UTC & 2021-10-07 01:56:40 UTC respectively. The two objects we are discussing here were basically stuck in flight but never really made all the way to the destination DB to be created on our storage system within that very seconds.

We apologize for the inconvenience caused due to this. At this point, we would like to request you to re-upload the same Veeam backup job again to the primary repository and offload it to the cloud tier.""""

[Gostev]

The reality is, there's no data management vendor out there which never had data loss bugs, or can magically ensure they won't have more of such bugs in future. This is why 3-2-1 rule exists in the first place, specifically the "two different medias" part (most commonly ignored these days).

But wow! Having worked with some other cloud object storage vendors on similar suspected data loss issues, I must admit Wasabi has pretty impressive logging out of the box. Instant troubleshooting vs. "We enabled something on the backend so now you go and try to reproduce the issue again, but let us know ASAP when you do because these extended logs are rotated very quickly".

[deduplicat3d]

I agree that nothing is 100%. It's just worrisome since I started just a week ago, and have only uploaded 25TB so far. Maybe just my bad luck

The other complication seems to be that there isn't a simple way to rescan and reupload missing objects. Veeam support is working on this. I'm assuming it's because cloud object failures are very uncommon.

[deduplicat3d]

PS given that this is my capacity tier, I wasn't planning on following the 3-2-1 for my GFS (monthlies and annual). Maybe in select circumstances for annual backups to archive. Not sure what best practices for 3-2-1 and GFS is

[Gostev]

Customers who need two copies of GFS backups (usually for compliance) keep one copy of GFS backups in regular on-prem backup repositories, and another copy either on tape or in object storage.

[gtelnet]

PS given that this is my capacity tier, I wasn't planning on following the 3-2-1 for my GFS (monthlies and annual). Maybe in select circumstances for annual backups to archive. Not sure what best practices for 3-2-1 and GFS is

We had the same plan originally but learned that if you want to ensure you can restore data from a GFS backup when requested, make sure you have two copies on two different sets of media. If you don't have 2 copies, you might as well have none.