Discussions related to using object storage as a backup target.
Post Reply
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

S3 error: The specified version does not exist (Wasabi)

Post by deduplicat3d »

I think I'm having a related issue. Veeam v11. SOBR with a Capacity Tier using Wasabi. Immutability is enabled (from the beginning) following the Wasabi guide (versioning and object locking enabled). In Veeam set to immutable for 30 days. I set this up about 2 weeks ago, my jobs are finally almost done copying to the Capacity Tier, and then out of the 9 jobs that are being offloading, I am getting this same error (as above) on just 2 jobs.
10/7/2021 9:08:26 PM :: Processing <jobname> Error: REST API error: 'S3 error: The specified version does not exist. Code: NoSuchVersion', error code: 404
Other: Key: 'Veeam/Archive/veeam/<fullpath>.00000000000000000000000000000000.blk', VersionId: '<versionid>', HostId: '<hostid>'

Should I contact support? It's strange that the object wouldn't exist since it hasn't even been the 30 days.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: persistent offload error to capacity tier (backblaze)

Post by deduplicat3d »

Thanks. I submitted a support request to Wasabi. If they can't figure it out, I probably will evaluate B2.

Would be interested to hear if Veeam is familiar with this on Wasabi specifically.

Not looking forward to having to push all of the data again.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by Gostev »

I've split your posts into the new topic. That other topic you picked was specific to Backblaze due to some confirmed bugs on their side, which are now fixed.

No, I don't remember seeing such an issue reported before for Wasabi.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by deduplicat3d »

Thanks. I opened tickets for Veeam and Wasabi
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by deduplicat3d »

FYI Wasabi support admitted that they "lost" the data. Working with Veeam support to fix the issue. It's pretty concerning that this can happen.

""""Thanks for your patience while we completed the investigation for the two objects. I coordinated with our operations team to find out complete details on this matter. During the short period of time within minutes [2021-10-07 1:56:14 UTC - 2021-10-07 1:57:25 UTC], few buckets were being migrated from the old database structure to the new database during the upgrade. We successfully migrated the entire bucket to the new architecture database at 2021-10-07 1:57:25 UTC, while this was taking place for your bucket the below objects attempted to get uploaded at 2021-10-07 01:56:38 UTC & 2021-10-07 01:56:40 UTC respectively. The two objects we are discussing here were basically stuck in flight but never really made all the way to the destination DB to be created on our storage system within that very seconds.

We apologize for the inconvenience caused due to this. At this point, we would like to request you to re-upload the same Veeam backup job again to the primary repository and offload it to the cloud tier.""""
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by Gostev »

The reality is, there's no data management vendor out there which never had data loss bugs, or can magically ensure they won't have more of such bugs in future. This is why 3-2-1 rule exists in the first place, specifically the "two different medias" part (most commonly ignored these days).

But wow! Having worked with some other cloud object storage vendors on similar suspected data loss issues, I must admit Wasabi has pretty impressive logging out of the box. Instant troubleshooting vs. "We enabled something on the backend so now you go and try to reproduce the issue again, but let us know ASAP when you do because these extended logs are rotated very quickly".
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by deduplicat3d »

I agree that nothing is 100%. It's just worrisome since I started just a week ago, and have only uploaded 25TB so far. Maybe just my bad luck

The other complication seems to be that there isn't a simple way to rescan and reupload missing objects. Veeam support is working on this. I'm assuming it's because cloud object failures are very uncommon.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by deduplicat3d »

PS given that this is my capacity tier, I wasn't planning on following the 3-2-1 for my GFS (monthlies and annual). Maybe in select circumstances for annual backups to archive. Not sure what best practices for 3-2-1 and GFS is
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by Gostev »

Customers who need two copies of GFS backups (usually for compliance) keep one copy of GFS backups in regular on-prem backup repositories, and another copy either on tape or in object storage.
gtelnet
Service Provider
Posts: 40
Liked: 16 times
Joined: Mar 28, 2020 3:50 pm
Full Name: GregT
Contact:

Re: S3 error: The specified version does not exist (Wasabi)

Post by gtelnet » 1 person likes this post

deduplicat3d wrote: Oct 14, 2021 2:31 am PS given that this is my capacity tier, I wasn't planning on following the 3-2-1 for my GFS (monthlies and annual). Maybe in select circumstances for annual backups to archive. Not sure what best practices for 3-2-1 and GFS is
We had the same plan originally but learned that if you want to ensure you can restore data from a GFS backup when requested, make sure you have two copies on two different sets of media. If you don't have 2 copies, you might as well have none.
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests