Discussions related to using object storage as a backup target.
Post Reply
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by pirx »

Case: 05112419 / 05079334

After our migration from v10 to v11a in September we had major problems, mainly regarding offloading. Now, more than 2 months later we still have VMs that are not offloaded - so no immutability, which is the main reason for us to use it. Some were offloaded a few times after the initial problems were fixed, but none of them were offloaded in the past 5 weeks.

We see errors like:

Code: Select all

Error: REST API error: 'Not Found', error code: 404
Error: REST API error: 'S3 error: The specified version does not exist. Code: NoSuchVersion', error code: 404 Other: Key: ....
We had to delete ArchiveIndex and rescan SOBR a lot of times after migration to get offload running again at all. So rescan was done successfully multiple times already.

Offloading seems to be very error prone, even in 11a. My problem is, that there seems to be no workaround and this case was already forwarded to R&D some weeks ago. Last thing I heard is, that they are busy with new release... Well, I like new releases too, but fixing bugs for customer that don't get their VMs offloaded would also be a nice thing. I've now escalated the case.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by Gostev »

They could have only meant first cumulative patch for 11a at this point, not some new release. Moreover, this patch specifically focuses on a few common object storage integration bugs, although I'm afraid the above is not a known issue with multiple support cases open.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by veremin »

I checked the cases internally: one seems to have been put on hold since the day the support team shared the SQL query with you (the one that presents information regarding offloaded restore points), the notes of second ticket suggest that QA team was planning to review the ticket this week, and according to the QA engineer I've just spoken with, he will try to analyze the case later today.

Thanks!
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by pirx »

Thx for the feedback, let's see how long it takes to fix it.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by veremin »

QA team have requested additional information from you, since some logs were originally missing.

I've been added to the internal discussion regarding your case, so I will follow the investigation closely and share the news here, as soon as I got some.

Thanks!
rljunggren
Lurker
Posts: 1
Liked: never
Joined: Apr 14, 2023 2:05 pm
Full Name: Rob
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by rljunggren »

Has anyone found a fix for this issue yet? I have this on one of my backups.
Mattias
Influencer
Posts: 14
Liked: 3 times
Joined: Jan 21, 2015 1:18 pm
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by Mattias »

Hi !

After going to V12 and starting to do direct backup to S3 i now get this error on two of my jobs. Anyone have any update on this before i make a support case to veeam?
galbitz
Enthusiast
Posts: 42
Liked: 5 times
Joined: May 17, 2018 2:22 pm
Full Name: grant albitz
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by galbitz »

I am seeing this frequently on v12 as well. Disabling the job, rebooting the server, rescanning the wasabi repo fixes the issue for a day, or two. I have seen it on all deployments I manage, but it occurs most frequently on large (3tb) vms, even though the incremental for these vms are reasonably small. It has worked for as little as a day, but never more then a week. The issue seems to crop up when cleanup is needed, I had 0 issues the first 30 days or so, but once there are restore points that need to be cleaned up things start not working. In addition to the error posted above I also see.

REST API error: 'Not Found', error code: 404
Agent failed to process method {Cloud.EnumOutdatedCheckpoints}.
galbitz
Enthusiast
Posts: 42
Liked: 5 times
Joined: May 17, 2018 2:22 pm
Full Name: grant albitz
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by galbitz » 1 person likes this post

An update on my end this may not apply to all, if you are using wasabi and have Comcast business then the issue is likely not veeam at all. Its their new "Comcast Business SecurityEdge" feature. Which hijacks random DNS requests and responds with an invalid address (a nord address). I temporarily made a fake wasabisys.com dns zone on our internal server and populated it with the correct dns records for the time being. Far from ideal. That did seem to stabilize the backups at least for today. I couldn't get a single one to succeed after all the usual tricks.
NorthGuard
Service Provider
Posts: 103
Liked: 6 times
Joined: Jan 19, 2022 4:48 pm
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by NorthGuard »

galbitz wrote: May 14, 2023 12:55 pm I am seeing this frequently on v12 as well. Disabling the job, rebooting the server, rescanning the wasabi repo fixes the issue for a day, or two. I have seen it on all deployments I manage, but it occurs most frequently on large (3tb) vms, even though the incremental for these vms are reasonably small. It has worked for as little as a day, but never more then a week. The issue seems to crop up when cleanup is needed, I had 0 issues the first 30 days or so, but once there are restore points that need to be cleaned up things start not working. In addition to the error posted above I also see.

REST API error: 'Not Found', error code: 404
Agent failed to process method {Cloud.EnumOutdatedCheckpoints}.
Are there any updates from Veeam on this issue?

It occurs at multiple clients and seems to be pretty random, though I most regularly on large VMs for me as well.

The jobs are backing up to a local QNAP ( using QuObjects ), so Internet providers aren't a variable.

I've had a couple of cases open with Veeam, but they close the cases because I don't follow up... though I am the one waiting for them?!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by Gostev »

QNAP QuObjects are problematic in general according to some other threads I saw, so this could be due to them.
Plus the error actually indicates some storage-side issues vs. an issue with Internet providers.

From the product perspective, there are no known issues or bugs that would cause such behavior.
No further posts in this thread for many months now is also a good indication.
NorthGuard
Service Provider
Posts: 103
Liked: 6 times
Joined: Jan 19, 2022 4:48 pm
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by NorthGuard »

Even though QuObjects on are Veeam's supported list?

object-storage-f52/unoffizial-compatibi ... 56956.html
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS REST API error: 'Not Found', error code: 404 OR The specified version does not exist. Code: NoSuchVersion'

Post by Gostev »

That is correct. Supported from API perspective does not automatically mean reliable, scalable etc.
Post Reply

Who is online

Users browsing this forum: No registered users and 15 guests