Using object storage as a backup target
Post Reply
kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Updating metadata for backup file. Storage has not been uploaded

Post by kewnev »

We have been experiencing the same issue.

"Updating metadata for backup file <xxxxx>.vib from index Error: Storage has not been uploaded."

In our case, the pattern I have noticed is that if the backup partially works but fails - ie atleast one VM successfully backs up, but not all of them (which then causes a retry, which then usually finishes successfully), the "Storage has not been uploaded" error will reference the VIB file that was created when the aforementioned backup failure happened. We did have a BCJ involved, but I disabled that and the issue is still occurring. From looking at the Veeam history, the offload begins as soon as the backup job fails (ie before the retry), However Veeam doesn't seem to run another offload for the backup job after the retry takes place (It does the 4-hourly SOBR offloads, but these seem to not detect/capture the restore points from the retried backup).
Anyway to get around this for now, I have set Veeam to prevent any offloads from occurring around the times that backups and retry takes place. Hopefully that will workaround the issue until v12 is released.

Note, the procedure for me to temporarily fix the problem is a few more steps than just doing a SOBR re-scan. I have to also manually delete the problematic VIB file from the file system first. Again, this is just a temporary fix because the problem will come up again with a different VIB file.

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Any chance you can provide us with a support case number so we can check how investigation went?

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

offload. incomplete restore points

Post by kewnev »

hi Veremin,

I last reported the issue on Dec 15, 2021 under case #05183365. (Due to covid issues in my area in January this year, I wasn't able to do what was last requested by the support tech, so the case automatically closed. I also didn't feel confident that the tech was on the right track for getting to the bottom of the issue (ie requesting for us to create & run a parallel storage extent with new s3 bucket, etc. - we'd then have to wait until the problem popped up again. It seemed a bit too much to do just for elimination with the extra local and S3 storage needed as well as the s3 offload bandwidth) so I didn't try to get the case re-opened).
I also reported the problem previous to that on Jul 21, 2021 under case #02168256. In that case I thought the problem was fixed because it went away after deleting the problematic VIB file(s) and re-scanning SOBR etc. So the case was closed. But then the problem occurred again.

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

So it does not seem that thorough investigation was conducted and neither root cause, nor the resolution was found. What we have now are two closed cases and educated guesses regarding the experienced issue.

Am I correct that the problem still persists? If so, may I ask you to collect the debug logs and create new support case with them attached? I will ask our QA team to check the ticket and confirm (or not) your findings.

Thanks!

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by kewnev » 1 person likes this post

What we have now are two closed cases and educated guesses regarding the experienced issue.
Yes, and the problem has been there for many months now. I had started losing some faith in the product.

The problem is still there, but after reading mcz's thread I added an offload prevention window which hopefully works around the issue. I have created a new case and uploaded debug logs (#05338465)

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

Just for information, in my case it's not always done by just using a different offload window - once there's the inconsistency between local and s3 index, it can be necessary to delete the local index (or the affected files/folder) and do a rescan after that (which might take a while). I'm just currently doing exactly that to get rid of the error hopefully.

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

Thank you for providing us with the new ticket. We will discuss the issue internally and reach you, once we have more information or we need a further assistance from you.

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

Good news, @kewnev, @mcz,

After reviewing the cases (which indeed are caused by the same issue) we've decided to create a hotfix - R&D team will start working on it soon.

The hotfix will be created for the latest product version (11a P20220302). If you haven't installed it yet, consider doing so within next few weeks, so that you are all set and prepared for the hotfix.

I will keep posting here about the hotfix creation and verification progress.

Thanks guys again for raising this; highly appreciated!

P.S.: kewnev, we will reach your support engineer and update him on the problem status.

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

Thanks Vladimir, these are great news! i didn't know that there was another patch available, I really do miss Anton's newsletter!

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by kewnev »

That's awesome news Vladimir, will certainly be a load off of my mind! I will be sure to apply that patch so that we are prepared for the hotfix.

Out of curiosity, was there something unique about the way mcz and myself had things set up that was triggering this issue? Just a bit surprised that no one else seems to have mentioned having this problem.

Many Thanks again.

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

In my case it was the fact that the merge of the source backup job has interrupted the backup copy job and then the offload job started with an incomplete restore point, the rest was a chain reaction.

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

Just a coincidence of special circumstances that has neither popped up, nor been identified before :)

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by kewnev »

hi Veremin,
Any ETA on this patch by any chance?
Many Thanks.

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

The hotfix has been created and been shared with the case owners. I will double check the situation internally and come back with the results of my findings. Thanks!

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by kewnev »

Thanks Vladimir, FYI I haven't received any notification about the hotfix yet.

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

I haven't received any updates too, but I don't have an active case right now...

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

The hot fix has been indeed shared with the case owners, but it seems that both cases have been closed due to no response, so this might have been the reason of not sending the notification.

Good news is the hot fix is still available, so if you can re-open your original tickets or create new ones, I will make sure the hot fix ends up in your hands this time :)

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by kewnev »

hi Vladimir, I have now opened a new case to obtain this hotfix. Case #05437019.
Many Thanks,
Nev

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

Hi Vladimir,

my new case is #05434549

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin » 1 person likes this post

Thanks you, guys, for opening new tickets - we have just contacted your case owners and shared the hotfix with them - so you should get it soon.

kewnev
Enthusiast
Posts: 88
Liked: 22 times
Joined: Jun 17, 2012 1:09 pm
Full Name: Nev
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by kewnev »

Thanks very much Vladimir. I have applied and tested the hotfix, and so far all is looking good (have also updated the case to report this). Many Thanks, this issue had been causing us grief for more than a year!

Before Hotfix:
19/05/2022 6:00:38 AM :: Starting performance tier offload at 19/05/2022 6:00:34 AM
19/05/2022 6:00:38 AM :: 10 backups will be moved to the capacity tier
19/05/2022 6:00:38 AM :: 1 backups will be copied to the capacity tier
19/05/2022 6:00:39 AM :: Processing ADL MAIN VMware backup (to Local and S3) Error: Storage has not been uploaded.
19/05/2022 6:02:23 AM :: Object storage repository cleanup
19/05/2022 6:02:24 AM :: Offload finished with error at 19/05/2022 6:02:24 AM

After Hotfix:
19/05/2022 8:51:56 AM :: Starting performance tier offload at 19/05/2022 8:51:52 AM a
19/05/2022 8:51:57 AM :: 10 backups will be moved to the capacity tier
19/05/2022 8:51:57 AM :: 1 backups will be copied to the capacity tier
19/05/2022 8:51:57 AM :: Processing ADL MAIN VMware backup (to Local and S3)
19/05/2022 10:47:13 AM :: Object storage repository cleanup
19/05/2022 10:47:13 AM :: Offload finished at 19/05/2022 10:47:13 AM

Forcing an offload ("run tiering job now") shows that there is no longer anything left to process:
19/05/2022 11:06:50 AM :: Starting performance tier offload at 19/05/2022 11:06:47 AM
19/05/2022 11:06:51 AM :: 0 backups will be moved to the capacity tier
19/05/2022 11:06:51 AM :: 0 backups will be copied to the capacity tier
19/05/2022 11:06:51 AM :: Object storage repository cleanup
19/05/2022 11:06:51 AM :: Offload finished at 19/05/2022 11:06:51 AM

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

I've also applied the hotfix but I'm still observing the situation. So far so good, it looks like it's working as expected (without issues). Thanks Vladimir!

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin » 1 person likes this post

Glad to hear that :). We do appreciate your efforts of not only making us aware of the issue, but also pushing us gently with the hotfix creation. By the way, the issue is already fixed in v12 builds. Thanks!

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz » 1 person likes this post

Ok I can also confirm that the hotfix is working perfectly. No issues since then and the offload job does its job when it's time to do so. Thanks guys, the developers, PMs and all people involved again have done and excellent job!

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

Thank you, Michael, for your kind words (I have already passed them to R&D team), and also thank you for your efforts to bring this issue to our attention - you help us make our product better, which we deeply appreciate!

mcz
Veeam Legend
Posts: 688
Liked: 145 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by mcz »

Vladimir, my error is back :(
I assume it might be due to the fix "Fix_391749_02eb0ba1e6" which I received from veeam support yesterday (for fixing a broken index). Is the hotfix in this thread not included in the new one? Or is it probably a new permutation of our original bug?

Thanks!

veremin
Product Manager
Posts: 19857
Liked: 2137 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Updating metadata for backup file. Storage has not been uploaded

Post by veremin »

Michael, it might happen that the old fix got removed with the installation of a new one (cause it was not a cumulative patch or similar). I recommend raising this with the support team and referencing the previous issue along with the hotfix. The support engineer should be able to escalate the ticket further and ask the R&D team to create a joint fix. Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest