Updating metadata for backup file. Storage has not been uploaded

[kewnev]

We have been experiencing the same issue.

"Updating metadata for backup file <xxxxx>.vib from index Error: Storage has not been uploaded."

In our case, the pattern I have noticed is that if the backup partially works but fails - ie atleast one VM successfully backs up, but not all of them (which then causes a retry, which then usually finishes successfully), the "Storage has not been uploaded" error will reference the VIB file that was created when the aforementioned backup failure happened. We did have a BCJ involved, but I disabled that and the issue is still occurring. From looking at the Veeam history, the offload begins as soon as the backup job fails (ie before the retry), However Veeam doesn't seem to run another offload for the backup job after the retry takes place (It does the 4-hourly SOBR offloads, but these seem to not detect/capture the restore points from the retried backup).
Anyway to get around this for now, I have set Veeam to prevent any offloads from occurring around the times that backups and retry takes place. Hopefully that will workaround the issue until v12 is released.

Note, the procedure for me to temporarily fix the problem is a few more steps than just doing a SOBR re-scan. I have to also manually delete the problematic VIB file from the file system first. Again, this is just a temporary fix because the problem will come up again with a different VIB file.

[veremin]

Any chance you can provide us with a support case number so we can check how investigation went?

[kewnev]

hi Veremin,

I last reported the issue on Dec 15, 2021 under case #05183365. (Due to covid issues in my area in January this year, I wasn't able to do what was last requested by the support tech, so the case automatically closed. I also didn't feel confident that the tech was on the right track for getting to the bottom of the issue (ie requesting for us to create & run a parallel storage extent with new s3 bucket, etc. - we'd then have to wait until the problem popped up again. It seemed a bit too much to do just for elimination with the extra local and S3 storage needed as well as the s3 offload bandwidth) so I didn't try to get the case re-opened).
I also reported the problem previous to that on Jul 21, 2021 under case #02168256. In that case I thought the problem was fixed because it went away after deleting the problematic VIB file(s) and re-scanning SOBR etc. So the case was closed. But then the problem occurred again.

[veremin]

So it does not seem that thorough investigation was conducted and neither root cause, nor the resolution was found. What we have now are two closed cases and educated guesses regarding the experienced issue.

Am I correct that the problem still persists? If so, may I ask you to collect the debug logs and create new support case with them attached? I will ask our QA team to check the ticket and confirm (or not) your findings.

Thanks!

[kewnev]

What we have now are two closed cases and educated guesses regarding the experienced issue.

Yes, and the problem has been there for many months now. I had started losing some faith in the product.

The problem is still there, but after reading mcz's thread I added an offload prevention window which hopefully works around the issue. I have created a new case and uploaded debug logs (#05338465)

[mcz]

Just for information, in my case it's not always done by just using a different offload window - once there's the inconsistency between local and s3 index, it can be necessary to delete the local index (or the affected files/folder) and do a rescan after that (which might take a while). I'm just currently doing exactly that to get rid of the error hopefully.

[veremin]

Thank you for providing us with the new ticket. We will discuss the issue internally and reach you, once we have more information or we need a further assistance from you.

[veremin]

Good news, @kewnev, @mcz,

After reviewing the cases (which indeed are caused by the same issue) we've decided to create a hotfix - R&D team will start working on it soon.

The hotfix will be created for the latest product version (11a P20220302). If you haven't installed it yet, consider doing so within next few weeks, so that you are all set and prepared for the hotfix.

I will keep posting here about the hotfix creation and verification progress.

Thanks guys again for raising this; highly appreciated!

P.S.: kewnev, we will reach your support engineer and update him on the problem status.

[mcz]

Thanks Vladimir, these are great news! i didn't know that there was another patch available, I really do miss Anton's newsletter!

[kewnev]

That's awesome news Vladimir, will certainly be a load off of my mind! I will be sure to apply that patch so that we are prepared for the hotfix.

Out of curiosity, was there something unique about the way mcz and myself had things set up that was triggering this issue? Just a bit surprised that no one else seems to have mentioned having this problem.

Many Thanks again.

[mcz]

In my case it was the fact that the merge of the source backup job has interrupted the backup copy job and then the offload job started with an incomplete restore point, the rest was a chain reaction.

[veremin]

Just a coincidence of special circumstances that has neither popped up, nor been identified before

[kewnev]

hi Veremin,
Any ETA on this patch by any chance?
Many Thanks.

[veremin]

The hotfix has been created and been shared with the case owners. I will double check the situation internally and come back with the results of my findings. Thanks!

[kewnev]

Thanks Vladimir, FYI I haven't received any notification about the hotfix yet.

[mcz]

I haven't received any updates too, but I don't have an active case right now...

[veremin]

The hot fix has been indeed shared with the case owners, but it seems that both cases have been closed due to no response, so this might have been the reason of not sending the notification.

Good news is the hot fix is still available, so if you can re-open your original tickets or create new ones, I will make sure the hot fix ends up in your hands this time

[kewnev]

hi Vladimir, I have now opened a new case to obtain this hotfix. Case #05437019.
Many Thanks,
Nev

[mcz]

Hi Vladimir,

my new case is #05434549

[veremin]

Thanks you, guys, for opening new tickets - we have just contacted your case owners and shared the hotfix with them - so you should get it soon.

[kewnev]

Thanks very much Vladimir. I have applied and tested the hotfix, and so far all is looking good (have also updated the case to report this). Many Thanks, this issue had been causing us grief for more than a year!

Before Hotfix:
19/05/2022 6:00:38 AM :: Starting performance tier offload at 19/05/2022 6:00:34 AM
19/05/2022 6:00:38 AM :: 10 backups will be moved to the capacity tier
19/05/2022 6:00:38 AM :: 1 backups will be copied to the capacity tier
19/05/2022 6:00:39 AM :: Processing ADL MAIN VMware backup (to Local and S3) Error: Storage has not been uploaded.
19/05/2022 6:02:23 AM :: Object storage repository cleanup
19/05/2022 6:02:24 AM :: Offload finished with error at 19/05/2022 6:02:24 AM

After Hotfix:
19/05/2022 8:51:56 AM :: Starting performance tier offload at 19/05/2022 8:51:52 AM a
19/05/2022 8:51:57 AM :: 10 backups will be moved to the capacity tier
19/05/2022 8:51:57 AM :: 1 backups will be copied to the capacity tier
19/05/2022 8:51:57 AM :: Processing ADL MAIN VMware backup (to Local and S3)
19/05/2022 10:47:13 AM :: Object storage repository cleanup
19/05/2022 10:47:13 AM :: Offload finished at 19/05/2022 10:47:13 AM

Forcing an offload ("run tiering job now") shows that there is no longer anything left to process:
19/05/2022 11:06:50 AM :: Starting performance tier offload at 19/05/2022 11:06:47 AM
19/05/2022 11:06:51 AM :: 0 backups will be moved to the capacity tier
19/05/2022 11:06:51 AM :: 0 backups will be copied to the capacity tier
19/05/2022 11:06:51 AM :: Object storage repository cleanup
19/05/2022 11:06:51 AM :: Offload finished at 19/05/2022 11:06:51 AM

[mcz]

I've also applied the hotfix but I'm still observing the situation. So far so good, it looks like it's working as expected (without issues). Thanks Vladimir!

[veremin]

Glad to hear that . We do appreciate your efforts of not only making us aware of the issue, but also pushing us gently with the hotfix creation. By the way, the issue is already fixed in v12 builds. Thanks!

[mcz]

Ok I can also confirm that the hotfix is working perfectly. No issues since then and the offload job does its job when it's time to do so. Thanks guys, the developers, PMs and all people involved again have done and excellent job!

[veremin]

Thank you, Michael, for your kind words (I have already passed them to R&D team), and also thank you for your efforts to bring this issue to our attention - you help us make our product better, which we deeply appreciate!

[mcz]

Vladimir, my error is back
I assume it might be due to the fix "Fix_391749_02eb0ba1e6" which I received from veeam support yesterday (for fixing a broken index). Is the hotfix in this thread not included in the new one? Or is it probably a new permutation of our original bug?

Thanks!

[veremin]

Michael, it might happen that the old fix got removed with the installation of a new one (cause it was not a cumulative patch or similar). I recommend raising this with the support team and referencing the previous issue along with the hotfix. The support engineer should be able to escalate the ticket further and ask the R&D team to create a joint fix. Thanks!

[gjbglobalit]

Having the same problem w/ "Storage not upladed" and have opened case #05945540 to track it.

3/14/2023 1:13:41 PM :: Updating metadata for backup file xxxxxxxx1-prod.vm-86895D2023-03-09T020005_6ADD.vib from index Error: Storage has not been uploaded.

[Mildur]

Hello Gregory

Thanks for the case number.
Let's wait what our support will find out about the issue.

Best,

Fabian

[lorism]

I have the same issue as of last night.
Raised ticket: #05949811

17/03/2023 12:14:53 PM :: Updating metadata for backup file xxxxx2023-02-13T180102_BFC6.vib from index Error: Storage has not been uploaded.