rescan fails -> System.Xml.XmlException: The root element does not exist

[mcz]

Dear PM's,

I've tried to move a backup chain from repository A to repository B, but it failed with the error message above. I've tried to figure it out on my own and eventuell I've found the problem: There were some files on the old repository which had 0 bytes in size. So obviously a backup job run but failed (probably due to "out of disk space") and so the orphaned vib stayed there and the next run of the job was successful. It looks like that this isn't a problem when you're going on with the existent chain but as soon as you move the whole thing and do a fresh rescan, it will cause problems...

Probably you wannt to adapt the code there to make it more bulletproof in the future. Just btw, the same exception gets thrown when your vbm has 0 bytes size. Instead of throwing the error, veeam should just check against the 0 size and then just re-create the vbm on its own (instead of doing it manually).

I think those fixes would be quite easily implemented. If they are is not my decision

Thanks!

[PetrM]

Hi Michael,

Thanks a lot for useful feedback! May I ask you to explain how did you come to the conclusion that your rescan issue is related to orphaned .vibs? Basically, rescan parses only .vbm content and creates the necessary database entries based on parsing result, it should not take into account any orphaned files remaining on file system.

Thanks!

[mcz]

Well... I found the (mentioned) exception in the logs and it pointed to the vbm file. That file was fine, had content and was the same as on the source. So I thought it must be something different. Doing a ls -l showed me some files with 0 size and once I've cleaned up all of them (vib's and also vlm's) the next rescan was fine. No other changes were made, so that's why I've made that conclusion... You'll find the latest log bundle (when the error occured) in the case 05559189

Should you need more information for the investigation, let me know. Thanks Petr!

[PetrM]

Your reasoning fully makes sense but I'm surprised to hear that rescan is affected by orphaned files. I will investigate this issue deeper with our QA and support engineers.

Thanks!

[mcz]

Petr, I've forgotten a little detail (just realized it yesterday while driving a car ): I also had some orphaned vib's from 2021 on the repository, those vib's had a certain size. For sure they can't be imported because there was no vbk on the repo, maybe they were responsible for the failed rescan?

[PetrM]

Hi Michael,

Many thanks for info! It's difficult to say what was the reason without debug logs analysis, I need to discuss this issue with my colleagues before I can make any conclusions.

Thanks!

[mcz]

sure, take your time

[PetrM]

Hi Michael,

So far, there is no understanding how orphaned .vibs can affect rescan logic, rescan reads .vbm content only and updates configuration database accordingly. I assume you deleted any files with 0 bytes size, some of them were metatdata .vbm files, and rescan simply didn't see these corrupted .vbm-s and completed successfully. I think .vib files deletion played no role in your case.

Speaking about automatic .vbm recreation by backup job, I wouldn't do it because such an approach would be dangerous and would provoke even more serious issues as we never know the real reason of it. We don't see this specific problem quite often and it's better to research the root cause with our support engineers than to build automated workflows for eliminating symptoms.

Thanks!

[mcz]

So far, there is no understanding how orphaned .vibs can affect rescan logic, rescan reads .vbm content only and updates configuration database accordingly

Right... Creating a new repository with only a vbm does import the whole chain, allthough there is no chain at all. So I have nothing to add here...

Speaking about automatic .vbm recreation by backup job, I wouldn't do it because such an approach would be dangerous and would provoke even more serious issues

ok, but I just would like to remind that the fix is nothing more than just delete that file - if your vbm is 0 bytes, there's no onger anything to read, right? I mean veeam recreates the vbm on its own when there's no vbm, so I don't see any danger here.

But it's okay and I understand that it's simply not possible to implement every possible feature. Probably the next one

Thanks!

[PetrM]

Hi Michael,

I agree that the fix seems to be simple at first sight, but the main risk here is that the issue with .vbm itself proves that there is a bigger systemic problem which must be investigated. Therefore, it's not safe to recreate metadata automatically until the main issue is resolved because we cannot be sure that you won't encounter other issues with the "repaired" chain in all possible scenarios, for example during restore or import or evacuation from one extent to another etc. Anyway, thanks for your feedback and keep sharing your ideas!

Thanks!