Case #07195175 Seeding backup issue

[Robert-xl]

Hi Guys,

Just want to ask about a certain situation I'm in.

I created an active full backup (backup copy to disk) - done using an external hard drive and then after the job completed we had this hard drive shipped over to our offsite.

Now when trying to map the backup file, we get the error:
3/22/2024 8:29:18 AM :: Processing TMG-FS1 Error: Failed to reload metadata bank. Declared and actual CRC are different for all bank snapshots.
Failed to download disk 'TMG-FS1-flat.vmdk'.
Reconnectable protocol device was closed.
Failed to upload disk. Skipped arguments: [TMG-FS1-flat.vmdk];
Agent failed to process method {DataTransfer.SyncDisk}.

Support clarified that I may have missed a step to disable the job after the active full thats why it created a vib.

So background is this job involves a machine 15TB in size and they havent got the most reliable internet. So we had to do the seeding process. Create the VBK locally and ship it to our offsite.

I understand that this may be also because our client only uses v11 and our service provider side is already v12 - and the next logical answer will be to redo everything using a v12 - but I am just wondering if there is some workaround i can use for:

1) Make the job forget that it had a vib and focus reading the active full. I tried only placing VBK and VBM in the repository but i guess it doesnt work that easy right?

2) any other suggestions? I have tried using https://www.veeam.com/kb3187 from service provider side but its not generating any backup restore points nor any errors.

[Robert-xl]

Another thing is if we can somehow make changes/modification to the VBM file? so it doesnt read the increment only the active full? dont know if thats possible but it would be helpful to know

[Gostev]

Actually, the error above indicates a massive storage-level data loss/corruption. Such error cannot never be due to "missing some step". Please try using a different external media to ship your backup.

Also please note V11 is no longer supported since last month. So yes, you should start from upgrading your client to a supported product version.

[Robert-xl]

Ok then thank you!

[Robert-xl]

Hey guys we had to redo everything, and now we are seeing a different error.

tenant is now on v12.
Active Full is OK and Complete.

4/11/2024 12:19:24 PM :: Error: All instances of the storage metadata are corrupted.
Failed to open storage for read access. Storage: [TMG-FS1.1D2024-03-29T003020_EAAD.vbk].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{f04e84d7-c2e3-427d-831c-d9b4e32eefad}]. CHMOD mask: [0].
Agent failed to process method {DataTransfer.RestoreText}.

[Gostev]

This error also indicates a storage-level corruption. Did you replace the faulty storage device you used for seeding in your first attempt?

[Robert-xl]

if the seeding device in the first place was faulty, then there should be a problem with the initial active full right?

[Gostev]

No, silent data corruptions are completely unnoticeable to software until it tries to read the data back.
This is why backup products implement integrity checks, which always require reading the data back.

[Robert-xl]

Hi team - this is resolved.

Apparently the first instances of us transferring the VBK from the HDD to our Cloud Repo, it was incomplete.

Veeam support suggested to re-transfer it again, and this time I allotted quite some time to monitor it (around 2-3 days completion) and Boom it works.

So the metadata corruption is just caused by incomplete file transfer (the first time I had a hunch because it finished way to fast).

Please close this thread and thanks for all the amazing suggestions!