All instances of storage metadata are corrupt

[LeisJohn]

Good Morning,
I'm having issues with my Backup Copy Jobs failing with the error "All instances of storage metadata are corrupted". I have opened a Case # 00723311 with Veeam and they are escalating it up the ladder. I wouldn't be so worried if it were just 1 job but, it 2 days I've had 9 Backup Copy Jobs fail. Here is my question, Has anyone else experienced this with their Backup Copy Jobs and If so, How did you resolve the issue?

Backup infrastructure:
Server 2012 running Veeam 8.0.0.917 (Latest Patch)
ExaGrid Storage devices EX32000, EX7000, and a EX13000(Offsite) All shares are utilizing the "ExaGrid-Veeam Accelerated Data Mover" Transport method.
--Daily Backup Jobs going to the ExaGrid EX32k and EX7k ONSITE
*Backup Copy Jobs going to the ExaGrid EX32k and EX7k ONSITE (2 restore Points, 8-Weekly, 12-Monthly, 7-Yearly)
**Backup Copy Jobs going to the ExaGrid EX13k OFFSITE (5 restore Points, 2-Weekly, 3-Monthly, 7-Yearly)

[Gostev]

Hi, John.

Unfortunately, this error means that recent backup files going from the latest full backup are FUBAR. They are not good for anything at this point, can only be thrown away.

Basically, with our backup file format, we store SIS block metadata twice for redundancy, and never update those instances at the same time, so at least one of them always remains "good" even if another one gets corrupted during the update (for example, incomplete write due to a power loss). Now, this error means that both of those records got corrupted, and we can no longer make sense out of data blocks stored in SIS storage. Normally, this condition can only happen due to a storage level corruption of the backup file - sadly, things do not look good for ExaGrid here...

If you search the forums for the quoted error, you will see similar topics from other users. These are not very common, the last forum case I remember was about half a year ago, and was caused by a data corruption condition in the Riverbed storage appliance, which they promptly fixed with the updated firmware.

Thanks!

[LeisJohn]

Gostev,
So here is my question, How does the Metadata in the VBK file get corrupted if the merge was successful?