All instances of the storage metadata are corrupted.

[olofc]

Hi,
I am getting an error that occured yesterday after the veeam backup server rebooted during a backup, we are getting this message in every backup and we cannot do any restore from the actual backupjob. New backupjobs works fine however.

Dir failed, storageFileName "F:\Backup\Backup.vbk"
All instances of the storage metadata are corrupted.

How can i recover from this?

[Vitaliy S.]

Hello Olof,

Please contact our support (support@veeam.com) regarding this question, also do not forget to provide us all the logs from Help->Support Information, however it seems like your backups got corrupted due to that reboot, but let us see the logs to see if we could help you restore from it.

Thank you.

[Gostev]

Also please include information on what is the storage you are backing up to. We have specifically tested all available backup storage types in the scenario with hard resetting Veeam Backup server while backup job is running, and so chance of such corruption should be extremely low. We should be able to investigate why this has happened from job logs.

[zak2011]

I encountered the same error while trying to import backups copied using the backup copy job in v7 from an offsite external drive. The job statisitics say the backup copy job completed successfully in Veeam. However during the import the same error comes in.

[Vitaliy S.]

What is the external drive? Is it a regular disk? Do you have a support case ticket opened on this?

[zak2011]

The external drive is a Western digital My book Essentials. I have a support case opened on this Case # 00439056. The backup copy job statistics say that the job completed successfully and the full transformation was completed.
But the import does not succeed. By default the number of restore points is set to 2.
So once backup job completed for this particular job, I could see a vbk and a vbm file on the target repository ( external drive). However for other jobs also that copied to the same drive with similar retension settings, there is a vbk, vib and a vbm file. So does this mean the next restore point was not copied? If it was strangely there was no warnings or indications about this.
Thanks

[Vitaliy S.]

Arun,

However for other jobs also that copied to the same drive with similar retension settings

So only 1 backup copy job is affected by this? Or other jobs are just regular backup jobs?

Thanks!

[zak2011]

Hi Vitaliy,

Two Backup copy jobs were affected. All other backup copy jobs seems to be fine.

Thanks.

[Vitaliy S.]

Ok, need to understand what is the difference between these jobs, keep us updated on the support team research.

[zak2011]

Yes, will do Vitaliy.

[mongie]

I've had this with a v6.5 backup chain, unfortunately, the response from support was eventually "Oh dear, looks like they're un-recoverable".

[electricd7]

Just had the same experience. Nothing could be done and we had to abandon the backup-chain and begin a new one. Support suggested it was a storage problem and to check storage. We are writing to a a 4 head distributed gluster filesystem with over 200TB. That will prove to be difficult to say the least.

[Gostev]

You can catch the issues like this very early on by using periodic SureBackup job with full scan option enabled (new v7 feature). This will physically read all data blocks holding the tested restore point, and ensure that the content of the block matches the CRC stored with the block.

[zak2011]

I have had the same experience and support says nothing can be done and there is no other way to determine at which step and at which drive the data get corrupted.
If Veeam job statistics say the job completed successfully and the backup is copied why does this happen?

[Gostev]

Most likely some issue with the storage, in the past years we've been mostly seeing this with low end NAS featuring questionable performance optimizations. For example, the some storage will not handle FLUSH command correctly, returning success before the data hits the disks to achieve better performance numbers. Or, as simple as storage not writing some other data instead of the data we asked it to write due to faulty hardware (in 6 years and with 80'000+ customers, we've seen it all by now). It is very important that the storage does not "cheat" us.

Veeam backup file format includes two identical records of storage metadata for redundancy, and they are never updated at the same time, for at least one to remain "good" in case of a crash, or data corruption occurring during the file update.

So, it is simply impossible to break both metadata instances when:
1) Storage writes the data we asked it to write (not the case with faulty RAID controllers)
2) Storage medium stores the data correctly (not the case with silent corruptions issues aka bit rot)
3) Properly processes FLUSH command. The latter is a synchronous call for us, meaning we issue FLUSH command and wait for the data to hit the disks before moving on. If storage returns success but flush does not really happen, and we start updating another instance of metadata while first one is unsaved, and this is one way to run into the said issue.

In human language, the issues look like:
1) We ask storage to write "MOM", but it writes "DAD" instead and returns success.
2) We ask storage to write "MOM", and it writes "MOM" and returns success, but if you try to read the data block, you will get "MAM".
3) We ask storage to commit the write of "MOM" to disks, and it returns success but does not actually write data to disks, keeping it in buffer for a short period of time for performance optimization purposes.

Answering your question, we can only judge on these reported successes, so we mark the job as successful. This is why, it is very important to use SureBackup to verify that what was written into the backup file is what we asked, especially once you get this error at least once and your backup storage becomes a suspect.

[zak2011]

Thankyou Gostev. In my case, the first time I got this error on a new Western Digital 4TB drive, I was told Storage was the issue. However the next day I brought another new Western Digital My Book 4TB ext drive, and the same thing happens just for two jobs. It was quite difficult and frustrating to hear from support again Storage was the problem.
I can run Sure Backups..can it be linked for backup copy jobs?
Is there a certified list of Storage Products ( external drives included) which has been tried and tested by Veeam which we can use?

Thanks

[Vitaliy S.]

Currently backup copy jobs cannot be linked to SureBackup jobs, but you can use this workaround > Backup Copy - Application Group

Veeam does not certify storage vendors, so it's definitely worth using SureBackup jobs.

[Gostev]

Using SureBackup against Backup Copy jobs is a bad design (which is exactly why it is not supported). You should be testing your backups with SureBackup only once, at source (in the primary backup repository), before you copy them. Backup Copy jobs have built-in verification engine that will ensure that VM data in backup copies is bit-identical to the source backups. It will do so by physically reading the data from backup files, and thus will catch any corruption issues. Check out the advanced setting of your Backup Copy job, you can control the verification schedule there.

As far as your issue, I would try connecting your external drives to a different server, and if that does not help, trying a non-WD drive. Also, please let me know the support case ID you have referenced in your last post, as I am curious to take a look at support findings in this case.

[Gostev]

Western Digital My Book 4TB ext drive

Keep in mind that consumer hard drives have very high URE (Unrecoverable Read Error), and most people do not realize how high it is. Here is the rule of thumb on how much data you need to read from the storage medium before running into an unrecoverable read error:

Consumer SATA > 1 error per 10 TB
Enterprise SATA > 1 error per 100 TB
Enterprise FC/SAS > 1 error per 1 PB
Tape (LTO) > 1 error per 10 PB (if handled properly)
Tape (Enterprise) > 1 error per 1 EB (if handled properly)

This means that in your case, if you write 4TB backup file to your storage, and immediately read it back twice (for example first to test, then to perform actual restore), you will get on average 1 corrupted bit, and thus infamous CRC error on restore.

For home NAS, this is completely irrelevant, because you will not notice a bad bit with photos, music or videos (and there is no CRC validation on those). Where this becomes an issue is with software that does not trust the storage, and checksums all the data it deals with (such as data protection software).

[zak2011]

Thanks for the recommendations, Gostev. This is case 00439056.
The prime reason I was taken aback was when I was asked to do a sudden restore it didnt work as I was quite confident that I could recover from the external drive ,since just the previous day the backup copy job completed the job with full success (as per the Veeam backup copy job statistics). So this means..i dont need further verification right?
Honestly I wouldnt have a problem if all my backup copy jobs on the ext media were unrecoverable.. then I would surely suspect the drive.
But since this happened only in the case of only one or two jobs from which I needed to restore..it was quite a let down. The same vbk which was on the primary backup repository worked without any problem.( i was a bit fortunate to keep the retension period a bit longer).
The vbk I was talking about was just 200GB each copied to a 4TB ext drive.

Thanks!

[mbiesheuvel]

I saw this thread while searching for the same error. We also have a customer with a WD 3TB disk (My Book). The files we copied to the USB3.0 Disk are backups of 1,5 TB - 2 TB. Shame is that the same backup can be restored from the NAS but when copied to USB disks (WD) gets corrupted while our copy program (SyncBack) says it has been succesfully completed.

Is it just WD then that is giving problems? I find it hard to believe.

[lp@albersdruck.de]

Keep in mind that consumer hard drives have very high URE (Unrecoverable Read Error), and most people do not realize how high it is. Here is the rule of thumb on how much data you need to read from the storage medium before running into an unrecoverable read error:

Consumer SATA > 1 error per 10 TB
Enterprise SATA > 1 error per 100 TB
Enterprise FC/SAS > 1 error per 1 PB
Tape (LTO) > 1 error per 10 PB (if handled properly)
Tape (Enterprise) > 1 error per 1 EB (if handled properly)

This means that in your case, if you write 4TB backup file to your storage, and immediately read it back twice (for example first to test, then to perform actual restore), you will get on average 1 corrupted bit, and thus infamous CRC error on restore.

For home NAS, this is completely irrelevant, because you will not notice a bad bit with photos, music or videos (and there is no CRC validation on those). Where this becomes an issue is with software that does not trust the storage, and checksums all the data it deals with (such as data protection software).

Thanks for this numbers Gostev.
I am glad I run my storage on ZFS and do regular scrubs to catch (and correct) this early.

As for the WD external drives, I seem to remember there was a report that some WD series controllers had problems with large transfers over USB. I can't find that report anymore though

[mcz]

This thread is fantastic, thanks very much!

[Igor N. M.]

Hi!

I have a new pontential client that have a big problem:

The backup server was attacked with Ransomware and the backup files were "encrypted".

But looking at the files with an Exadecimal editor, apparently the data is there. It looks like it corrupted only the file header.
Is it possible to redo this header?
I tried to copy from another server, but I could not open the new file with "VBK Extract". It says that the header is corrupted.

I could not open a support because the client uses a free version.

Message when trying to open the encrypted file: Length can not be less than zero. Parameter name: length

Message when trying to open the modified file (Hex): All instances of the storage metadata are corrupted

[P.Tide]

Hi Igor,

You or the client can request an evaluation license so that you can open an evalutation support ticket. Meanwhile please take a look at this article.

Thanks

[Igor N. M.]

Hi. I opened a call with Veeam.

I read that the file 'vib' has two indices, correct? I believe the first one (header) has been destroyed. The start of files is "blank". But up front, they are intact. I believe that have only damaged the 1st Mb of the files.

How is the structure of these vbk files? I wanted to understand more about the internal structure.

[P.Tide]

I read that the file 'vib' has two indices, correct?

Yes, that's correct.

I believe that have only damaged the 1st Mb of the files.

Nevertheless, without correct meta it is hardly possible to make any sense of that data. If the second instance of meta had survived, you wouldn't see the word "All" in the error message.

Thanks

[Igor N. M.]

Nevertheless, without correct meta it is hardly possible to make any sense of that data. If the second instance of meta had survived, you wouldn't see the word "All" in the error message.

So. Actually I do not think it corrupted the two indices. It seems to have erased only the first 256Kb of the file.
If the index stays after this point, I think you can recover it, perhaps manually, or by running a scan on it (If you have any tools for this)

Support ID: 03093820
Veeam support has seen the file too, but he can not do much.
I saw that it has a tool, VeeamR that can do some checks.
I did not have much time with them yesterday, I was pretty tired. I would like to try again today. Do you know if it's possible to get this tool? (To B&R 9.5 vkb file)

[P.Tide]

Well, apparently both of them are gone. If they are not, then why would the message clearly say that "All instances of the storage metadata are corrupted" ? Also please tell us what is that "VeeamR" tool that you are referring to, where did you see it?

If you believe that there is something what can be done, then please escalate the ticket.

Thanks

[Igor N. M.]

Well, apparently both of them are gone. If they are not, then why would the message clearly say that "All instances of the storage metadata are corrupted" ?

So. I think it could be something like the inode table of a filesystem. The first is at the beginning of the file. The second block, in the middle or at the end. Something like.
And Veeam looks at the beginning of the file location of the table. As everything is empty, he says he did not find it. But if you look in the file, it will be somewhere.

Just as it corrupts an FS and we use GetDataBack or Partition Magic to retrieve the table.

I'm wondering what the VKB / VIB database is, where its "MBR" has been lost. But the data is there.
Even when I look at the file, I clearly see the name of the VMs and their settings. Soon after, what appears to be the VM itself.

Also please tell us what is that "VeeamR" tool that you are referring to, where did you see it?

I saw when the Veeam support went to the server. VeeamR tool.
They downloaded it in TeamViewer, but they deleted the folder with the files at the end of the support. (And GetDataBack and others don't run on this server. Server is corrupted by virus)