Availability for the Always-On Enterprise
zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: All instances of the storage metadata are corrupted.

Post by zak2011 » Oct 21, 2013 9:45 am

Thankyou Gostev. In my case, the first time I got this error on a new Western Digital 4TB drive, I was told Storage was the issue. However the next day I brought another new Western Digital My Book 4TB ext drive, and the same thing happens just for two jobs. It was quite difficult and frustrating to hear from support again Storage was the problem.
I can run Sure Backups..can it be linked for backup copy jobs?
Is there a certified list of Storage Products ( external drives included) which has been tried and tested by Veeam which we can use?

Thanks

Vitaliy S.
Veeam Software
Posts: 21631
Liked: 1297 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: All instances of the storage metadata are corrupted.

Post by Vitaliy S. » Oct 21, 2013 10:39 am

Currently backup copy jobs cannot be linked to SureBackup jobs, but you can use this workaround > Backup Copy - Application Group

Veeam does not certify storage vendors, so it's definitely worth using SureBackup jobs.

Gostev
Veeam Software
Posts: 23116
Liked: 2917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: All instances of the storage metadata are corrupted.

Post by Gostev » Oct 21, 2013 2:19 pm

Using SureBackup against Backup Copy jobs is a bad design (which is exactly why it is not supported). You should be testing your backups with SureBackup only once, at source (in the primary backup repository), before you copy them. Backup Copy jobs have built-in verification engine that will ensure that VM data in backup copies is bit-identical to the source backups. It will do so by physically reading the data from backup files, and thus will catch any corruption issues. Check out the advanced setting of your Backup Copy job, you can control the verification schedule there.

As far as your issue, I would try connecting your external drives to a different server, and if that does not help, trying a non-WD drive. Also, please let me know the support case ID you have referenced in your last post, as I am curious to take a look at support findings in this case.

Gostev
Veeam Software
Posts: 23116
Liked: 2917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: All instances of the storage metadata are corrupted.

Post by Gostev » Oct 21, 2013 2:38 pm

zak2011 wrote:Western Digital My Book 4TB ext drive
Keep in mind that consumer hard drives have very high URE (Unrecoverable Read Error), and most people do not realize how high it is. Here is the rule of thumb on how much data you need to read from the storage medium before running into an unrecoverable read error:

Consumer SATA > 1 error per 10 TB
Enterprise SATA > 1 error per 100 TB
Enterprise FC/SAS > 1 error per 1 PB
Tape (LTO) > 1 error per 10 PB (if handled properly)
Tape (Enterprise) > 1 error per 1 EB (if handled properly)

This means that in your case, if you write 4TB backup file to your storage, and immediately read it back twice (for example first to test, then to perform actual restore), you will get on average 1 corrupted bit, and thus infamous CRC error on restore.

For home NAS, this is completely irrelevant, because you will not notice a bad bit with photos, music or videos (and there is no CRC validation on those). Where this becomes an issue is with software that does not trust the storage, and checksums all the data it deals with (such as data protection software).

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: All instances of the storage metadata are corrupted.

Post by zak2011 » Oct 21, 2013 3:04 pm

Thanks for the recommendations, Gostev. This is case 00439056.
The prime reason I was taken aback was when I was asked to do a sudden restore it didnt work as I was quite confident that I could recover from the external drive ,since just the previous day the backup copy job completed the job with full success (as per the Veeam backup copy job statistics). So this means..i dont need further verification right?
Honestly I wouldnt have a problem if all my backup copy jobs on the ext media were unrecoverable.. then I would surely suspect the drive.
But since this happened only in the case of only one or two jobs from which I needed to restore..it was quite a let down. The same vbk which was on the primary backup repository worked without any problem.( i was a bit fortunate to keep the retension period a bit longer).
The vbk I was talking about was just 200GB each copied to a 4TB ext drive.

Thanks!

mbiesheuvel
Influencer
Posts: 20
Liked: 1 time
Joined: Dec 13, 2009 7:23 am
Full Name: Michel Biesheuvel
Contact:

Re: All instances of the storage metadata are corrupted.

Post by mbiesheuvel » Feb 25, 2014 6:50 pm

I saw this thread while searching for the same error. We also have a customer with a WD 3TB disk (My Book). The files we copied to the USB3.0 Disk are backups of 1,5 TB - 2 TB. Shame is that the same backup can be restored from the NAS but when copied to USB disks (WD) gets corrupted while our copy program (SyncBack) says it has been succesfully completed.

Is it just WD then that is giving problems? I find it hard to believe.

lp@albersdruck.de
Enthusiast
Posts: 81
Liked: 32 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: All instances of the storage metadata are corrupted.

Post by lp@albersdruck.de » Feb 25, 2014 6:59 pm

Gostev wrote: Keep in mind that consumer hard drives have very high URE (Unrecoverable Read Error), and most people do not realize how high it is. Here is the rule of thumb on how much data you need to read from the storage medium before running into an unrecoverable read error:

Consumer SATA > 1 error per 10 TB
Enterprise SATA > 1 error per 100 TB
Enterprise FC/SAS > 1 error per 1 PB
Tape (LTO) > 1 error per 10 PB (if handled properly)
Tape (Enterprise) > 1 error per 1 EB (if handled properly)

This means that in your case, if you write 4TB backup file to your storage, and immediately read it back twice (for example first to test, then to perform actual restore), you will get on average 1 corrupted bit, and thus infamous CRC error on restore.

For home NAS, this is completely irrelevant, because you will not notice a bad bit with photos, music or videos (and there is no CRC validation on those). Where this becomes an issue is with software that does not trust the storage, and checksums all the data it deals with (such as data protection software).
Thanks for this numbers Gostev.
I am glad I run my storage on ZFS and do regular scrubs to catch (and correct) this early.

As for the WD external drives, I seem to remember there was a report that some WD series controllers had problems with large transfers over USB. I can't find that report anymore though :(

mcz
Expert
Posts: 215
Liked: 26 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: All instances of the storage metadata are corrupted.

Post by mcz » Sep 25, 2017 7:40 am

This thread is fantastic, thanks very much!

Igor N. M.
Influencer
Posts: 18
Liked: 3 times
Joined: Oct 23, 2013 12:38 am
Full Name: Igor N. M.
Contact:

[MERGED] How to restore .vib file - ransom

Post by Igor N. M. » Jul 09, 2018 8:10 pm

Hi!

I have a new pontential client that have a big problem:

The backup server was attacked with Ransomware and the backup files were "encrypted".

But looking at the files with an Exadecimal editor, apparently the data is there. It looks like it corrupted only the file header.
Is it possible to redo this header?
I tried to copy from another server, but I could not open the new file with "VBK Extract". It says that the header is corrupted.

I could not open a support because the client uses a free version.

Message when trying to open the encrypted file: Length can not be less than zero. Parameter name: length

Message when trying to open the modified file (Hex): All instances of the storage metadata are corrupted

PTide
Veeam Software
Posts: 4431
Liked: 365 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted.

Post by PTide » Jul 09, 2018 8:46 pm

Hi Igor,

You or the client can request an evaluation license so that you can open an evalutation support ticket. Meanwhile please take a look at this article.

Thanks

Igor N. M.
Influencer
Posts: 18
Liked: 3 times
Joined: Oct 23, 2013 12:38 am
Full Name: Igor N. M.
Contact:

Re: All instances of the storage metadata are corrupted.

Post by Igor N. M. » Jul 10, 2018 12:18 am

Hi. I opened a call with Veeam.

I read that the file 'vib' has two indices, correct? I believe the first one (header) has been destroyed. The start of files is "blank". But up front, they are intact. I believe that have only damaged the 1st Mb of the files.

How is the structure of these vbk files? I wanted to understand more about the internal structure.

PTide
Veeam Software
Posts: 4431
Liked: 365 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted.

Post by PTide » Jul 10, 2018 10:20 am

I read that the file 'vib' has two indices, correct?
Yes, that's correct.
I believe that have only damaged the 1st Mb of the files.
Nevertheless, without correct meta it is hardly possible to make any sense of that data. If the second instance of meta had survived, you wouldn't see the word "All" in the error message.

Thanks

Igor N. M.
Influencer
Posts: 18
Liked: 3 times
Joined: Oct 23, 2013 12:38 am
Full Name: Igor N. M.
Contact:

Re: All instances of the storage metadata are corrupted.

Post by Igor N. M. » Jul 10, 2018 5:04 pm

Nevertheless, without correct meta it is hardly possible to make any sense of that data. If the second instance of meta had survived, you wouldn't see the word "All" in the error message.
So. Actually I do not think it corrupted the two indices. It seems to have erased only the first 256Kb of the file.
If the index stays after this point, I think you can recover it, perhaps manually, or by running a scan on it (If you have any tools for this)

Support ID: 03093820
Veeam support has seen the file too, but he can not do much.
I saw that it has a tool, VeeamR that can do some checks.
I did not have much time with them yesterday, I was pretty tired. I would like to try again today. Do you know if it's possible to get this tool? (To B&R 9.5 vkb file)

PTide
Veeam Software
Posts: 4431
Liked: 365 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted.

Post by PTide » Jul 10, 2018 8:50 pm

Well, apparently both of them are gone. If they are not, then why would the message clearly say that "All instances of the storage metadata are corrupted" ? Also please tell us what is that "VeeamR" tool that you are referring to, where did you see it?

If you believe that there is something what can be done, then please escalate the ticket.

Thanks

Igor N. M.
Influencer
Posts: 18
Liked: 3 times
Joined: Oct 23, 2013 12:38 am
Full Name: Igor N. M.
Contact:

Re: All instances of the storage metadata are corrupted.

Post by Igor N. M. » Jul 10, 2018 10:59 pm

PTide wrote:Well, apparently both of them are gone. If they are not, then why would the message clearly say that "All instances of the storage metadata are corrupted" ?
So. I think it could be something like the inode table of a filesystem. The first is at the beginning of the file. The second block, in the middle or at the end. Something like.
And Veeam looks at the beginning of the file location of the table. As everything is empty, he says he did not find it. But if you look in the file, it will be somewhere.

Just as it corrupts an FS and we use GetDataBack or Partition Magic to retrieve the table.

I'm wondering what the VKB / VIB database is, where its "MBR" has been lost. But the data is there.
Even when I look at the file, I clearly see the name of the VMs and their settings. Soon after, what appears to be the VM itself.

PTide wrote:Also please tell us what is that "VeeamR" tool that you are referring to, where did you see it?
I saw when the Veeam support went to the server. VeeamR tool.
They downloaded it in TeamViewer, but they deleted the folder with the files at the end of the support. (And GetDataBack and others don't run on this server. Server is corrupted by virus)

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 30 guests