All instances of the storage metadata are corrupted.

Availability for the Always-On Enterprise

Re: All instances of the storage metadata are corrupted.

Veeam Logoby zak2011 » Mon Oct 21, 2013 9:45 am

Thankyou Gostev. In my case, the first time I got this error on a new Western Digital 4TB drive, I was told Storage was the issue. However the next day I brought another new Western Digital My Book 4TB ext drive, and the same thing happens just for two jobs. It was quite difficult and frustrating to hear from support again Storage was the problem.
I can run Sure Backups..can it be linked for backup copy jobs?
Is there a certified list of Storage Products ( external drives included) which has been tried and tested by Veeam which we can use?

Thanks
zak2011
Expert
 
Posts: 367
Liked: 41 times
Joined: Tue May 15, 2012 2:21 pm
Full Name: Arun

Re: All instances of the storage metadata are corrupted.

Veeam Logoby Vitaliy S. » Mon Oct 21, 2013 10:39 am

Currently backup copy jobs cannot be linked to SureBackup jobs, but you can use this workaround > Backup Copy - Application Group

Veeam does not certify storage vendors, so it's definitely worth using SureBackup jobs.
Vitaliy S.
Veeam Software
 
Posts: 21168
Liked: 1259 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: All instances of the storage metadata are corrupted.

Veeam Logoby Gostev » Mon Oct 21, 2013 2:19 pm

Using SureBackup against Backup Copy jobs is a bad design (which is exactly why it is not supported). You should be testing your backups with SureBackup only once, at source (in the primary backup repository), before you copy them. Backup Copy jobs have built-in verification engine that will ensure that VM data in backup copies is bit-identical to the source backups. It will do so by physically reading the data from backup files, and thus will catch any corruption issues. Check out the advanced setting of your Backup Copy job, you can control the verification schedule there.

As far as your issue, I would try connecting your external drives to a different server, and if that does not help, trying a non-WD drive. Also, please let me know the support case ID you have referenced in your last post, as I am curious to take a look at support findings in this case.
Gostev
Veeam Software
 
Posts: 22401
Liked: 2676 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: All instances of the storage metadata are corrupted.

Veeam Logoby Gostev » Mon Oct 21, 2013 2:38 pm

zak2011 wrote:Western Digital My Book 4TB ext drive

Keep in mind that consumer hard drives have very high URE (Unrecoverable Read Error), and most people do not realize how high it is. Here is the rule of thumb on how much data you need to read from the storage medium before running into an unrecoverable read error:

Consumer SATA > 1 error per 10 TB
Enterprise SATA > 1 error per 100 TB
Enterprise FC/SAS > 1 error per 1 PB
Tape (LTO) > 1 error per 10 PB (if handled properly)
Tape (Enterprise) > 1 error per 1 EB (if handled properly)

This means that in your case, if you write 4TB backup file to your storage, and immediately read it back twice (for example first to test, then to perform actual restore), you will get on average 1 corrupted bit, and thus infamous CRC error on restore.

For home NAS, this is completely irrelevant, because you will not notice a bad bit with photos, music or videos (and there is no CRC validation on those). Where this becomes an issue is with software that does not trust the storage, and checksums all the data it deals with (such as data protection software).
Gostev
Veeam Software
 
Posts: 22401
Liked: 2676 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: All instances of the storage metadata are corrupted.

Veeam Logoby zak2011 » Mon Oct 21, 2013 3:04 pm

Thanks for the recommendations, Gostev. This is case 00439056.
The prime reason I was taken aback was when I was asked to do a sudden restore it didnt work as I was quite confident that I could recover from the external drive ,since just the previous day the backup copy job completed the job with full success (as per the Veeam backup copy job statistics). So this means..i dont need further verification right?
Honestly I wouldnt have a problem if all my backup copy jobs on the ext media were unrecoverable.. then I would surely suspect the drive.
But since this happened only in the case of only one or two jobs from which I needed to restore..it was quite a let down. The same vbk which was on the primary backup repository worked without any problem.( i was a bit fortunate to keep the retension period a bit longer).
The vbk I was talking about was just 200GB each copied to a 4TB ext drive.

Thanks!
zak2011
Expert
 
Posts: 367
Liked: 41 times
Joined: Tue May 15, 2012 2:21 pm
Full Name: Arun

Re: All instances of the storage metadata are corrupted.

Veeam Logoby mbiesheuvel » Tue Feb 25, 2014 6:50 pm

I saw this thread while searching for the same error. We also have a customer with a WD 3TB disk (My Book). The files we copied to the USB3.0 Disk are backups of 1,5 TB - 2 TB. Shame is that the same backup can be restored from the NAS but when copied to USB disks (WD) gets corrupted while our copy program (SyncBack) says it has been succesfully completed.

Is it just WD then that is giving problems? I find it hard to believe.
mbiesheuvel
Influencer
 
Posts: 20
Liked: 1 time
Joined: Sun Dec 13, 2009 7:23 am
Full Name: Michel Biesheuvel

Re: All instances of the storage metadata are corrupted.

Veeam Logoby lp@albersdruck.de » Tue Feb 25, 2014 6:59 pm

Gostev wrote:
zak2011 wrote:Western Digital My Book 4TB ext drive

Keep in mind that consumer hard drives have very high URE (Unrecoverable Read Error), and most people do not realize how high it is. Here is the rule of thumb on how much data you need to read from the storage medium before running into an unrecoverable read error:

Consumer SATA > 1 error per 10 TB
Enterprise SATA > 1 error per 100 TB
Enterprise FC/SAS > 1 error per 1 PB
Tape (LTO) > 1 error per 10 PB (if handled properly)
Tape (Enterprise) > 1 error per 1 EB (if handled properly)

This means that in your case, if you write 4TB backup file to your storage, and immediately read it back twice (for example first to test, then to perform actual restore), you will get on average 1 corrupted bit, and thus infamous CRC error on restore.

For home NAS, this is completely irrelevant, because you will not notice a bad bit with photos, music or videos (and there is no CRC validation on those). Where this becomes an issue is with software that does not trust the storage, and checksums all the data it deals with (such as data protection software).


Thanks for this numbers Gostev.
I am glad I run my storage on ZFS and do regular scrubs to catch (and correct) this early.

As for the WD external drives, I seem to remember there was a report that some WD series controllers had problems with large transfers over USB. I can't find that report anymore though :(
lp@albersdruck.de
Enthusiast
 
Posts: 81
Liked: 31 times
Joined: Mon Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec

Re: All instances of the storage metadata are corrupted.

Veeam Logoby mcz » Mon Sep 25, 2017 7:40 am

This thread is fantastic, thanks very much!
mcz
Expert
 
Posts: 206
Liked: 19 times
Joined: Tue Jul 19, 2016 8:39 am
Full Name: Michael

[MERGED] How to restore .vib file - ransom

Veeam Logoby Igor N. M. » Mon Jul 09, 2018 8:10 pm

Hi!

I have a new pontential client that have a big problem:

The backup server was attacked with Ransomware and the backup files were "encrypted".

But looking at the files with an Exadecimal editor, apparently the data is there. It looks like it corrupted only the file header.
Is it possible to redo this header?
I tried to copy from another server, but I could not open the new file with "VBK Extract". It says that the header is corrupted.

I could not open a support because the client uses a free version.

Message when trying to open the encrypted file: Length can not be less than zero. Parameter name: length

Message when trying to open the modified file (Hex): All instances of the storage metadata are corrupted
Igor N. M.
Influencer
 
Posts: 18
Liked: 3 times
Joined: Wed Oct 23, 2013 12:38 am
Full Name: Igor N. M.

Re: All instances of the storage metadata are corrupted.

Veeam Logoby PTide » Mon Jul 09, 2018 8:46 pm

Hi Igor,

You or the client can request an evaluation license so that you can open an evalutation support ticket. Meanwhile please take a look at this article.

Thanks
PTide
Veeam Software
 
Posts: 4027
Liked: 336 times
Joined: Tue May 19, 2015 1:46 pm

Re: All instances of the storage metadata are corrupted.

Veeam Logoby Igor N. M. » Tue Jul 10, 2018 12:18 am

Hi. I opened a call with Veeam.

I read that the file 'vib' has two indices, correct? I believe the first one (header) has been destroyed. The start of files is "blank". But up front, they are intact. I believe that have only damaged the 1st Mb of the files.

How is the structure of these vbk files? I wanted to understand more about the internal structure.
Igor N. M.
Influencer
 
Posts: 18
Liked: 3 times
Joined: Wed Oct 23, 2013 12:38 am
Full Name: Igor N. M.

Re: All instances of the storage metadata are corrupted.

Veeam Logoby PTide » Tue Jul 10, 2018 10:20 am

I read that the file 'vib' has two indices, correct?

Yes, that's correct.

I believe that have only damaged the 1st Mb of the files.

Nevertheless, without correct meta it is hardly possible to make any sense of that data. If the second instance of meta had survived, you wouldn't see the word "All" in the error message.

Thanks
PTide
Veeam Software
 
Posts: 4027
Liked: 336 times
Joined: Tue May 19, 2015 1:46 pm

Re: All instances of the storage metadata are corrupted.

Veeam Logoby Igor N. M. » Tue Jul 10, 2018 5:04 pm

Nevertheless, without correct meta it is hardly possible to make any sense of that data. If the second instance of meta had survived, you wouldn't see the word "All" in the error message.


So. Actually I do not think it corrupted the two indices. It seems to have erased only the first 256Kb of the file.
If the index stays after this point, I think you can recover it, perhaps manually, or by running a scan on it (If you have any tools for this)

Support ID: 03093820
Veeam support has seen the file too, but he can not do much.
I saw that it has a tool, VeeamR that can do some checks.
I did not have much time with them yesterday, I was pretty tired. I would like to try again today. Do you know if it's possible to get this tool? (To B&R 9.5 vkb file)
Igor N. M.
Influencer
 
Posts: 18
Liked: 3 times
Joined: Wed Oct 23, 2013 12:38 am
Full Name: Igor N. M.

Re: All instances of the storage metadata are corrupted.

Veeam Logoby PTide » Tue Jul 10, 2018 8:50 pm

Well, apparently both of them are gone. If they are not, then why would the message clearly say that "All instances of the storage metadata are corrupted" ? Also please tell us what is that "VeeamR" tool that you are referring to, where did you see it?

If you believe that there is something what can be done, then please escalate the ticket.

Thanks
PTide
Veeam Software
 
Posts: 4027
Liked: 336 times
Joined: Tue May 19, 2015 1:46 pm

Re: All instances of the storage metadata are corrupted.

Veeam Logoby Igor N. M. » Tue Jul 10, 2018 10:59 pm

PTide wrote:Well, apparently both of them are gone. If they are not, then why would the message clearly say that "All instances of the storage metadata are corrupted" ?


So. I think it could be something like the inode table of a filesystem. The first is at the beginning of the file. The second block, in the middle or at the end. Something like.
And Veeam looks at the beginning of the file location of the table. As everything is empty, he says he did not find it. But if you look in the file, it will be somewhere.

Just as it corrupts an FS and we use GetDataBack or Partition Magic to retrieve the table.

I'm wondering what the VKB / VIB database is, where its "MBR" has been lost. But the data is there.
Even when I look at the file, I clearly see the name of the VMs and their settings. Soon after, what appears to be the VM itself.


PTide wrote:Also please tell us what is that "VeeamR" tool that you are referring to, where did you see it?


I saw when the Veeam support went to the server. VeeamR tool.
They downloaded it in TeamViewer, but they deleted the folder with the files at the end of the support. (And GetDataBack and others don't run on this server. Server is corrupted by virus)
Igor N. M.
Influencer
 
Posts: 18
Liked: 3 times
Joined: Wed Oct 23, 2013 12:38 am
Full Name: Igor N. M.

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: chris.sprague and 46 guests