Comprehensive data protection for all workloads
Post Reply
richard_setford
Lurker
Posts: 2
Liked: never
Joined: Sep 23, 2010 6:16 am
Full Name: Richard Setford
Contact:

Backup failure

Post by richard_setford »

Hi,

Running Veeam Backup 4.1 and I'm getting this error shortly after the job starts:

'All instances of the storage metadata are corrupted.'

What do I need to do to clear this and get my backups running again?

Thanks,

Richard
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup failure

Post by Gostev »

Hi Richard, you should re-create your backup job. Thanks!
richard_setford
Lurker
Posts: 2
Liked: never
Joined: Sep 23, 2010 6:16 am
Full Name: Richard Setford
Contact:

Re: Backup failure

Post by richard_setford »

Thanks Anton.

Out of interest, do you know what causes this problem?

I've just updated from 4.1 to 4.1.2 in case that helps.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup failure

Post by Vitaliy S. »

Richard, we should be able tell you more if you send all the product log files to our technical team. Basically, it seems like there was some kind of a target storage problem.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup failure

Post by Gostev »

Richard, are you backing up to a NAS (network share) or locally?
Does this storage use RAID? Did you perform RAID rebuild recently?
jkeiser
Lurker
Posts: 2
Liked: never
Joined: Feb 03, 2011 9:00 pm
Full Name: justin keiser

Re: Backup failure

Post by jkeiser »

so basically what you're saying is that this backup messed up so we should start over and we're on our own? seems to happen too often. garbage.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup failure

Post by Gostev »

jkeiser wrote:so basically what you're saying is that this backup messed up so we should start over and we're on our own?
Hi Justin, no we never said anything like this. We were actually responding to Richard above.
What issue are you experiencing, the same error? What is your support case number?
damocue
Veeam ProPartner
Posts: 45
Liked: never
Joined: Feb 16, 2011 9:10 am
Full Name: David Molina
Location: Valencia, Spain
Contact:

Re: Backup failure

Post by damocue »

Hello,

I have the same issue. Is there any option different to rebuild the jobs?
My support ticket is ID#5176860.

Regards,
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup failure

Post by Gostev »

Wow, you have bumped a really old thread with that post... these kind of issues certainly do not happen very often.

Basically, this error means incredible corruption of the backup file... there is a perfect army term for that condition, FUBAR. You don't have to rebuild the job, but you most definitely need to initiate active full backup in that job. And prior to that, I would recommend changing the backup repository in the job settings to the one located on a different storage, because these kind of issues can only be caused by very bad underlying storage issues.

Thanks!
genachka
Novice
Posts: 4
Liked: never
Joined: Mar 04, 2013 10:58 pm
Full Name: Genachka
Contact:

Re: Backup failure

Post by genachka »

Gostev,

I know this thread is really old, and it was bumped by someone once before, but I can't seem to escape this problem.

I've now tried for several months at a time one each version of Veeam going back to 5.0 (and have tried on 6, 6.1, and now on 6.5). I've applied patches, including now on 6.5 patch #1. But after about a month or two of backups I am back to getting the error: "Error: Client error: All instances of the storage metadat are corrupted.

Veeam is running on a physical Windows 7 64bit workstation that has 8GB RAM, i7 2.8GHz quad core. The OS and Veeam are on different disk from the destination storage. The backup destination is a local dedicated disk (7200 RPM Sata III) that is 3TB, with 1.5TB free. The backup source is currently Esxi 5 (have also tried in the past with Esxi 4 with same issues). When Veeam successfully backed up, it reported that it transferred 28GB and Read 90GB. So there is plenty of space. The backup is set to run on Monday, Wednesday, Friday and Saturday. It is configured as incremental with a synthetic full created on Wednesday and is set to transform previous full backup chains into rollbacks.

After getting the corrupt issue, it obviously no longer is able to make a backup and I've ended up recreating the job each time. What is going on? It seems that it's not a reliable backup solution. Please help!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup failure

Post by Vitaliy S. »

Hello, please include support case ID when posting about any technical issue, as requested when you click New Topic. Thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure

Post by foggy »

genachka wrote:I've now tried for several months at a time one each version of Veeam going back to 5.0 (and have tried on 6, 6.1, and now on 6.5). I've applied patches, including now on 6.5 patch #1. But after about a month or two of backups I am back to getting the error: "Error: Client error: All instances of the storage metadat are corrupted.
From my perspective, as this error persists irrespective of the Veeam B&R version and the same storage for backups was used with every version you tried, then the issue indeed is the storage itself. You could try another storage, as it is advised above, to check this.
genachka wrote:After getting the corrupt issue, it obviously no longer is able to make a backup and I've ended up recreating the job each time.
You do not need to recreate the job every time, it should be sufficient to perform an active full instead (this will start the whole backup chain anew reading the entire VM data from source).
genachka
Novice
Posts: 4
Liked: never
Joined: Mar 04, 2013 10:58 pm
Full Name: Genachka
Contact:

Re: Backup failure

Post by genachka »

foggy wrote: From my perspective, as this error persists irrespective of the Veeam B&R version and the same storage for backups was used with every version you tried, then the issue indeed is the storage itself. You could try another storage, as it is advised above, to check this.
I've tried other backup solutions which do not have this problem with the same storage. That rules out the storage from my perspective.
foggy wrote:After getting the corrupt issue, it obviously no longer is able to make a backup and I've ended up recreating the job each time.
You do not need to recreate the job every time, it should be sufficient to perform an active full instead (this will start the whole backup chain anew reading the entire VM data from source).
If I already have a full backup set to scheduled as part of the job every Wednesday, shouldn't it automatically have corrected itself?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure

Post by foggy »

genachka wrote:If I already have a full backup set to scheduled as part of the job every Wednesday, shouldn't it automatically have corrected itself?
As I can see from your first post, this is a synthetic full (that consolidates previous full backup and a chain of increments into a new full backup), not an active one that grabs the data from the source storage. Synthetic full takes data from the files that you already have on the backup repository - and they are corrupt.
florencemoran
Lurker
Posts: 1
Liked: never
Joined: Mar 05, 2013 5:46 am
Full Name: Florence Moran
Contact:

Re: Backup failure

Post by florencemoran »

i am getting the same error message and tried each and every suggestion here but nothing worked ,any one else ?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure

Post by foggy »

Florence, have you already contacted support team with debug logs for your jobs? Actually the first step to perform when getting any technical issue.
genachka
Novice
Posts: 4
Liked: never
Joined: Mar 04, 2013 10:58 pm
Full Name: Genachka
Contact:

Re: Backup failure

Post by genachka »

foggy wrote:As I can see from your first post, this is a synthetic full (that consolidates previous full backup and a chain of increments into a new full backup), not an active one that grabs the data from the source storage. Synthetic full takes data from the files that you already have on the backup repository - and they are corrupt.
Ah, ok that makes sense. So I guess there's no need to have an active full scheduled (not sure if that's even possible?) unless there is a corruption?

Also, what happens in this case... If I had a backup job set with the synthetic full and incremental with 14 days of rention and after a month it gets corrupted but I leave it running as it for 2 weeks. When I do an active full now, does it automatically delete the previous healthy set and remove the corrupted items? Do they say behind and need manual clean up? If for two weeks it was complaining that it's corrupt, does it override the previous good?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure

Post by foggy »

genachka wrote:So I guess there's no need to have an active full scheduled (not sure if that's even possible?) unless there is a corruption?
Our best practices recommended to perform active full on monthly basis. And that also takes the possibility of corruption into account.
genachka wrote:Also, what happens in this case... If I had a backup job set with the synthetic full and incremental with 14 days of rention and after a month it gets corrupted but I leave it running as it for 2 weeks. When I do an active full now, does it automatically delete the previous healthy set and remove the corrupted items? Do they say behind and need manual clean up? If for two weeks it was complaining that it's corrupt, does it override the previous good?
Retention policy is applied at the end of the successful job run only. So you will always have the specified amount of good restore points available (even if you have months of failing backups). Active full will start the new chain and older restore points will be deleted according to the retention settings.
genachka
Novice
Posts: 4
Liked: never
Joined: Mar 04, 2013 10:58 pm
Full Name: Genachka
Contact:

Re: Backup failure

Post by genachka »

foggy wrote: As I can see from your first post, this is a synthetic full (that consolidates previous full backup and a chain of increments into a new full backup), not an active one that grabs the data from the source storage. Synthetic full takes data from the files that you already have on the backup repository - and they are corrupt.
It's now been a couple of months that I've followed the recommendations and the backup was running fine until the past month. It started to fail again. The error I'm getting is:

Code: Select all

Failed to delete oib '[vmname=<vm name deleted>:creation_time=5/11/2013 1:54:39 AM:alg=Transform:id=d7dfe363-f467-49ad-ad14-d5a76db6653e:point_id=d7457ff2-1978-4a50-bdf2-925298c95605:storage_id=be4aa1d4-a3fb-41f3-a44d-40eba5b81dda]' from 'V:\Veeam Backups\192.168.x.xxx\Weekly - Full - Backup of 192.168.x.xxx vSphere 5\Weekly - Full - Backup of 192.168.x.xxx vSphere 5.02013-05-11T010436.vbk', redo 'V:\Veeam Backups\192.168.x.xxx\Weekly - Full - Backup of 192.168.x
And in details of each VM column it says Job has failed. See logs for details. Which logs are those and where are they? And what does above error mean?

I do have a active full scheduled each wednesday.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure

Post by foggy »

Is there a chance that you run out of the disk space on the target repository? Anyway, please contact our support team directly as it is advised when you have any kind of technical issue. Thanks.
Post Reply

Who is online

Users browsing this forum: K-SOLdier, kyle.shuberg, ybarrap2003 and 164 guests