Backup failure

[richard_setford]

Hi,

Running Veeam Backup 4.1 and I'm getting this error shortly after the job starts:

'All instances of the storage metadata are corrupted.'

What do I need to do to clear this and get my backups running again?

Thanks,

Richard

[Gostev]

Hi Richard, you should re-create your backup job. Thanks!

[richard_setford]

Thanks Anton.

Out of interest, do you know what causes this problem?

I've just updated from 4.1 to 4.1.2 in case that helps.

[Vitaliy S.]

Richard, we should be able tell you more if you send all the product log files to our technical team. Basically, it seems like there was some kind of a target storage problem.

[Gostev]

Richard, are you backing up to a NAS (network share) or locally?
Does this storage use RAID? Did you perform RAID rebuild recently?

[jkeiser]

so basically what you're saying is that this backup messed up so we should start over and we're on our own? seems to happen too often. garbage.

[Gostev]

so basically what you're saying is that this backup messed up so we should start over and we're on our own?

Hi Justin, no we never said anything like this. We were actually responding to Richard above.
What issue are you experiencing, the same error? What is your support case number?

[damocue]

Hello,

I have the same issue. Is there any option different to rebuild the jobs?
My support ticket is ID#5176860.

Regards,

[Gostev]

Wow, you have bumped a really old thread with that post... these kind of issues certainly do not happen very often.

Basically, this error means incredible corruption of the backup file... there is a perfect army term for that condition, FUBAR. You don't have to rebuild the job, but you most definitely need to initiate active full backup in that job. And prior to that, I would recommend changing the backup repository in the job settings to the one located on a different storage, because these kind of issues can only be caused by very bad underlying storage issues.

Thanks!

[genachka]

Gostev,

I know this thread is really old, and it was bumped by someone once before, but I can't seem to escape this problem.

I've now tried for several months at a time one each version of Veeam going back to 5.0 (and have tried on 6, 6.1, and now on 6.5). I've applied patches, including now on 6.5 patch #1. But after about a month or two of backups I am back to getting the error: "Error: Client error: All instances of the storage metadat are corrupted.

Veeam is running on a physical Windows 7 64bit workstation that has 8GB RAM, i7 2.8GHz quad core. The OS and Veeam are on different disk from the destination storage. The backup destination is a local dedicated disk (7200 RPM Sata III) that is 3TB, with 1.5TB free. The backup source is currently Esxi 5 (have also tried in the past with Esxi 4 with same issues). When Veeam successfully backed up, it reported that it transferred 28GB and Read 90GB. So there is plenty of space. The backup is set to run on Monday, Wednesday, Friday and Saturday. It is configured as incremental with a synthetic full created on Wednesday and is set to transform previous full backup chains into rollbacks.

After getting the corrupt issue, it obviously no longer is able to make a backup and I've ended up recreating the job each time. What is going on? It seems that it's not a reliable backup solution. Please help!

[Vitaliy S.]

Hello, please include support case ID when posting about any technical issue, as requested when you click New Topic. Thanks!

[foggy]

I've now tried for several months at a time one each version of Veeam going back to 5.0 (and have tried on 6, 6.1, and now on 6.5). I've applied patches, including now on 6.5 patch #1. But after about a month or two of backups I am back to getting the error: "Error: Client error: All instances of the storage metadat are corrupted.

From my perspective, as this error persists irrespective of the Veeam B&R version and the same storage for backups was used with every version you tried, then the issue indeed is the storage itself. You could try another storage, as it is advised above, to check this.

After getting the corrupt issue, it obviously no longer is able to make a backup and I've ended up recreating the job each time.

You do not need to recreate the job every time, it should be sufficient to perform an active full instead (this will start the whole backup chain anew reading the entire VM data from source).

[genachka]

From my perspective, as this error persists irrespective of the Veeam B&R version and the same storage for backups was used with every version you tried, then the issue indeed is the storage itself. You could try another storage, as it is advised above, to check this.

I've tried other backup solutions which do not have this problem with the same storage. That rules out the storage from my perspective.

After getting the corrupt issue, it obviously no longer is able to make a backup and I've ended up recreating the job each time.
You do not need to recreate the job every time, it should be sufficient to perform an active full instead (this will start the whole backup chain anew reading the entire VM data from source).

If I already have a full backup set to scheduled as part of the job every Wednesday, shouldn't it automatically have corrected itself?

[foggy]

If I already have a full backup set to scheduled as part of the job every Wednesday, shouldn't it automatically have corrected itself?

As I can see from your first post, this is a synthetic full (that consolidates previous full backup and a chain of increments into a new full backup), not an active one that grabs the data from the source storage. Synthetic full takes data from the files that you already have on the backup repository - and they are corrupt.

[florencemoran]

i am getting the same error message and tried each and every suggestion here but nothing worked ,any one else ?

[foggy]

Florence, have you already contacted support team with debug logs for your jobs? Actually the first step to perform when getting any technical issue.

[genachka]

As I can see from your first post, this is a synthetic full (that consolidates previous full backup and a chain of increments into a new full backup), not an active one that grabs the data from the source storage. Synthetic full takes data from the files that you already have on the backup repository - and they are corrupt.

Ah, ok that makes sense. So I guess there's no need to have an active full scheduled (not sure if that's even possible?) unless there is a corruption?

Also, what happens in this case... If I had a backup job set with the synthetic full and incremental with 14 days of rention and after a month it gets corrupted but I leave it running as it for 2 weeks. When I do an active full now, does it automatically delete the previous healthy set and remove the corrupted items? Do they say behind and need manual clean up? If for two weeks it was complaining that it's corrupt, does it override the previous good?

[foggy]

So I guess there's no need to have an active full scheduled (not sure if that's even possible?) unless there is a corruption?

Our best practices recommended to perform active full on monthly basis. And that also takes the possibility of corruption into account.

Also, what happens in this case... If I had a backup job set with the synthetic full and incremental with 14 days of rention and after a month it gets corrupted but I leave it running as it for 2 weeks. When I do an active full now, does it automatically delete the previous healthy set and remove the corrupted items? Do they say behind and need manual clean up? If for two weeks it was complaining that it's corrupt, does it override the previous good?

Retention policy is applied at the end of the successful job run only. So you will always have the specified amount of good restore points available (even if you have months of failing backups). Active full will start the new chain and older restore points will be deleted according to the retention settings.

[genachka]

As I can see from your first post, this is a synthetic full (that consolidates previous full backup and a chain of increments into a new full backup), not an active one that grabs the data from the source storage. Synthetic full takes data from the files that you already have on the backup repository - and they are corrupt.

It's now been a couple of months that I've followed the recommendations and the backup was running fine until the past month. It started to fail again. The error I'm getting is:

Code: Select all

Failed to delete oib '[vmname=<vm name deleted>:creation_time=5/11/2013 1:54:39 AM:alg=Transform:id=d7dfe363-f467-49ad-ad14-d5a76db6653e:point_id=d7457ff2-1978-4a50-bdf2-925298c95605:storage_id=be4aa1d4-a3fb-41f3-a44d-40eba5b81dda]' from 'V:\Veeam Backups\192.168.x.xxx\Weekly - Full - Backup of 192.168.x.xxx vSphere 5\Weekly - Full - Backup of 192.168.x.xxx vSphere 5.02013-05-11T010436.vbk', redo 'V:\Veeam Backups\192.168.x.xxx\Weekly - Full - Backup of 192.168.x

And in details of each VM column it says Job has failed. See logs for details. Which logs are those and where are they? And what does above error mean?

I do have a active full scheduled each wednesday.

[foggy]

Is there a chance that you run out of the disk space on the target repository? Anyway, please contact our support team directly as it is advised when you have any kind of technical issue. Thanks.