Reverse Differential corruption

[jeromep]

Hi,

I would like to know if it is normal that the vbk appears corrupted if the backup is interrupted, for example, by unplugging the USB in the middle of a backup and trying to restore from that disk read only?

Restoring single files worked from one VM but full restore and VM files restoration fails complaining about a missing VRB.

[Gostev]

Hi, sure - since the backup storage has failed during backup, the latest restore point will be "corrupted" (incomplete) until the following incremental pass "fixes" the VBK file. You will still be able to restore from earlier restore points though. Thanks.

[jeromep]

I also tried restoring anterior versions, not the latest but 3 days ago and it failed, it is not production critical we're only testing recovery. In that case would you suggest using incremental+synthetic+reverse?

[jeromep]

Sorry for the post spam but I just tried with a fresh copy of a reverse differential and it fails again, the file system is read only, would that affect VM files restoration?

Edit:
This backup was imported on another PC.

[Gostev]

If the backup run did not complete successfully because the backup storage was pulled out during backup, you cannot restore from the corresponding restore point. This is expected as the data is incomplete. I would suggest to not unplug your backup storage while backing up to it

[jeromep]

Like I said, I tried with a complete backup, now I switched to Incremental so I have all my VRBs, a VBK and VIBs and I still get the same error.

I imported the backup on another PC via Veeam, the media is read only (as it is EXT4), files are readable and single file restore still work, I tried restoring a single .vmx file and it still fails.

[Gostev]

For any technical issues it would be best to work directly with out support, as this would require full logs research. The issues might not be connected at all.

My main point here is that USB storage devices are handled in a special manner by Windows, there is good reason for that "Remove USB device safely" thing, this ensures that write cache is dumped to the storage. So generally I would not recommend pulling the USB storage when there is heavy I/O going.

Although I do like to say that life is too short to remove USB safely

[sbos]

Hi, I'm linking to this thread, althoug my situation is different.

Also I reda this thread:
http://www.veeam.com/forums/viewtopic.p ... ion#p12828
but I think is referred to old version.

I'm working with DR, using backup to a remote site, with linux target.
I set up a reverse incremental with 1 restore point.
Due to bandwidth constraints, they take several hours.
So it is quite possible to have network problems, that take away the connection for a while. Also, we can suppose that the disaster happens in the middle of a backup.

I'm quite disapponted to discover that in this case, I'm not able to recover the VM, (neither in the previous state).
I would expect that the VBR agent keeps the old data available until the new has been received !!!

IS there any setup to fix this?

[Gostev]

I would expect that the VBR agent keeps the old data available until the new has been received !!!

That's exactly what it does, so previous state is always recoverable.

[sbos]

Maybe I did some mistake in the setup, but can assure that this is not the case.
Also I have several situations, where this happened.
Should I have more that one restore point? enable forward incremental?

Should I open a support case?

let me know
TIA
sergio

[Gostev]

As long as you have good restore point produced by successful backup run, you should always be able to restore to it even if the following restore point is corrupted. Basically, full backup file is updated in transactionally, so we always know where the "good" data is. If you can produce situation when this is not the case, please open a support case and let us take a look.

Sometimes disaster can affect more than just the file being processed. For example, we just closed a support case recently were customer could not restore a single restore point after having some storage issues. It appeared that this was caused by RAID5 rebuild process, which was corrupting all backup files on the volume due to bad controller logic (this was reproducible).

Other common problem is backing up to a NAS. Certain low-end NAS devices have questionable performance improvements and do not handle flush command correctly, telling our engine that the data block was written when in fact it was not (still hanging in cache), which of course produces inconsistency and unrecoverable backup files in case of power loss.

So, usually this needs to be investigated on case-by-case basis. Actual engine of transactional VBK updates is solid (have not been changing for a few years now), so typically this comes down to corruption due to the actual storage device. But of course we want to investigate every occasion, to make sure the issue is not with our code.

Thanks!

[sbos]

As long as you have good restore point produced by successful backup run, you should always be able to restore to it even if the following restore point is corrupted. Basically, full backup file is updated in transaction, so we always know where the "good" data is. If you can produce situation when this is not the case, please open a support case and let us take a look.

Ok. Just before opening a case, let me understand how it should work.
Our VBR system is backing up to a linux remote system (WAN @8mbit/s).
The backup is set up for reverse incremental, with a single restore point. (no history needed).

At this point we still did not had a disaster, but we had a network loss, and the backup finished.
So, I publish via NFS the backup and copy the VM to a different datastore.

If I go to see the status of the backup it tells me that some machine is incomplete.
If I try to restore it, the publishing on NFS is ok,
but if I try to move to a production datastore the VM , it fails.
The same happens if I stop manually the backup (the message tells me that "the backup will be left in a non consistent state").

If you tell me I will go on opening a case to the support, my doubt is that I'm missing some point in the config.

thank you.
sergio

[Gostev]

Yes, basically you are trying to restore from the only restore point you have, which is inconsistent and will stay in this state until the next job pass restores it back to last know good state. You want to set the amount of restore points to keep to 2 at least. This way, you will always have consistent and restorable point available, no matter what. Thanks.

[sbos]

That's an hard learned lesson!!!

Will test it ASAP.
thank you

Sergio