Full backup broken - Veeam backup is succesfull

[pizang]

I tried to restore some file from backup. I was unsuccessful and log showed there is a decompression error. I tried to restore that file from last (full) backup and again the same problem. I have run a full backup and after that I was able to recover one month old file. This showed that my full backup was broken even though daily backup was finished with ok status.

I described the situation to Veeam support. They said that Veeam Backup couldn't break backup itself and error occurred during transfer to storage. I am using Dell MD 1000 attached to SAS Perc controller so I doubt it is a source of my problem.

Anyway, I think Veeam should introduce an utility to test backup on backup file level. I know about SureBackup but it is an overkill if you just want to test a backup file. Of course I can recover VMs to test backups but it is laborious.

So, the question is:
- do you trust your backup?
- do you test your backups?
- do you want a test utlity?

[Gostev]

error occurred during transfer to storage. I am using Dell MD 1000 attached to SAS Perc controller so I doubt it is a source of my problem

Trust me, this happens - albeit rarely. Past 2 months I have seen it twice, 4TB and 2TB of production VM data corrupted on production SAN respectively. Veeam Backups where stored elsewhere, so both customers where able to recover 100% of lost VMs successfully (both are participating on these forums btw).

This is exactly why you want your "real" backups off SAN. And this is exactly why you want to test your backups to be sure that you can recover.

I think Veeam should introduce an utility to test backup on backup file level?

OK, can you please elaborate how do you envision such test performed?

I know about SureBackup but it is an overkill if you just want to test a backup file. Of course I can recover VMs to test backups but it is laborious.

This (second sentence) is exactly what SureBackup addresses for you. With SureBackup, zero labor is involved in testing actual recoverability of each and every restore point - for every backup file, and every VM in it.

[pizang]

Trust me, this happens - albeit rarely. Past 2 months I have seen it twice, 4TB and 2TB of production VM data corrupted on production SAN respectively. Veeam Backups where stored elsewhere, so both customers where able to recover 100% of lost VMs successfully (both are participating on these forums btw).

Everyone and everything makes mistakes. That is why we do D2D2D backup. And that is why I cannot agree that "Veeam cannot break you backup". No system is flawless. Neither Veeam, nor Microsoft nor Dell.

OK, can you please elaborate how do you envision such test performed?

Quite simple. When Veeam is restoring data it is doing some CRC check (that's why I have uncompression error). I would like to have option to check this code for whole VBK, just read it from SAN and check for CRC. This is exactly the same process like in WINZIP when you are checking an archive.

This (second sentence) is exactly what SureBackup addresses for you. With SureBackup, zero labor is involved in testing actual recoverability of each and every restore point - for every backup file, and every VM in it.

From what I understood SB will recover every VM and boot it up to check whether it is ok. Of course it is a very good veryfication but it is also a little bit complicated in same cases as I think. What about storage space? I have a file server that has a total of 6TB space of which 2TB are occupied. What if I am out of my free space?

Of course I would use SB for Exchange, SQL etc. but for my file server? I think that backup integrity verification is enough.

[pizang]

I should read more before I have written previous post

Ok, so SB is presenting VM on a virtual NFS to ESX host. That is why it is not necessary to restore whole backup to SAN. From what I understood there is an virtual NFS driver that is reading data from compressed backup as ESX is requesting it. Great. But How can you tell that whole backup is intact? Lets say we are checking file server. System boots up and we are happy, we try to recover some files day after the verification and we have CRC error because this file was not touched during SB verification. Am I right?

[Gostev]

Quite simple. When Veeam is restoring data it is doing some CRC check (that's why I have uncompression error). I would like to have option to check this code for whole VBK, just read it from SAN and check for CRC. This is exactly the same process like in WINZIP when you are checking an archive.

OK, I see. Actually, we have this type of check implemented in the code today. I will check with devs if it can be triggered manually. We did not make it automatic/periodic, because it can take too much time to read many terabytes of backup data (not to mention that it is going to flood the network). May be we should add a way to schedule this too, or ability yo manually run it.

OK, so SB is presenting VM on a virtual NFS to ESX host. That is why it is not necessary to restore whole backup to SAN. From what I understood there is an virtual NFS driver that is reading data from compressed backup as ESX is requesting it. Great. But How can you tell that whole backup is intact? Lets say we are checking file server. System boots up and we are happy, we try to recover some files day after the verification and we have CRC error because this file was not touched during SB verification. Am I right?

You are right, but only for the case where you choose to only perform system boot up test. Remember that SureBackup allows you to additionally run custom test scripts against test VMs. Think about running chkdsk as a part of this test, for example. It will do much more than regular block extraction test, as it would also check NTFS integrity (something you have no insight into with simple backup file integrity check).