Backup verification of large number of VM

[pizang]

Hello,

Three days ago I tried to restore data from one of my VMs. Veeam failed with Zlib compression error. I have opened support request but the answer was "Your backup became corrupted". That is strange as they are stored to directly attached Dell MD1000 storage that never showed any problem with any operation.

I am little bit disappointed as I hoped that "verify backup integrity" check box actually works.

Now I have to test the rest of VMs. I have about 30 of them. I have Enterprise version of Veem5.01. How to do this not creating huge number of Virtual Labs etc?

And why, oh why, you cannot give us a simple verification that would do a TRUE integrity check??? (I asked for it a year ago)

[Gostev]

Hi, full integrity check requires reading whole backup file to verify each block, and thus it will take many hours - impossible to do during every backup cycle.

You only need 1 virtual lab since all VMs are verified sequentially (power on, verify, power off). Only VMs with core infrastructure services are powered on constantly while the SureBackup job runs. This is described in details in the sticky FAQ topic.

Actually, your request has been noted and the feature will be implemented, most likely as an optional part of SureBackup job. We just cannot deliver every requested feature immediately for obvious reasons... if we do not stick to original release plans and keep adding features, we would never release.

Thanks!

[pizang]

Gostev,

Thank you for a clarification. I have skimmed through a manual and noticed a warning that you cannot power up more than 3 VMs simultaneously - I thought it work that way.

It solves my problem - back to lab.

[Gostev]

The max number of VMs verified in parallel is configurable in SureBackup job settings, but default 3 is a good number (vPower NFS is not super fast storage anyway to support to many VMs running in parallel). Thanks!

[pizang]

I was thinking about it... as you use kind of synthetic NFS device is it possible to pass verification with broken backup? What I mean is you have VM with 2 disk, system and data. Data is huge 2TB as it is a file server. You create verification job. Even though backup can be broken in disk 2 area I guess that verification can be successful. Am I right?

[Gostev]

If you only limit your verification to verifying successful VM boot up, then of course you are right. This is exactly why we provide ability to use custom scripts as a part of verification process. If you want to check every data bit belonging to specific application, just create your custom verification script specific to the application. For example, for Exchange you can use isinteg or eseutil to validate all mailbox store files and Exchange health.

[pizang]

I ask this question because of SOX compliance where I have to test my backups. That means I have to create custom script - scandisk? - to test it. It will be highly ineffective comparing to checking backup directly from backup server. Only these two technics - surebackup and full backup testing will verify it for SURE.

Do not get me wrong. I think Veeam is a very good product and you do a very good job and you listen to your users but that feature is crucial. I hope you will implement it in next version.

[Gostev]

I ask this question because of SOX compliance where I have to test my backups. That means I have to create custom script - scandisk? - to test it. It will be highly ineffective comparing to checking backup directly from backup server. Only these two technics - surebackup and full backup testing will verify it for SURE.

Actually, I disagree with this statement. If you want to be SURE that your data is recoverable, existing SureBackup functionality with the custom application verification script is the only way to 100% make sure recoverability of your service from backup. Yes, checking backup directly from backup server will be able to spot physical backup data integrity issue, but this does not really remove the necessity of using custom script (which would spot not only physical data integrity issue, but also application level data integrity problems).

Adding full backup file integrity check essentially means adding redundant functionality. It would certainly be helpful for those not using application verification scripts and may add false sense of security, but still generally speaking it is quite useless if we are talking about achieving 100% service recoverability from backup.

Going back to your scandisk example, checking full backup file integrity will not spot NTFS MFT corruption issues or file system problems. You will get green checkbox for your backup file integrity, but your system will still not boot when needed because of corrupted MFT. This would not be the case with scandisk. See what I am talking about?

So, if you are seriously thinking the feature you are requesting will help you with SOX compliance, then you are totally wrong (sorry). However, the good news is that the product already has *today* what you need to achieve SOX compliance, so it is only a matter of starting to use the functionality.

[pizang]

I agree that backup integrity verification will not give you 100% guarantee.

So, what custom script will you use for testing my 4TB file server? For me chkdsk /r should be ok as it test whole disk but I think it will be much slower that integrity check.

I think a lot of users think that if they successfully run fileserver in virtual lab and VM is pingable and has all disks it means that they have proper backup. Wrong! Without checking every file on all virtual disks you cannot be sure that backup is intact. Am I rigth?

[tsightler]

So, what custom script will you use for testing my 4TB file server? For me chkdsk /r should be ok as it test whole disk but I think it will be much slower that integrity check.

I'm curious what fileserver you have that contains SOX data and how you're meeting the SOX requirements of serialization and immutability with the SOX data that is on this server? Most companies that I'm aware that require SOX compliance use a document management/archive systems that meets all of the SOX requirements and then backup those systems using SOX compliant backup/recovery techniques rather than try to apply SOX compliant procedures to every backup in the company. I don't believe that keeping files on a fileserver and backing them up, no matter how many times you verified them, would actually get you anywhere near SOX compliance.

[pizang]

Maybe a fileserver is not a very good example here (in case of SOX complience). I just wanted to state that I do have to test backups as some of them (not exactly fileserver) are in scope of SOX. And I think that it is a good idea to test your backup even though it does not fall into SOX scope.

[tsightler]

Agreed, and the only way to really test backups is to perform restores, thus most companies have some type of restore testing. Even in these cases I doubt very many of them go so far as to actually test each file, they simply restore the systems, perform some spot testing on the files and systems, and if all looks good assume the restore process worked. We use Surebackup to verify the recoverability of our systems, but we still perform periodic restore testing of critical systems with compliance requirements.

[Gostev]

Hi Tom, is there anything specific that SureBackup is missing that requires you to perform such additional testing on specific systems? Is there anything we could add from our side? I would be interested to learn more. Thanks!

[pizang]

Gostev,

I agree with Tom. How will you test a fileserver or a application that has no automatic test abilities (for example an app with a proprietary DB that has no tools to check integrity)?

[Gostev]

Just tell me how you are testing it manually today, and I will tell you how this process can be automated any manual process in IT can be automated.

[tsightler]

Just tell me how you are testing it manually today, and I will tell you how this process can be automated any manual process in IT can be automated.

It doesn't really have anything to do with IT. Testing DR impacts the entire business. We use the same scripts for DR testing that we use for regression testing for major upgrades to our ERP and manufacturing systems. These scripts are performed by subject matter experts in the various business disciplines and verify that our business processes are not negatively impacted when running from the DR environment. Our portion of the testing is simply to restore the systems into the DR environment and get them accessible.

While we have not yet done so, we also plan to begin running from our DR environment for 1 week a year starting in the near future. No better way to test DR and stay practiced than to actually run from your DR environment.

As far as what Veeam can do, well, I just don't think there's anything that is a replacement for actually restoring data because it's all about verifying the process. The process followed by Surebackup is completely different from the processes used for a full system restore so it's just not going to be the same. To verify recoverability you have to verify that your processes for performing the recovery works. Surebackup is a great technology that will help you feel confident that the backups you'll use for that recovery are good, and that's what we use it for, but Surebackup does not verify the full recovery process, nor can it.