How to REALLY test your dataintegrity

[frankive]

I was doing another, regular disaster recovery test for a customer yesterday.
And as always, with Veeam, all servers is available in very short time, and we invite the customer to do some sporadic test on a few files, database etc.
But while they was testing the database etc. and said "everything looks great as usual!", it made me think, "How do you really now that ALL of your data is 100% what you expect them to be?" "How can you know that nobody has altered your database in small ways, which is almost impossible to notice, but at same make the data extreme WRONG"?

The same night I also watched the french TV-series "Infiniti" where they are docking a satelitte to ISS. Everything looks nice, since they are looking at the data they are expecting, but of course underneath it is altered/changed, and BOOM... the satelitte crash into ISS. Sorry for that spoiler....

Anyway, with regular files we can always confirm this with checksum. (or HOW do we know the hacker has not altered the checksum on the website....?)

As you see; What I am searching for is a good answer if, eventually, my customers starts to ask question how Veeam help us to know the data is not altered. In my opinion it seems like an impossible questions to answer, but I was wondering if anyone on the Veeam Teeam could share some ideas how, or eventually, if Veeam are working on this kind of problem?

[Mildur]

Hi Frank

Build-In test are only possible to a specific level. Not all customers environment are similar.

We have the health check to make sure that the restore point can be used for a restore.
And then we have our automatic SureBackup and SureReplica Jobs to test if the VM in the backup can be powered on, the network is pingable and application were started successfully.
Further you can use Secure Restore in the SureBackup Job to check for pieces of malicious software inside the vm.

To check for something, we don't have build in tests, you can write your own test scripts to customize the test for the environment you want to test. If you have special application or databases which you want to test against known values, this could be a way. Write your own script.

"How do you really now that ALL of your data is 100% what you expect them to be?" "How can you know that nobody has altered your database in small ways, which is almost impossible to notice, but at same make the data extreme WRONG"?

This is also difficult for Veeam to find out such things. I would say, the only one who can know this, is the DB Admin or the enduser who works with the database.
You could use instant db recovery to mount two databases from two different restore points and do the comparison of the content together with the DB Admin.
Of course, this is more a manual test and takes time.

[Steve-nIP]

SureBackup. I guess you could run a script to check against known values.
You could make a pre-backup powershell script that took some important values or files and hashed them, and stored the hash in the backup.
Using SureBackup I guess you could then have that script run in the test environment and compare the previously stored hash.

There's not much more I could even possibly see you could do, because if your source data is considered invalid, the backup, even though it is 100% valid, would then be considered "invalid", despite no errors having occurred in taking or storing the backup. What you're asking for is, as far as I can tell, something that can never exist outside the realms of AI modelling: a way to check every value in a database to make sure it matches what you "expect", in a way which is not rigidly defined.

[jorgedlcruz]

I would totally recommend Veeam Disaster Recovery Orchestrator,
I know that usually this product is aimed to be your best ally to invoke DR, etc. But it is certainly a powerful tool, to make simple checking the Data Integrity.

VDRO has everything that Steve and Fabian mentioned, plus more, you can have different Plans configured from different sources:

• From a VM Replica
• From a VM CDP
• From a NetApp/HPE Storage Snapshot
• From a Backup, or Backup Copy
• More coming soon

Per every plan, you can add your different workloads using vSphere Tags, or VONE business views (VDRO has an embed VONE), and per every plan you can add as many steps, and scripts as you need, not just in the plan, which is great, but as well inside every workload. These scripts can be whatever you imagine, checking a hash, a login to a DB and performing a query, you name it.

VDRO will allow you to automate these steps, have a calendar to even trigger themselves, and have great PDF reporting per every test.

You can use SureBackup as well, as it is included in VBR, but if you have a chance, and time, ping your local SE, set up a demo, or POC with him/her and give VDRO a go, you will not regret it.