File Integrity: diffing backups

[cyburdine]

So I've searched around a bit on the forums and have not run across this request or need but wanted to know if this was possible. Part of our requirements for the data we host is to be able to detect unwanted file changes. (currently we are looking at something like Tripwire) But as I got to thinking about it, I wondered if Veeam Backup might be able to do the same thing. (or perhaps already does)

The idea is that if I have a backup from yesterday, and I make another backup today, Veeam Backup could show me what files were changed (or even better) compare it to a list of files I want to monitor and send me an alert if those files were changed. It's basically doing a diff between the two backups, and testing the files (or directories) to see if they are in the watch list and send an alert if they are on the watch list.

I figured I'd throw this out there because we have been toying with spending $80k on Tripwire (yet another system to keep track of) and if this could be implemented or if it exists we are willing to seriously consider paying for it!

Does anyone else out there have a need for this or are you solving this issue in a different way?

-justin

[Gostev]

Justin, do you mean comparing files inside image-level backups?

[cyburdine]

yeah, I could see it happening several different ways during or post backup or in a different application altogether.

[Gostev]

This could be possible to script with PowerShell and v4:

- Open Backup1.vbk for guest FLR restore (VM drives will be mounted as volume on Veeam Backup server)
- Enum all required guest files and calculate their hashes with script, store data in file1
- Stop FLR for Backup1.vbk (volume dismounted)
- Open Backup2.vbk for guest FLR restore (VM drives will be mounted as volume on Veeam Backup server)
- Enum all required guest files and calculate their hashes with script, store data in file2
- Stop FLR for Backup2.vbk (volume dismounted)
- Compare file1 and file2 with fc

Not so simple to script, but hey... you save 80K

[cyburdine]

yet again why I love this company... we will indeed give this a shot... now, to get a copy of 4.0

[cyburdine]

Your suggestion worked like a charm. I have not automated the mounting of the vbk, which should be fairly trivial with Autoit. But I ran the process manually using fsum.exe and now have it exporting a list of changed files when there is a failure. Thanks again for the process!

-justin

[Gostev]

Great! Actually, I've been browsing some autotest logs for the latest build, and found that one of the tests is doing exactly this as well (mounting backup, obtaining file hashes and them comparing them with file hashes from original VM). This is done nightly as part of new build testing... so definitely 100% suitable and working approach.

Our autotests are 100% PowerShell based, so you should not need to deal with Autoit either (with v4).

[Gostev]

Here's some pieces of code from that autotest. To clarify, we are using specially generated VM image with *.dat files with same name as their MD5 hash, to make autotest simpler. So basically we are comparing actual file hash with the hash taken from file name. You would need to dump hashes to the output file, and compare the two output files instead.

"Main" script (Backup 4.0 PowerShell Extension required)

Code: Select all

Add-JobBackup MD5 VDDK "san;nbd" "My Computer" "C:\VmBackups" "MD5" -Objects MD5
Start-Job MD5
$rest = Start-FileRestore (Get-RestorePoint MD5 MD5)
CheckDatFile($rest.Drive)
Stop-FileRestore ($rest)

File checker (as I explained, this is specific to our autotest, so just FYI)

Code: Select all

function CheckDatFile ($drive) {
      ls ($drive)*.dat |% {
            $fileName = $_.Name
            $fileMD = Hash-MD5($drive + $fileName)
            $fileRMD = $fileName.Replace(".dat", [System.String]::Empty)
      
            Write-Host "File:" $fileName "  MD5:" $fileMD " Size: " $_.
            if ($fileMD -ieq $fileRMD) {
                  write-host " Pass " -backgroundcolor green -foregroundcolor darkblue
            }
            else {
                  write-host "*FAIL*" -backgroundcolor red
            }
      }
}

Function that calculates MD5 hash of the supplied file

Code: Select all

function Hash-MD5 ($file) {
      $hasher = [System.Security.Cryptography.MD5]::Create()
      $inputStream = New-Object System.IO.StreamReader ($file)
      $hashBytes = $hasher.ComputeHash($inputStream.BaseStream)
      $inputStream.Close()
      $builder = New-Object System.Text.StringBuilder
      $hashBytes | Foreach-Object { [void] $builder.Append($_.ToString("X2")) }
      $output = New-Object PsObject
      $output | Add-Member NoteProperty HashValue ([string]$builder.ToString())
      $output.hashvalue
}

Hope this helps

[Gostev]

Please note that Veeam cmdlets will be renamed slightly in the release build to conform with PowerShell naming specification.