HP Data Protector and Veeam - Only Backup Latest .VBK File

[chjones]

Hi all,

We have been using Veeam Backup & Replication for the past 9 months and HP Data Protector for the past 5 years. When we implemented Veeam we went the path of using Veeam's Reversed Incremental Backups as this always creates the latest backup as a a full .vbk file with all previous restore points as .vbr files.

At the end of each backup job in Veeam I call a Powershell script to check if the job was successful and the date and time. If it is after 6pm on a Friday or anytime on Saturday or Sunday a call is made to HP Data Protector to start a Backup Specification. This specification points at the folder where the Veeam Backups are written to for that job and has a filter to only include the .vbk file. This ensures the latest full backup is written to tape each week.

In the coming months we will be installing a new HP StoreOnce 4430. The best practice from Veeam and HP is to use Forward Incremental backups which create a full .vbk, then incremental .vbr files, then another full (either an active full backup or a synthetic full). I've done some initial testing with forward incrementals and this is indeed how it works, and it seems to work quite well.

When we switch to this method I am struggling with how I will ensure the latest .vbk file is written to tape by Data Protector. Since we will keep the last few weeks of restore points, at least, on disk there will be multiple .vbk files in the destination folder on the StoreOnce. Since the name of the the .vbk file has a timestamp in the filename it makes things difficult to ensure only the latest .vbk is offloaded to tape each weekend.

I can't seem to find a way in HP Data Protector to only backup the latest file of a given filetype whenever a backup specification is run. I am struggling to work out how to do this. My only idea, which I have no idea how I would implement is to try and do the following:

1. Check if the Veeam Backup Job was successful via Powershell as a post command
2. Scan the folder for the Veeam Backup and get the file name of the latest .vbk file
3. Modify Data Protector's backup selection list by directly modiying the configuration file on the Cell Manager
4. Make a call to Data Protector to start the job

I am really hoping there is another, easier, way to do this. Has anyone else had success with destaging only the latest .vbk file to tape regularly and automatically?

Thanks,

Chris

[chjones]

So, we got this working if the Backup Repository is a Windows Server. We've configured the Veeam Post Backup Powershell to find the name of the latest .vbk file, and then create a hardlink to the file in a sub-folder called "Tape". I've configured Data Protector to backup the Tape Folder.

I've tested this and it seems to work well.

However, the StoreOnce Appliances are not Windows Servers, so no idea if/how Hardlinks will work on them. I assume they will be running some version of Linux so we could try and use SSH to create symbolic links which Data Protector then backups up.

Im really hoping someone out there has done something similar we could leverage.

[veremin]

Hi, Chris.

I’m wondering whether it’s possible to instruct Data Protector to backup individual file.

Assuming that it’s possible, all you have to do it is to find in a given folder the latest .vbk file and pass it afterwards to HP Data Protector:

In case of Windows OS, the latest .vbk file can be found using the following script:

Code: Select all

$SourcePath = "Path to your repository"
$TargetPath = "Path to location to which you’re willing to copy .vbk-files"
$LastVBK = Get-ChildItem $SourcePath\*.vbk| Sort-Object CreationTime -Descending | Select-Object -First 1| Get-ChildItem –name
#HP Data Protector specific commands

In case of Linux, you can use this one (BASH):

Code: Select all

LatestVBK=$(find /original-location '*.vbk' | sort -rn | head -n1)

Hope this helps.
Thanks.

[chjones]

Hi Vladimir,

Thanks for the reply. That might actually be an option. I've been checking the HP Data Protector commands and there may be a way to do it. At the moment I have created a backup job in Data Protector and the Powershell the Veeam Jobs runs calls that job to start. Rather than this I might be able to instead run a command to backup a specific file.

Using omnib.exe and it's command line options you can specify a filesystem backup and use the filesystem option "-only MatchPattern" to only backup the required data. So, I've already got the code working to determine the latest VBK file, and this works with a UNC path and the StoreOnce will be presented to Veeam as a CIFS\NAS Share so determining the latest VBK should be ok.

My only concern is I'm not sure if I can pass a UNC path to Data Protector with the -only attribute and the name of the VBK, whether Data Protector will take that UNC path. But, it's now on my to-do-list to work out if I can do it.

Thanks for the thought ... I hadn't considered passing all the attributes for the backup to Data Protector, I'd only considered pre-creating a job and then calling that job to run. I'll let you know how I go.

I'm hoping HP release the StoreOnce Virtual Appliance pretty soon (our Account Manager advises it should be released this month) so we can test with.

Thanks.

[veremin]

My only concern is I'm not sure if I can pass a UNC path to Data Protector with the -only attribute and the name of the VBK, whether Data Protector will take that UNC path. But, it's now on my to-do-list to work out if I can do it.

From my perspective, there wouldn’t be any issues with that. However, it would be appreciated, if you update the topic with the results you get. Thanks.

[chjones]

So I worked out the command I can send to Data Protector to only backup the latest VBK file. Its not an elegant solution as you can't load balance across multiple backup devices (we have a tape library with two drives), it just uses which drive is specified last in the command (tried and tested this).

We found that by manually creating a backup job to only backup that single file we could look at the config file that data protector creates and use that as the base for what options to specify in the command. It works. It's not the best solution, but we'll use it for now. The other issue is that the job is listed in the history as "Interactive", it doesn't name the job with the description you can specify in the command line so any alerts, emails, reports, etc all just show that "Interactive" was run. we have multiple Veeam Backup jobs (based on VM Folders) so we will just see a swag of interactive email alerts as jobs complete or fail. It's not ideal, but will work for now.

We're investigating whether we can create a named job in Data Protector like we do at present, then have the post veeam Powershell modify the -only section of the job configuration file after each run with the name of the latest VBK file. Then we just execute the job on the specific days. We have other guys that monitor the status of the email alerts from data protector and it just makes it easier for everyone. I'm not comfortable yet with modifying the config via a powershell, but we've tested manually editing the file and it works as expected which is good, so now to get a foolproof script to do it for us .... famous last words perhaps

[rurouni]

Hi i m very interested about this thread as i m trying myself to make a monthly Full saved on Windows FS exported to tape with DP. I had no clues about how to do it, you gave me some ideas, please don t hesitate to share the scripts.

We re thinking about extracting all the full backup at once with one job only. Cons, we ll have only one tape written at a time in our library

[rurouni]

Hi,
I wrote a script thanks to what you said which takes all the latest vbks from Veeam Repository Folder and creates a Hardlink in a Folder Dedicated for Export to Tape with Dataprotector or another tool.

here it is

Code: Select all

$VeeamRepository = "E:\backups" # Path To Repository
$DPPath = "E:\DP\" # Backup To Tape Path.
Get-ChildItem $VeeamRepository | ForEach-Object { 
    $SourcePath = $_.FullName+"\"
    $LastVBK = Get-ChildItem $SourcePath*.vbk| Sort-Object CreationTime -Descending | Select-Object -First 1| Get-ChildItem –name
    if ($LastVBK){
    $SourceFile = $SourcePath+$LastVBK
    $TargetFile = $DPPath+$LastVBK
    Fsutil hardlink create $TargetFile $SourceFile
    }
}

Feel free to make it better since i'm not a scripting expert.

[veremin]

Glad to hear that our input was useful. Also, thanks for sharing the information with the community; much appreciated.

[chjones]

Hi rurouni,

That code you posted is quite good. That's almost exactly what we came up with too. We then have a Data Protector job that is set to backup the path you have as $DPPath. Then you have your script start the DP Job using omnib.exe and specify the mode as full and you're good to go.

Unfortunately you dont get a nice history in Data Protector of the VMs inside a VBK from several months ago that you wrote to tape. But as long as you name your Veeam Jobs sensibly then you should know which VBK has the VM you require to restore.

HP have released the StoreOnce VSA with a 60 day trial so I deployed a couple of those last week (requires at least 1TB free on a VMFS datastore as it requires storage to be added in 1TB chunks and actually writes a zero in every block, so thin provisioning doesn't help). Unfortunately even though the StoreOnce runs RedHat Enterprise Linux, it's heavily customised to the point we can't remotely connect it via scripts and create mount points.

We just ordered 20 or so StoreOnce Physical Appliances (hopefully start receiving them in the next 4-6 weeks) that we'll be deploying around the country and have all remote offices replicating back to our central datacentres. We're heavy Veeam users in the datacentres and need a way to get the data off to tape. Hardlinks are out, so are reverse incrementals. We had to get creative quickly.

We haven't integrated our script solution with my current Veeam PowerShells but the idea we have, which seems to be the best one (which I was against originally but now I like it), is to re-write the Data Protector Job each weekend to only include the latest VBK file from the NAS Share on the StoreOnce.

1. Run the jobs as Forward Incrementals with an Active Full (did a test of a 30GB VM to the StoreOnce VSA ... Active Full takes 13 mins, and Synthetic Full took 1.5 hours ... I wasn't expecting that much of a difference, but WOW! No synthetics for us then)
2. Finish a Veeam Job and check if it was successful then proceed
3. Check the time the job started (if after 6pm Friday or anytime Saturday or Sunday) then proceed
4. Use the Get-ChildItem and CreationTime code you listed to get the latest VBK for that job
5. Re-write the config file for the Data Protector Job using the -tree option to specify to only backup the latest VBK
6. Write the job config to the \config\server\barlists folder on the Data Protector Cell Manager
6. Run the Data Protector job via the omnib.exe command line

I'll try and post some scripts once we get it all compiled and tidied up. At the moment we're just doing tests here and there and we have code all over the shop. Once I get it all together and streamlined I'll post what we come up with.

[veremin]

Unfortunately you dont get a nice history in Data Protector of the VMs inside a VBK from several months ago that you wrote to tape. But as long as you name your Veeam Jobs sensibly then you should know which VBK has the VM you require to restore.

Another good reason to stick to Veeam Backup and Replication 7, once it’s released.

The upcoming version of VB&R will provide an extensive way of how your tape backup data can be tracked - you will know on which tape medias particular VMs reside on, what medias are required in order to restore certain VM, how copying/restore process should be handled, etc.

Active Full takes 13 mins, and Synthetic Full took 1.5 hours

This post might shed a little bit more light on why this happened.

Thanks.

[rurouni]

I would totally agree with you v.Eremin, if only Veeam 7 could manage tapes directly from the Proxies.

Our Backup infrastructure is designed around a Virtual Backup server and Physical Proxies. The fact that only the Backup Server can access Tapes Libraries is quite annoying.
This role should be delegated to the proxy since in our case they're also hosting the Repositories for Backup flows optimizations.

Until then, We have to stick with DP for Disk to tape backups which is also not perfect since we loose visibility about which VMs are on which Tapes.

[veremin]

I would totally agree with you v.Eremin, if only Veeam 7 could manage tapes directly from the Proxies.

According to the plan, we will support such scenario in the next product release. Thanks.