Backup to Tape only last backup with reversed incremental

[sbbots]

Hi, Matt,
You mean sometimes it happens that there are more than two full backups in given directory that haven't been copied yet to tape mediums by file to tape job, and you want to copy only the latest file, right?

Exactly.

With 14 restore points, the backup repository will have a minimum of (3) and a maximum of (4) full backup VBK files in it at any given time. I cannot fit the entire repository on a single LTO-5 tape, which is what I want to use because no one is around on weekends to swap out multiple tapes. So when I run a semi-monthly tape backup using files-to-tape, I have a script that copies only the latest VBK file to another directory and then runs an unscheduled tape backup job which copies that directory to tape. It works just fine, but having an option that would let me schedule a files-to-tape job that goes to the backup repository itself and only copies the latest VBK file to tape would prevent the extra steps of having to copy that latest VBK to a new directory using a script...and then running the tape job.

If this is not possible, it isn't a big deal. My script already does the job fine. Thanks.

[veremin]

Got it. These forums are thoroughly monitored for potential features and improvement. So, thank you for the feedback; much appreciated.

[Isaksson]

Hello.
I am using incremental backup to disk with synthetic full every Saturday and i store 10 restore points. (so i will have more that just 1 full backup on disk)
I have two tape jobs, weekly and monthly just to have different retention policy on the tapes.

I am starting the tape job with powershell and in the script i am checking if it should be weekly or monthly tape job that should start.

I am not sure how Veeam is handling this but when i run the Monthly tape job (1 time a month) will it also duplicate to tape the full disk backup that the weekly job already has duplicated to tape?
Or will it only process the last full backup, and if not, is it possible to just duplicate the last full backup out to tape?

[foggy]

Kristoffer, as mentioned in this thread above, currently it is not possible to copy only latest .vbk file to tape using backup to tape job. A workaround is to use file to tape job in conjunction with the suggested script.

[bungle]

I have been running backup jobs (one VM per Job) and replication jobs (one VM per job), I have now added a tape library (24 slot 2 drive LTO 5). The schedule is full/synthetic backup Friday and incremental rest of week, I now need to copy my backup files to tape in a GF system (the full backup file for each VM created first Friday of month, media set monthly retension 1 year) (the full backup file created for each VM on Friday excluding the first Friday of month, media set weekly retension 1 month).

Because the D2D as been running for the past month with no tape drive connected there are between 5 and 3 full backups on the repositiories, when I now setup the backup to tape job it is going to include every full backup which is about 32 TB of data to go to LTO 5 (22 tapes), so the issue I have is this tape set which will contain the monthly full I want to retain for a year and weekly fulls I want to retain for a month.

What I need is to be able to select individual full backup files e.g. all the ones on disk that were created first Friday of month (monthly full), all the second Friday of month (weekly full) and so on.

I dont want to tie up 22 tapes (nearly half my stock level) for a year just for the February backup, I know going forward I can use the GFS workaround and once the fulls currently on disk are put to tape the next scheduled jobs will only do the last full but at the moment I need a solution to split the full backups into 3-4 tapes sets with different retension levels, any suggestions please.

[Isaksson]

Thanks for the reply.
I really hope that this feature (Process only last .vbk) will be added in the next version of Veeam.
Until then its hard to use the duplicate to Tape feature in a good way.

[MaartenK]

........
Because the D2D as been running for the past month with no tape drive connected there are between 5 and 3 full backups on the repositiories, when I now setup the backup to tape job it is going to include every full backup which is about 32 TB of data to go to LTO 5 (22 tapes), so the issue I have is this tape set which will contain the monthly full I want to retain for a year and weekly fulls I want to retain for a month.

What I need is to be able to select individual full backup files e.g. all the ones on disk that were created first Friday of month (monthly full), all the second Friday of month (weekly full) and so on.
.......

Hi bungle,

As mentioned the script is an option. Or use Reverse Incremental backups instead. If you use this option there will be only one full backup on disk. This reduces space and backup to tape works good, and only backs up 1 *.vbk.

We use an extra Copy Job for a GFS option offsite. 3 week, 3 months etc. But the company also wants an tape backup.

But veeam must make an option in an upcoming patch to only backup the last full *.vbk file.

[Escapo IT]

Or use Reverse Incremental backups instead. If you use this option there will be only one full backup on disk. This reduces space and backup to tape works good, and only backs up 1 *.vbk.

Unless you do an 'Active Full' backup at some point, then there will be 2 VBK files. I gather it is still recommended to periodically perform such a backup, so you will be faced with this scenario on occasion.

[foggy]

I gather it is still recommended to periodically perform such a backup, so you will be faced with this scenario on occasion.

Yes, here are recommendations on the active full frequency for different backup modes.

[bungle]

Okay so I thought I had sorted the backup to tape schedule, I accepted the first time a backup to tape job runs it will backup all vbk files so the first time the weekly backup ran I ended up with a set of 17x LTO5 tapes with upto 4x vbk for each VM.

The following Monday the weekly job ran again this time it contained 1x vbk for each VM and spanned 7x LTO5 tapes this is what I expected.

Again next time it run another set of 7x LTO5 tapes with just the 1x vbk for each VM everything fine so far.

This is where it went wrong, you see for compliance I need to have different retention for the tape jobs so Weekly is 4 weeks, Monthly is 12 months and Yearly is no append, all tape jobs run on Monday morning (the full backups run Friday evening and can run into Sunday evening), so I had two jobs scheduled to run at same time Weekly and Monthly, no problem I just disable the weekly job just before the first Monday of the month (when the monthly job runs) which I did then what happened was the Monthly ran and copied every vbk file to tape not the just latest as I expected, so another set of 17+ tapes which take nearly a week to run.

I can now assume when the next weekly runs instead of 1x vbk for each VM I will end up with 2 it will not register that last weeks full was offloaded to tape as it was under a different tape job and every monthly and yearly job set will span 17+ tapes.

So our calculation for number of tapes we required 7 x 12 = 84 and 7 x 4 = 28 Total tapes 112

Now looks like this 17 x 12 = 204 and 7 x 3 = 21 and 12 Total tapes 237 thousands of pounds in extra media costs

So I either scrap the Monthly and Yearly backup to tape jobs and just run weeklys with a retention period of 4 weeks for all sets and use a tape log to specify when media can be reused, fingers crossed that whoever is bringing the tapes back from the offsite fire safe returns with the right ones.

Or I change the backup to disk to jobs to create less Full backups (reverse incremental not recomended for tape plus offload to tape is 2+ days) so I still need a new full every week otherwise my weekly to tape doesnt work so I reduce the restore points from 30 to 10 this then reduces my capability of the nice quick restores from disk to half.

[veremin]

Hi, Mark, if you're after GFS rotation scheme, you can switch backup mode to reversed incremental and use the workaround that I've proposed in the adjacent thread.

This way, you will have:

1) 7 Daily .vbk.
2) 4 Weekly .vbk.
3) 12 Monthly .vbk.

Thanks.

[bungle]

Thanks Vladimir

I will have a look and see if it would work in my environment.

[veremin]

You're welcome. Feel free to contact us, should any help be needed. Thanks.

[RPS]

Hi, Mark, if you're after GFS rotation scheme, you can switch backup mode to reversed incremental and use the workaround that I've proposed in the adjacent thread.

This way, you will have:

1) 7 Daily .vbk.
2) 4 Weekly .vbk.
3) 12 Monthly .vbk.

Thanks.

Is better GFS rotation support going to be added in future patches? I thought that this rotation was most commonly used and would have been easy to implement right off the bat.

[Dima P.]

Hello Paul,
We are looking in the possibility of implementing easy to use GFS rotation mechanism into VBR console, but there is no ETA on this so far. Thank you for your feedback!

[markhumphreys]

I am currently setting up backups to tape using the configured repositories, I only want to backup the last Full backup. When running th job I get the last 3 Full's being archived to tape, how do i configure it to only backup the last FULL?

[veremin]

Hi, Mark, with the current implementation the whole backup chain will be archived to tape mediums by backup to tape job. You can put into use certain scripts and file to tape job in order to bypass predefined behavior. Thanks.

[DreSelen]

When testing Joerg Riether's script, it failed, because of filepath not found.
I made a few modifications,

Save as TAPE_BACKUP.PS1, run schedule a batch

Code: Select all

# The goal is to only backup the LAST VBK file, and not all the small reverse incrementals
# It creates hardlinks from latest vbk files, erase online tape media, ADD tape to specified pool 
# and start backup job created in VEEAM. Tested with VEEAM 7.0 Patch 3 (7.0.0.839) / server 2008R2 standard
#
# Joerg Riether august 2013 / Mod: Dre Selen march 2014
# 
# NOT for ANY production use!
# only for experimetal purposes in testlabs
# please do NOT use this if you are not 100% sure what you are doing, this will erase your tape media!



# Settings

# Link to backup location. Folder has to be available! 
# This is were your VEEAM BACKUP JOB stores the VBK files (Reversed incremental)
$backuppath = "D:\Backup\VEEAM Backup\"

# Link to Hardlink location. Folder has to be available!
$linkpath = "D:\backup\hardlinks\"

# Name of TAPE Job created in VEEAM. The job should backup the content specified in $linkpath
$JobName = "VeeamTapeJobName"

# Name of Media Pool were tapes should reside
$PoolName = "VeeamTapePoolName"



# Deletes all hardlinks previously created
del $linkpath*.*

# this will find the latest vbk files for you recursively starting from your backup path.

$files = dir $backuppath -filter "*.vbk" -rec
$filteredfiles = $files | group directory | foreach {@($_.group | sort {[datetime]$_.creationtime} -desc)[0]}

# this loop will separate path and name and it will create your hard links

ForEach ($File in $filteredfiles)
        {
        $FilePath = $File.DirectoryName
        $FileName = $File.Name
        cmd /c mklink /H "$linkpath$Filename" "$Filepath\$Filename"
	}

Add-PSSnapin VeeamPSSnapin
# Erase Tape
Get-VBRTapeMedium | Erase-VBRTapeMedium  -Confirm:$false
# Inventory Tape
Get-VBRTapeMedium | Start-VBRTapeInventory
# Move online tape to pool
$tape = Get-VBRTapeMedium | Where-Object {$_.IsOnline -like "True"}
Move-VBRTapeMedium -Medium $tape -MediaPool $PoolName -Confirm:$false
# Start the VEEAM backup job created in VEEAM
Start-VBRJob -Job $JobName -FullBackup

Batch file to schedule
(A seperate batch file created because its easier to schedule than native powershell in windows scheduler)

Thus the backup to tape job does not need to be scheduled!! It will be scheduled by the windows scheduler

Backup.cmd

Code: Select all

cd c:\<LOCATION OF YOUR SCRIPTS>
powershell ".\TAPE_BACKUP.ps1"

Save both files (PS1 and CMD file) in the same folder

[MaartenK]

When testing Joerg Riether's script, it failed, because of filepath not found.
I made a few modifications,

Save as TAPE_BACKUP.PS1, run schedule a batch

Batch file to schedule
(A seperate batch file created because its easier to schedule than native powershell in windows scheduler)

Thus the backup to tape job does not need to be scheduled!! It will be scheduled by the windows scheduler

Backup.cmd

Save both files (PS1 and CMD file) in the same folder

thanks. I will give it a try. If veeam can integrate this script into v8 . It would be great

As mentioned we now use Reverse incrementals and that also works good.

[veremin]

If veeam can integrate this script into v8 . It would be great

I doubt we would integrate custom scripts in our product, but we do consider giving users the ability to copy only latest backup chain/backup in one of the next product releases.

Thanks.

[fcmajor]

Hello

I have a backup job for 2 VMs. These are going to a NAS.
After this job finishes it should do a duplicate to tape job.

My question is, can I manually select one full backup and duplicate it to a tape?
i want to do this because there was an error in the duplicate job and for the last 2 weeks there are no duplicates done.

When I now start the duplicate job veeam wants to duplicate all backup files that were created in these two weeks. I want only the last full backup on the tape.

[foggy]

Jens, currently this is possible using file to tape job only and with the help of some additional scripting.

[itpfp]

So I have been running (successfully) a backup to tape job which should do one thing and one thing only: backup files that were created by a normal daily backup job of vm's.

Until... I had to cancel a running daily backup job because I needed to vacate hyper-v hosts for a reboot. The running daily backup job was at the time locking vm's (because they were backed up) preventing me from vacating the hosts. I therefore cancelled this job, than vacated all the hosts and rebooted them.

I did simply restart the job (retry) but after about one hour, the job was so slow (it went with 50 mb/s or so) I decided to kill it, delete it from backups and simply run a new fulll daily backup.

This is where the linked tape backup job is no longer working, as it is insisting to backup the file created by the killed job, instead of simply taken the one that actually did complete.

Now it is obvious that this problem wouldn't have happened did I not cancel the Original job, but sometimes these things happen and they happen for a reason !

To cut this long story short, there doesn't seem to be a way for the backup to tape to use ONLY THE LAST backup on which it was based, instead of trying to put all sessions of that job to tape.

The Obvious question now would be, is there a way to prevent the tape backup job from still including that particular backup onto tape, and equally the one that ran after it (as I manually backed it up already).

How does the tape backup determine what to backup and is there a way to trick it into simply not including old backups (such as setting a certain state on the session ?).

The end result of this is that the tape job will start and only attempt to backup the daily backup that ran this night. Ignoring the one that ran last night and the one that was cancelled by me.

edit: recreating the job is even more disastrous, it will try to include even more older jobs

it seems this feature needs so work to be usefull !

[veremin]

Currently, it's not possible to instruct backup to tape job to copy only the latest .vbk file. The only way to do it is to utilize combination of file to tape job and hardlinks (or PS scripts). Thanks.

[itpfp]

Thanks,

I saw a script that does this. To be honest, I love powershell and I use Windows taks scheduler. BUT why should I have to, Veeam has a scheduler on board, but apparently offers almost no flexibility or settings.

Case in point the inability to set a job specific notification when using tape jobs, which in the real world is a neccessity.

In our real world, we have a third party that brings tapes offsite on a daily basis. Yet veaam is incapable of sending that third party a simple notification that contains the tape barcode and it's location (which are plain Obvious requirements for such a scenario). Yet wtih normal VM backups, the option to send out a job specific notification does exist.

So here we go, yet another powershell script to alleviate Veeam's inflexibility. Because obviously that third party doesn't need to know what other jobs are running, and even IF we choose to do it this way, the joblog only displays barcode and tape name and not location (which is a painfully Obvious requirement for a tape library, if it was only to prevent that third party from having to search through 48 slots to find the tape).

Now this silly problem, apparently it insist to backup files that haven't yet been backed up to tape, fine I get that. But why in the love of god would Veeam cancel the job if it cannot find the first file it needs to do, instead of simply logging a warning and proceed with the next file ? Sorry but that is just plain stupid.

You didn't answer my question as to how the job to tape job determines which files it need exactly, as that way I can avoid having yet another powershell script outside of Veeam's schedule. As in the rare case this ever happens again, I could simply create an empty file with the name and be done with it, instead of backup up over 1 TB of data which is useless to me, just because veeam insist that it includes cancelled backups in this type of tape backup.

[itpfp]

Anyway, I simply created all the files it ask for and now the backup is running and is backing up the file that I need + a few empty files that I had to make to get the job to start. I presume the next day it will return to normal.

[JeremyS132]

I ended up implementing this but modified the code a bit. I have it stripping out only the jobs I want. I am however having an issue. I am using a .bat in scheduled tasks calling the .ps1. The issue that I am having is the scheduled task is stuck in a Running state. I have to end the job manually. Any thoughts on this?

The Veeam job is running and completing but the scheduled task is not.

[veremin]

Haven't you tried to run PowerShell directly via Windows Scheduler, instead of calling it out from .bat file? Or it's not possible to skip the .bat part, since it is responsible for finding the latest full backup? Thanks.

[JeremyS132]

I haven't. The .bat just calls the .ps1 file to run.

Code: Select all

powershell ".\Test_Backup.ps1" > log.txt

I will try running the powershell script directly and see what happens. The .ps1 is responsible for finding the latest full backup.

For anyone wondering this is the lines of code I added to only get the .vbk's I want to push to tape.

Code: Select all

Start-Sleep -s 5
$files2 = dir $linkpath -filter "*.vbk" -rec
cd $linkpath
Remove-Item $files2 -Exclude Server*

[itpfp]

I did ask the question once more, let's try and ask it again:

How does the backup to tape option determine which backups to include ?

Why does it work a few days, and then all of sudden it runs a backup and not only includes the latest file but a file from two days ago, that already has been written to tape ?

The way how this is released is simply unworkable. As now instead of three tapes, suddenly it requires six tapes, as it is doing files twice for reasons completely unknow.

Can someone please explain how veeam actually determines what to backup and how this can be checked with powershell as I am dying to find out why veeam excercises this behaviour.