feature request - FileToTape Files Exclusions...and DFS

[richhartdr]

Hey guys...just a feature request for v7 and tape support: Can't believe this is not in there...but it's not. You can't exclude folders or files from a FileToTape backup job... There are file masks for what to INCLUDE...but not what to EXCLUDE.

I ran into this while trying to exclude a hidden dfsrprivate (Microsoft DFS) folder that was under the root directory I'm trying to backup...Veeam did not like that folder at all and would lock up, not to mention it's unneeded as far as backing up anyway. This was a deal breaker so far, but I'm trying to see if I can backup this folder (with the dfsrprivate folder within) some other way...using permissions or anything. If there are any ideas or people have backed up DFS shares, please enlighten me! This said, Veeam will not be working for us to use for tape...sadly...we were excited to possibly get off of using two backup software's (the other being Symantec).

Basically, we DFS our remote office file server network shares back to corporate, and backup and archive all offices from these DFS destinations to tape...was hoping Veeam would allow us to switch...

Thanks for a great product no less though!

[dualdj1]

Second on an Exclusion option.

[veremin]

Thanks, guys, for the feedback; highly-appreciated. We may consider adding this functionality in one of the future releases.

For now, if you want to exclude a given folder from archiving to tape and , meanwhile, you don’t want to go over all files one by one, manually including them, you can create a simple script that will add a new “file to tape" job and handle the manual file selection routine for you. Thanks.

[richhartdr]

@v.Eremin - Good thought, but please don't hold back the details...

I'm browsing the forum PS beginner pages, command reference doc (http://veeampdf.s3.amazonaws.com/guide/ ... QL2ernA%3D), and searching all over the forums and yet fail to see any remove- *folder* cmdlets with ANYTHING related to TAPE jobs...(it's new in v7 is probably the reason) So is it documented somewhere else, or do I try and view all the new commands (and understand them) just from powershell itself? Do you have more specifics on your thoughts for this "simple script" for me being a noob and this being my first powershell (for Veeam)?

Ex. Would you expect something like the following workflow?
1) Use Windows powershell to grab a list of folders, excluding the one I need excluded, and apply to a variable (table?). (no Veeam-specific powershell snap-in required)
2) Use Veeam powershell cmdlets to create (or edit???) a file-to-tape job. (would require Veeam specific TAPE cmdlets, which I can't find)
3) Add the folders from step 1's variable (table?) to the newly created Veeam job. (again, would require Veeam specific TAPE cmdlets)
4) Run the job. (normal Veeam commands and I know how to do this)

...basically, yes, I could create a "script" to backup my VM infrastructure and files on my own, but then why would I buy Veeam or pay maintenance? In fact, the answer of "jsut create a simple script" could probably be used for most forum questions, if the poster was not obligated to offer any ideas how they see this being buiilt in their mind...even if it some top level generics like the workflow above...

...I really wonder how a simple folder/file exclusion option was missed as a development topic when tape was being added...seems a no-brainer...like the file masks, which Veeam DID include.

Thanks for your help though, I know this probably sounds like I don't appreciate your answer...I do....but all to often I see "just create a simple script" posted with nothing to back it. I'm sorry your audience here is not a veteran Veeam PS creator...and as such I require a little more <sigh> hand holding...

Cheers v.Eremin

[richhartdr]

Can someone tell me the Veeam powershell commands (or better, tell me where to learn v7 and tape commands myself!) for the following:

1) List the file/folders, server, and file masks currently in a tape job (hopefully knowing how these are storred, I'll be able to pick an output format from my "Get-ChildItem -Directory -Recurse -Exclude *dfsrprivate*" command so I can edit or add an entry to the tapejob)

2) Edit OR Add/Remove an existing file/folders, server, and file masks currently in a tape job

...I'll construct the rest to, hopefully, get what I need. Thanks in advance...

PS - Not looking for handouts, but not finding a lot of documentation on the v7 powershell tape commands yet...and the built-in get-help docs from the Veeam plug-in are not very helpful either.

[richhartdr]

Getting closer...this is making a new one, but I'll figure out the edit afterwards...

Can I merge these?
Add-VBRTapeFilesJob -Name "test" -Server "servername" -Path A:\folder\folder\folder -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"
...and...
gci -Path \\folder\folder\folder -Exclude *dfsrprivate*,*anyotherfileorfolder*
...something like:
Add-VBRTapeFilesJob -Name "test" -Server "servername" -Path A:\folder\folder\folder | -Exclude *dfsrprivate*,*anyotherfileorfolder* -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"

The -path parameter take pipes, but I'm not fully sure how they work yet...still looking...

[richhartdr]

Boom...well, got this far today...I can create a new files2tape job and populate it with subfolders excluding the dfsrprivate like I need:

Code: Select all

write-host "Starting Veeam Files2Tape Job test"
Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue
$JobName = "test"
$DirList = Get-ChildItem -Directory A:\folder\letter -Exclude *dfsrprivate*,*plusexcludeadditionalitems*
Add-VBRTapeFilesJob -Name $JobName -Server "servername" -Path $DirList -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"

So...remaining questions:
1) Does this look "clean" and is it the best way?
2) Will it cover all files/folders? I noticed that running this gci (Get-ChildItem) command, you had to specifically call out -hidden or -system to display those...but what I have should do the trick since now I'm not relying on Veeam to exclude...
3) How can I just EDIT a job versus creating a new one...I'm guessing making a new one (and deleting the old one) will casue havoc on job history/restores? Implications of this?
4) Best practice for kicking this off (after adding del job, or re-making as edit job, etc.)...from Windows Scheduler? Other?

...thanks for everyone's inspiration on this, my first Veeam powershell! Woot! ;P

[veremin]

Does this look "clean" and is it the best way?

Yep, it does. I was about to create something similar, but, then saw you had done it yourself

Will it cover all files/folders?

As far as I can understand, it will cover all the folders (except hidden one) that are located under A:\folder\latter, as well as, files that are stored in these folders.

How can I just EDIT a job versus creating a new one

This is a known issue that nowadays you can’t change source settings of existing file to tape job via Powershell. This problem is likely to be addressed in one of the upcoming releases.

Best practice for kicking this off (after adding del job, or re-making as edit job, etc.)...from Windows Scheduler?

It’s completely up to you how to initiate start of this job. Thanks.

[richhartdr]

Thank you for your time and verification v.Eremin, it is very much appreciated.

[richhartdr]

Actually, one last question(s) since you answered that I can only make a new job each time. How exactly with this behave in Veeam? Will it keep the same hierarchy in the Files section of Veeam, or will each job make a new tree? And can I delete the previous job each time and retain the files and the information to allow me to restore for the tape (and therefore job) it was originally backed up from? Also, will it retain the ability to run incrementals...or will every time be a full?

Sorry for the last blast, but got to be sure it's in working order doing this custom.

[richhartdr]

Additional question since the original does not seem to scale well with having multiple file paths to backup...(each one being a list of several folders of course, excluding the dfsrprivate). Questions is this:

I have the actual command:

Code: Select all

Add-VBRTapeFilesJob -Name $JobName -Server "This Server" -Path $DirListSum2 -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"

...and when trying to build the $DirLsitSum variable...it just will not take anything I throw at it (object, etc)...minus a single string:

Code: Select all

#Tried this:
$DirList1 = Get-ChildItem -Directory \\folder1 -Exclude *dfsrprivate* #string
$DirList2 = Get-ChildItem -Directory \\folder2 -Exclude *dfsrprivate* #string
$DirList3 = Get-ChildItem -Directory \\folder3 -Exclude *dfsrprivate* #string
$DirListSum = $DirList1,$DirList2,$DirList3 #object

#NEED to convert object back to string before submitting to Veeam's Add-VBRTapeFilesJob cmdlet
$DirListSum2 = $DirListSum | select-object #converted from object to string (I think)

#Using just this variable works fine:
Add-VBRTapeFilesJob -Name $JobName -Server "This Server" -Path $DirList1 -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"
$DirList1

#This also does NOT work:
Add-VBRTapeFilesJob -Name $JobName -Server "This Server" -Path $DirListSum -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"

#I've also tried [string] commands, -join commands, etc.  Help on Add-VBRTapeFilesJob's -Path parameter is not enough to figure this out minus trial and error...which I can't seem to solve.

Also...this might be all for nothing since people are getting <1MB/s on their files2tape jobes anyway...and that's certainly not production doable:
http://forums.veeam.com/viewtopic.php?f=29&t=17802

Thanks!

[richhartdr]

This forum is awesome! Makes me figure out my own solutions...this works, but I swear I tried this earlier!

Code: Select all

$DirList = Get-ChildItem -Directory \\folder1,\\folder2,\\folder3 -Exclude *dfsrprivate*
Add-VBRTapeFilesJob -Name $JobName -Server "This Server" -Path $DirList -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"

I'll post the full code after I clean it up now...but like I said the <1MB/s limitation is a deal breaker unfortunately...so now I wait...

[richhartdr]

Final Code Snippet:

Code: Select all

#Veeam Files_To_Tape using \\folder1, \\folder2, and \\folder3 network paths:

#Enable to pass parameters from the command line.  Not applicable for the way my jobs are different though.
#param ([string]$JobName1,$JobName2 = "$(Read-Host 'Enter the new Veeam files2tape job name. [e.g. Files2Tape Network Backup]')")

#Notification job is starting
write-host "Veeam Files2Tape Job 'Files2Tape Network Backup' Started"

#Add the Veeam PowerShell Snap-in, if it's already loaded, continue silently
Add-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue

#Create Variable(s)
$JobName1 = "test"

#Delete variable contents (for testing)
#Remove-Variable -Name DirList1 -Force

#Get a list of all subdirectories under several paths.  The -Directory parameter also allows for folder exclusions
$DirList1 = Get-ChildItem -Directory \\folder1,\\folder2,\\folder3 -Exclude *dfsrprivate*,*anyotherfolderexclusion*

#Delete any previous job(s) with the same name
Remove-VBRJob -Job $JobName #I'm still researching how to get this to delete more than one, and also to not prompt...seems simple in the help, but I don't have somethign quite right

#Create new job and configure settings
#Add-VBRTapeFilesJob -Name $JobName -Server "This Server" -Path $DirList1 -MediaPool "Full Network Backup" -MediaPoolIncremental "Incremental Network Backup"

#Notification job has completed
write-host "Veeam Files2Tape Job 'Files2Tape Network Backup' Completed"

[richhartdr]

feature request (unless 'm missing it) - no option to select "Eject media once the job finishes" in the Add-VBRTapeFilesJob cmdlet...also several other things you can't set via powershell that's in the GUI as well. I keep hitting caveat after caveat...yuck.

[veremin]

Hi, Don.

Will it keep the same hierarchy in the Files section of Veeam, or will each job make a new tree?

There will be only one files tree under “Files” node.

And can I delete the previous job each time and retain the files and the information to allow me to restore for the tape (and therefore job) it was originally backed up from?

Yes, you will be able to restore files. By default, Veeam Backup & Replication will restore the latest version of files available on tape. If you want to restore files of different (previous) version, you should click "Backup Set" option in restoration wizard.

Also, will it retain the ability to run incrementals...or will every time be a full?

The new job will perform a full run.

(unless 'm missing it)

You’re correct here. Probably, we will add missing commandlets in one of the future releases. So, thanks for your feedback, much appreciated.

Thanks.

[richhartdr]

Many thanks v.Eremin, I appreciate your clear answers, and I'll look forward to the next release, as I'll have to wait until then to try...just too many caveats right now. It's a great first start though! Keep it up guys!

[horst]

Hi,

is it possible to exclude a folder from a copy to tape job?
for examle, I want to copy the data of a whole partition to the tape, with the exception of one folder.
Alternatively, I could also exclude a specific file-extension from the copy job?

[foggy]

Hello, if you are talking about Files to Tape job, then at the Files and Folders step of the wizard you can specify the mask for files in each of the folders that should be archived to tape. Also please see the topic above for considerations regarding excluding folders from archiving. Thanks.

[AJ83]

I would also like to see the exclusion option in the file masking options for file to tape jobs.

[veremin]

Your voice is counted. We do consider adding this functionality in one of the next releases. Thanks.

[fredbloggs]

so, it's now November 2014 and version 8 has been released, yet this is STILL missing as an option.

As mentioned earlier this does make it quite a difficult process to migrate to.

[Offtopic] As for speed, It appears ok'ish for me, about half the speed of backup exec but relatively usable and almost completes my backup in a weekend which I can live with in my situation

[Dima P.]

fredbloggs,

Glad you feel the performance boost! We will continue to work in the optimization direction for our tape functionality for the next releases. Exclude option for file to tape jobs was paused, but it might be included in the next major version.

[corsbj]

I've just implemented Veeam B&R and retired our DPM solution.
The Hyper-V backups are awesome, but the "file to tape" backup is giving me a headache.
It looks like it tries to backup the dfsrprivate folders, which it can't do. The backup speed crawls to about 600kb/s and through resource monitor on the fileserver I can see it's rummaging around in the dfsrprivate folder.

Is there no way to exclude that folder? Is B&R not DFS aware?
We're running 9.0.0.1491.

[Shestakov]

Hello Simon and welcome to the community!
Currently VBR is not DFS aware. We will consider it as a feature request.

[corsbj]

What would you suggest I do to back up our DFS to tape then?

[corsbj]

I've been searching both these forums and tried Google, but I haven't found an answer/post that was recent, so I'm gonna ask.
There doesn't seem to be a way to exclude files/folders, only include. I was sure this was possible, but I guess it's not. This is very much needed!
Every time the backup runs, I get a warning about some files it can't find. They're probably .log files I don't care about anyway, but I can't tell it to ignore them.
I'm saying probably, because it won't tell me which files it can't find. There's nothing in the event log of the source server or the report from the backup.
It should be possible to figure out which files it is.

So to sum up, please count my vote on these improvement suggestions:
File/directory exclusions
Better errormessages/more informative. Maybe an option to enable debug logging or something.

[veremin]

We've got similar requests previously, but the request has always been supplanted by more demanded functionality.

Anyway, thanks for the feedback; appreciated.

[rreed]

Just discovered I need to exclude about 2.1TB worth of folders from a couple of our tape jobs this morning. W/ all love and respect, for what appears to be about three years and/or versions running, could we get this feature bumped up to the top of the list please?

[lobo519]

+ 1 for exclusions. I am backing up a file server for quick tape restore becuase you can't restore file from VM on tape. I keep getting warnings about MS Office temp files that are created when a file is open.

[brianmac86]

Is there a way to filter out a certain file? I am currently using *.* as the file mask, but it would be nice to specify an ignore filter as well.

Thanks