Feature request & my hellish experience with tape

[fix-all-the-things]

Today was a great day. While working with support I got the last bits of information I needed to really understand how tape backups work and now understand why I kept seeing so many really strange problems. Tape backups have been plaguing me with problems for weeks and I came close to abandoning Veeam for my off-site backups.

The feature that really needs to be added is an option to backup every single disk job to tape regardless of whether or not the data has already been written and regardless of whether the data has been written to tape or has changed.

So here's our setup. We have a 2 host + SAN VMware cluster and Veeam is running on an older server with 8 x 2TB drives. Two in RAID1 for Server 2012, 5 in RAID5 for backup storage, and one hot spare assigned to both. Veeam was installed using the local sql server that comes with it because it would be really silly to make our restores dependent on one of the servers that's being backed up. The Veeam server is connected via iSCSI to the cluster.
The total amount of data we're backing up is about 3.6 TB, and backups plus incrementals takes up 5 TB of disk space.
For tape, we have an HP LTO6 8-tape library and rotate tapes daily. We don't set any kind of retention because there's been too many times we would be working remotely and needed to cancel a job but weren't there to swap the tapes. Also, if someone forgets to swap tapes we'd rather a tape got over-written than for a backup to be skipped. We also use crashplan to keep track of certain folders so we're not too worried about losing a 2-day old tape job.
Disk backups are reverse incremental so when the tape job runs it writes a full backup of every machine. (Don't ever run an active full job if you do this! You'll kill your disk space!)

So every once in a while I'd get a tape backup report that would have all green successes but one or more of the jobs would have been skipped and not written to tape. It turns out that Veeam keeps track of which jobs have been already written to tape, and if the data in the disk backup hasn't changed then it won't write it to tape again. This is great if you're doing incremental tape backups or if you never cancel tape jobs, but really really bad in my situation. If I'm working by remote and need to cancel a tape backup for some reason, when I restart it, it will not back up any of the jobs that were done before it was cancelled. This leaves me with an incomplete backup. Currently there's only two ways around this. Either run all the disk backup jobs again and hope they all have some changed data (powered off servers and templates don't have changed data), or make a clone of the tape job and run that then delete it when done. If there were a box I could check that said "SAVE ALL THE THINGS!" then this would never be a problem again.

Okay so a few more things about tapes.

BARCODES! If you're using a robotic library, barcodes are pretty much mandatory. Up until now my tape backups were done through Backup Exec and I've always used hand labeled tapes. Usually I had 4 sets labeled Mon - Thurs, 5 sets labeled Week1 through Week5 that were inserted on whichever Friday it was for the month, and then 12 sets for each month. This made it really easy to put in the proper tapes, really easy to keep backups for a full year, and made it really easy to find the proper tape needed for restores.
If they're manually labeled and you do a manual inventory, you can go to the tape properties and type the name and Veeam will remember it. Every time you switch tapes though you have to do a manual inventory. You then have to manually eject the tape or the backup won't run properly. You used to be able to do an inventory and eject with powershell in Ver 7 but they nuked it in Ver 8 so there's no way to automate this process anymore. This pretty much makes barcodes mandatory.
Also, If your backup runs out of space and asks for a new tape, you have to cancel the job. Veeam cannot run a manual inventory while a tape backup is running. If they're barcoded then it's automatic and it will pick right up and continue.

I also had some major problems with deduplication, since Veeam is running on Server 2012. Ver 7 handled it okay but ver 8 really did not like deduped data. Our tape backups went from 7 hours to over 2 days after the upgrade. After a lot of help from support and playing with all the settings for a few days I ended up just turning dedupe off but didn't know that 2012 doesn't un-dedupe the data when you do this. I had to run the powershell command that reverted all the data on the drive. It actually didn't cause any noticable performance problems except one. It looks like whatever file is being un-deduped gets locked so I had one disk backup job fail. It took 3 days to finish but otherwise the backups ran smoothly. After reverting all the data my tape backup times went back down dramatically.
I also learned that MS has said that there's a 100GB data change limit per day for deduplication. Any more than that and it can be difficult for the server to keep up and I was blowing that 100gb per day pretty easily. There also could have been issues since it was an older server without as much CPU power as newer ones.
In the end, after many hours of trying different settings, I found that Veeam compression was far superior. My disk backup time shrunk by about 33% too. None of the Veeam options that related to deduped data help as much as just turning it off and using Veeam's compression. I lost about 1.6 TB of space on a 7TB partition but it was well worth the performance increase.

One of the more important settings with disk backup optimization was in the backup repository settings. If you edit the repository and go to "Repository" there's a setting for maximum concurrent tasks. You have to test different numbers with your setup to find what's best. I started high at 8 and worked my way down. You want to pay attention to your backup report where it tells you where your bottleneck is. It will tell you the load percents for source, proxy, network, and target. Since the VM cluster is the source I wanted to make sure that number wasn't too high. I also didn't want the network too high either so other traffic wouldn't get throttled, but I did want to try and get the higher numbers to be on the Veeam server.
On my setup I found that if the max concurrent setting was too high my CPU would get killed and my desktop would get very laggy and it would take forever for the program to respond. I found that 8 and 7 were too high and they lagged out the desktop pretty badly, but 6 worked well. That let me know I was maxing out the performance of the Veeam server without killing it completely. My other load numbers for source, proxy, and network looked pretty good so I left it.
Doing this has another advantage too. You can set all of your disk backup jobs to start at the same time and not bother wasting any time staggering the start times around trying to find what works best. At first I was wasting lots of time setting different disk backup jobs to start at 7pm, 8pm, 10pm, etc and now they all just start at 7pm. Only 6 drives are allowed to backup at once and when one is done the next fires up automatically. Super easy! My total disk backup time for 3.6 TB of data went from about 3.5 hours to 1.5 since I'd not only optimized the backup, but I didn't have any idle time with a staggered schedule.

I also tried playing with different compression levels in the disk job settings but since I'm running an older server, optimal was best. If you've got some serious CPU horsepower then higher compression levels would probably be better.

Anyway, there's probably more but that's all I can remember in one sitting. Hopefully this helps some of you.

P.S. If you write a long post here, write it in word or notepad first. If your logon times out before hitting submit you'll lose your entire post. Fortunately for me, in Firefox when I hit back a few times it took me back to where I had written my post so I was able to copy/paste it!

[fix-all-the-things]

You know...I had so much stuff in my brain I totally forgot the main reason for my feature request!

When we're done with a server in VMware I shut it down and do one last backup to Veeam. Once I've verified that it's in Veeam and looks good I delete it from the VMware cluster. There is one disk backup job with the archived servers in it that is also included in the tape backup. The problem is that it only gets written to tape once since Veeam will not write a disk job to tape again if it's already been done and if there is no changed data. Every time we rotate tapes after that, those archived servers are not written to tape.

This means I have to create a new tape backup job with the archived servers in it and write it to a separate tape, then keep that separate tape somewhere outside of my normal rotated tapes. If I add a server to the archive, I have to do this all over again.

Having a switch that says "WRITE ALL THE THINGS!" would really be wonderful to have.

[Dima P.]

Hello James,
Finally, got time to read your post. First of all, I want to say thank you for sharing your experience with the community members - you did a great job!

It turns out that Veeam keeps track of which jobs have been already written to tape, and if the data in the disk backup hasn't changed then it won't write it to tape again.

If you not worried about retention at all – you can create a file to tape job and point it to your repository (as a simple folder). Set tape media set to cyclically overwrite and retention to one day. In my understanding once the tape is overwritten it would pick all the files in the folder again – so you will get all the backup chain fired to the tape

This pretty much makes barcodes mandatory.

In case of tape library usage yes. I mean, you can use it in the blind mode but that would not provide you the efficient management, right?

We also use crashplan to keep track of certain folders so we're not too worried about losing a 2-day old tape job.

May I ask you to check the Veeam Endpoint Backup ? Though it's in the BETA state it does provide the ability to backup files (entire PC or separate volumes) to the local drives or shared folders (backup to the VBR will be available in GA). Then, you could archive the necessary files to tape with VBR. I will be glad to receive you feedback and thoughts here on forums

[lukejf]

Great post James.
I have put off upgrading my customers to version 8 as I have found a few bugs with it myself

1. Writing to tape is much slower. I have two identical sites with identical hardware. Version 8 i get 49MBps to tape Version 7 I'm getting between 75 and upto 90.
2. We have found in version 8 it simply doesn't keep the library inventory up to date, we have to use a powershell script to inventory the library before backup. 7 does not seem to have this issue

[fix-all-the-things]

Hello James,
Finally, got time to read your post. First of all, I want to say thank you for sharing your experience with the community members - you did a great job! If you not worried about retention at all – you can create a file to tape job and point it to your repository (as a simple folder). Set tape media set to cyclically overwrite and retention to one day. In my understanding once the tape is overwritten it would pick all the files in the folder again – so you will get all the backup chain fired to the tape

What I wanted was for the archived servers to be written in the same tape job as everything else. This is not possible though.
This same problem is also what makes cancelling tape job so bad. If a job was written to tape, then the tape backup cancelled and restarted, those jobs get skipped. This makes it impossible to have a full tape backup for any day that the tape job gets cancelled. You have to wait until the next time a disk backup happens, then you can backup to tape.
This is why I really would like to see that feature added that lets me tell Veeam to just back it all up without checking to see if it had been, or if there were any changes. If I'm doing reverse incremental to disk and full backup to tape, this should be the default behavior anyway. It doesn't really make any sense anyway to have a full tape backup job skip jobs for any reason.

In case of tape library usage yes. I mean, you can use it in the blind mode but that would not provide you the efficient management, right?

I've had much better and easier tape management using manually labeled tapes in the past, both with stand-alone and robotic libraries. With manual labels in Backup Exec it was trivial to rotate daily, weekly, and monthly tapes. If today is Monday, then put in the tapes that say "Monday" on them in the drive. If it's the 3rd Friday, then put in the tapes that say "Week 3." With barcodes I'd have to make a chart showing which numbers go in which day and then keep the chart updated if the tapes ever changed. It may not be that much work to do that, but we're so slammed I really don't want to make anything more complex than it already is.

As it is, because barcodes are so mandatory, I've given up making daily/weekly/monthly backups and just do a standard rotation in numerical order. This has dramatically shrunk how far back in time I can go with my tapes. Veeam doesn't handle restores from tape nearly as well as Backup Exec did either, so it's not that big of a deal anyway. If it were possible to restore a single file from tape like Backup Exec can, then it would be a lot more annoying that I can't use manually labeled tapes.
In ver 7, once I figured out how to inventory and eject with scripts it wasn't too bad having non-barcoded tapes. Veeam took away the ability to run an inventory and eject through scripting though, so now barcodes are pretty much mandatory now. It's extremely annoying when an "upgrade" actually removes really important functionality from a program.

May I ask you to check the Veeam Endpoint Backup ? Though it's in the BETA state it does provide the ability to backup files (entire PC or separate volumes) to the local drives or shared folders (backup to the VBR will be available in GA). Then, you could archive the necessary files to tape with VBR. I will be glad to receive you feedback and thoughts here on forums

I don't see any details on how VEB works. Does VEB do change tracking of files? Will it let me choose a file and see all the dates that it was changed going back two years and let me recover from any of the dates listed?
We don't back anything up from workstations anyway since My Docs and Favorites are redirected to the users home folder on the server. There are no sensitive files on any desktop that need backing up since everything is on the file server in VMware. We don't have enough disk space to keep more than about a week of retention points for disk backups, so there's no way to use Veeam to recover files that were lost or damaged further back than that. I used to use my extended tape backup scheme to recover files older than a week, but that's not really a solution anymore since we have to use barcodes, and since we can't restore individual files directly from tape.
I was hoping to get away from Crashplan but right now it's by far the best solution for backup longevity. I really do miss being able to restore a single file from tape though and I really hope that as Veeam's tape backup matures that some of these really important features that Backup Exec has become available.

[fix-all-the-things]

Great post James.
I have put off upgrading my customers to version 8 as I have found a few bugs with it myself

1. Writing to tape is much slower. I have two identical sites with identical hardware. Version 8 i get 49MBps to tape Version 7 I'm getting between 75 and upto 90.
2. We have found in version 8 it simply doesn't keep the library inventory up to date, we have to use a powershell script to inventory the library before backup. 7 does not seem to have this issue

Have you tried the registry hack? Set HKLM\Software\Veeam\Veeam Backup and Replication\DisablePublicIPTrafficEncryption = 1 where DisablePublicIPTrafficEncryption is a new DWORD to be manually set. I'm not sure if this was fixed in Ver 8 patch 1 but you may want to check it.

If you're using Server 2012 and dedupe is turned on, that may cause it too. Ver 7 handled tape backups from a deduped drive just fine but 8 doesn't work very well at all for some people. Disabling it isn't enough either. You have to use powershell to revert the data too.
First check and make sure you have enough space to unoptimize. Open server manager and click "File and Storage Services" on the left, then click "Volumes" just to the right of that. You'll see how much free space the drive has and how much space is being saved.
If saved space is bigger than free space then LEAVE IT and DON'T unoptimize. Free up space first, like by reducing the number of retention points are kept on your disk backups and waiting a few days.

If you have enough space use the command "Start-DedupJob -Volume D: -Type Unoptimization" (assuming D: is the drive with your backups) and it will take a few days but it should make your tape backup faster. Any disk backups that try and use a file that Windows is un-deduping will fail too, but otherwise I didn't see any performance issues.

With your inventory problems, ver 8 pretty much requires barcodes on tapes. If you're not using them then you're going to have a very bad time. The Veeam documentation says you don't have to use them but they are pretty mandatory now.

[justind]

+1 for the "Active Full" option for Backup to Tape jobs. It really would be handy to be able to have the same .vbk file written to multiple tapes without the need to create a file to tape job

[Dima P.]

Hello James,

This is why I really would like to see that feature added that lets me tell Veeam to just back it all up without checking to see if it had been, or if there were any changes. If I'm doing reverse incremental to disk and full backup to tape, this should be the default behavior anyway. It doesn't really make any sense anyway to have a full tape backup job skip jobs for any reason.

Point taken. Will discuss your request with the team.

With barcodes I'd have to make a chart showing which numbers go in which day and then keep the chart updated if the tapes ever changed. It may not be that much work to do that, but we're so slammed I really don't want to make anything more complex than it already is.

Could I advise you to test the Veeam One reporting capabilities with VBR tape functionality, specifically this type of report Tape Media Retention Period

I don't see any details on how VEB works. Does VEB do change tracking of files? Will it let me choose a file and see all the dates that it was changed going back two years and let me recover from any of the dates listed?

In the existing BETA VBR repository destination is disabled, but once the GA version is released, it will be available. You could backup user’s files (without backing up the whole computers) to VBR repository with enabled encryption and then push it to tape for archiving.

[fix-all-the-things]

We don't use retention. If the tapes don't get switched, we'd rather have the current backup over-write data from the night before. If we set retention then it wouldn't run and our last backup would be from the day before.
We don't have the disk space to have long chains of incrementals to disk, and we don't have enough tapes to hold data going back very far in time, so we use Crashplan to restore files that were changed or deleted from last month, 6 months ago, or whenever. It would take an enormous amount of disk space to use Veeam to hold restores for more than about two weeks, while Crashplan can do it in a fraction of the space.

With VEB, when I've got a chance I might test it on roaming laptops. It might be useful for keeping Outlook archives on the local machine instead of in the user's home folder on the network.

[Dima P.]

James,
Thank you, looking forward to get a feedback from you regarding VEB then!