Architecting a Veeam Tape Solution

[nunciate]

I initially setup tape out using Veeam when I first upgraded and ended up with a complete failure and had to switch back to my SyncSort software for now. I would really like to use Veeam and dump SyncSort but I think I need some help architecting this so it will work. So here is my setup.

I have a Veeam physical server with 16 Cores and 16Gb of Ram. This server is 4GB fiber connected across to my production storage which consists of NetApp, Hitachi & EMC. I also have another NetApp that I perform all of my backups to. This unit is 8GB fiber connected to the Veeam physical server. I have LUNS setup and presented to the Veeam server so they appear as local drives. All of my backup jobs use those drives as repositories and the Veeam physical server as their proxy. Both the backup NetApp & Veeam server are 4GB fiber connected to a switch which has 4 HP Ultrium LTO6 drives fiber connected through a Quantum i80 tape library. BTW, you can add the Quantum Scalar i80 to your compatibility list.
I have 50 servers being backed up and they are divided between 20 different jobs. These are Windows servers of varying types like Exchange, SQL, and Web etc. I perform incremental backups from Mon-Thu with a synthetic full for most of these on Friday. I have a few that I do Active Fulls simply because the active full takes less time than the synthetic full. I also still have a few jobs that perform reverse incremental but I am working to migrate those away from reverse as that process eats up too many IPOS and kills my NetApp. I am in the process of upgrading my NetApp as I need more IOPS & Space. But for the most part with a bit of scheduling magic I am able to backup all of this data every week with little to no issues.

Now comes the hard part. I need to tape out all of this data. I use SyncSort NSB 4.0 to do that today with NDMP to tape jobs. If you know anything about SyncSort you know that it does not currently support incremental to tape via NDMP jobs. Its file to tape jobs run terribly slow, something like 10-20mb/s on my LTO6 drives. I need to tape out my daily increment jobs every night and store them offsite for 7 days. I also need to tape out my full backups weekly and store those offsite for 30 days. The first Friday of every month I want that weekly job to be stored on tape for 3 years and finally once a year I will store a full backup offsite indefinitely.

The issues I ran into were many. First and foremost I need to utilize all of my tape drives. Currently you can only run 1 job to tape per media set. So my thought is that I have to create a single media pool for every tape drive. Then I have to manually schedule my jobs so that I load balance them across all the media pools through the night. Then every day I come in I would need to move the tapes out of the media pool and into a pool with the appropriate retention period (i.e. Daily, Weekly, Monthly & Yearly). That seems like a lot of work but it is really the only way I can think of to do this. It also seems like I will be using up a lot of tapes as I doubt they will ever get filled up. The other issue I noticed was that when I move tapes from one pool to another I have to catalog them to be able to see what data is on them. I am not even sure if the retention period is really being updated properly when I move them to their holding pools.

Any ideas on this? Surely there are some folks trying to do this today. I’d love to know how others are doing tapes with Veeam with this many jobs and retention periods.

[tsightler]

My first question, why do you have 20 jobs with only 50VMs? That seems to add an unusual amount of complication. My first thought, perhaps to simple, but why not just break that down into 4, roughly equal jobs. Then create a media pool for each one, one for each of the full backups, but just use a singe media pool for incremental backups. Yes this would only use a single drive for the incrementals, but I can't really imagine that would be a problem with only 50 VMs.

Since you're taking your tapes offsite, why even bother tracking their retention in Veeam (at least until we have more advanced GFS retention). I'd probably just set a high retention, or perhaps even manual overwrite, on the full media pools, and manually blank tapes when I move them back in to the library. In other words retention would be manual. You have a pretty simple retention cycle so this should be pretty easy.

I understand that none of this is perfect, but I think if you look to simplify your setup it will be pretty easy. I'm also a little curious about how much data you have total. Let me know your thoughts and we'll keep working through it.

[nunciate]

I do plan to consolidate some of these jobs to get down to fewer over time however there are several that are very large servers. Some of these are VM file servers with large drives attached (multiple 2Tb drives) so they need their own job because the weekly synthetic full takes quite a bit of time to complete. These aren't stored on the production NetApp at this time so I can't simply snap mirror them to my backup NetApp either. One of the jobs grabs all of our Exchange servers and that can get pretty large as well. This is one where we just do a weekly active full along with our Exchange Vaulting server. But I do think I can consolidate down to maybe 10 instead of 20 jobs total. Right now I am battling space constraints until I get my storage upgrades approved so shuffling around large backups may take a while. My Veeam One console shows my repositories have 29.65Tb in full backups and 6.37Tb in incrementals. So that is 36.02Tb. I have about 5Tb left on that NetApp and I have an additional 20Tb on an array that is SAS connected to my Veeam server. I should note that I still have a lot of backup data on that NetApp from my SyncSort backups. I am still in the process of converting SyncSort server backups to Veeam jobs so again that takes up additional space that I have to shuffle around while always keeping 30 days worth of backups on the local disks.

I'll have to think about that manual retention idea. It might work. I would just have to track what tapes are used each day in a spreadsheet or database. Might be an easier solution.

[nunciate]

Just wanted to give some info on where I am with this solution. We have been working with NetApp and pulling performance statistics. I can tell you that a NetApp FAS-2240-4 with a single controller and 24x3Tb 7200RPM drives is just not enough. According to my reports the disks are 100% utilized when running multiple Veeam jobs to it. I also see 100% utilization when trying to perform 4 NDMP SMTAPE jobs with SyncSort.

One other major issue we have run into with NetApp is the block size. I have been working directly with a NetApp engineer on this. From what I understand the NetApp wants to write data in 4k blocks and Veeam wants to write in 256k sequential blocks. Something about the way NetApp creates volumes and lays down data causes some serious issues and over utilization.

I did some experimentation by adding that 20Tb local SAS array and my backups and tape jobs run very fast on that as compared to my NetApp. I will let you all know more as I get closer to a working solution.

[nunciate]

Did some additional testing with my 20Tb array. Backups to disk run much faster to local storage than to my NetApp FC attached. Tape outs are the most significant. I am getting speeds well above 100mb/s from local disk to LTO6 FC drives. Coming from the NetApp I am lucky to get 30mb/s.

As for my media pools I did some work on that as well. The best option I have working right now is 4 weekly pools and 4 daily pools. The weekly pools are set with no expiration and they daily's expire every 7 days. I keep track of my tapes in a spreadsheet for now and when they come back from offsite I manually move them to the free pool. I don't have to do that last step but I like to free them up from their current pool in case they are needed for other jobs. This seems to be working ok for now. I would love to see better cataloging of tapes so I don't have to track things in a spreadsheet. Maybe a report or view that shows each tape's retention period, expiration date and so on. To be fair my SyncSort environment requires a lot of manual intervention when it comes to managing offsite tapes as well. I would really love a simple report that says here are the tapes we just ran with a listing of what was on them and their expirations. That way I can just run that everyday, export those tapes and send them offiste. Then have a report for tapes due to return from offsite so I can compare it with what I actually receive back from each day.

[veremin]

Hi, Alan.

Have you already checked predefined VeeamOne report called “Tape Backups” that provides you with the details regarding backups that have been copied to tape medias? Even though currently it doesn’t have tape expiration date, it still might be helpful for you.

Also, I’ve just passed this information to the responsible person, and we will consider adding this functionality in one of the future Management pack releases. So, thank you for the feedback; much appreciated.

[nunciate]

I am having some issues with Veeam One right now. My backup systems show up in there but have no data associated with them. I tried the tape report but there is no data there as well.
I have a ticket open and it looks like some stuff is starting to come in so I will let you know once I can get the report to return data.

[tsightler]

It also doesn't list the VMs that are on the tape, although it does provide a link that you can click. This isn't really that useful for hard copy documentation. Could make the report a little ugly though, but I think just a list of VMs, similar to the way the tapes are listed, wouldn't be too bad.

[veremin]

As mentioned, Vitaliy has already had the feedback regarding currently existing report. So, chances are, one of the future Management Pack releases will boast with slightly improved report version.

Thanks.