Looking for Guidance with Tape

[seadave]

Hello-

I'm moving from CommVault Simpana to Veeam and while the VM backups have been awesome the tape portion is giving me problems.

In Simpana, I used to backup to disk and then an auxillary copy to tape would be run. Simpana was intelligent enough to figure out what had and hadn't been written to tape. It was fairly autonomous and made for an easy process each week when we had to collect tapes and send them offsite. A clean report was run that showed us all tapes used over the previous 7 days. We used this report to select the tapes and send them offsite.

I'm currently running Veeam 8.0.0.917. My server is a R730XD, dual x 12 core procs, 32GB RAM with 16 x 4TB SATA drives in RAID6. I'm getting ~120MB/s data transfer to the storage pool for backups. My tape drive is a TL2000 library with LTO4HH drive connected via the Dell iSCSI to SAS bridge which is direct connected to the server using a dedicated 1Gbs NIC. I used to get ~80MB/s transfers to that device but I'm routinely seeing ~50MB/s with Veeam. (I'm in the process of trying to eliminate the iSCSI to SAS bridge with a SAS HBA, but haven't found the right one yet. I'm thinking that will get me better throughput.

Some of the problems I'm having.

1. Tape jobs are running so long that they are being interrupted by backup jobs starting which then causes the corresponding backup to tape job to fail. Is the job able to recover from this or does it have to start from scratch again. If it fails do I have to return the tapes it used to the free pool? How can I avoid this? Do I need to set a window? Shouldn't the Veeam backup pause if a tape job is using it?
2. It isn't clear to me how I should protect my backup jobs when those are run in Incremental forever mode. Would I run a full backup initially and hold those tapes until a restore is needed and then only make incremental to tape jobs?
3. It isn't clear how I should setup media pools. Currently I have 9 backup jobs defined, a corresponding 9 tape jobs, and one media pool. The tape jobs are listed as secondary targets in their corresponding backup job. Should I have a media pool that corresponds for each backup job? For example:

Backup Job A -> Tape Job A -> Media Pool A
Backup Job B -> Tape Job B -> Media Pool B

or should have media pools related to the type of media?

Media Pool Week 1, Week 2, Week 3, Week 4 or Media Pool A Full, Media Pool A Incr? The media pool usage is kind of tripping me up.

4. Is there some location that lists the variables (and examples) one can use when auto naming media sets such as %date%?

5. Veeam One has me stumped. I cannot for the life of me figure out how to get a simple clean report of all tapes used over the last 7 days (very important for audits!).

6. Again, if a tape fails is Veeam able to recover from that job, or should the tapes be recycled to the free media pool for another try?

7. Do we need to schedule cleaning jobs?

Thanks in advance to any insight others have on how this works and yes I am trying to RTFM

[Dima P.]

Hello seadave,

1. Tape jobs are running so long that they are being interrupted by backup jobs starting which then causes the corresponding backup to tape job to fail. Is the job able to recover from this or does it have to start from scratch again. If it fails do I have to return the tapes it used to the free pool? How can I avoid this? Do I need to set a window? Shouldn't the Veeam backup pause if a tape job is using it?

Need a little background: how you schedule was set up and what was selected as a source for backup to tape job?

2. It isn't clear to me how I should protect my backup jobs when those are run in Incremental forever mode. Would I run a full backup initially and hold those tapes until a restore is needed and then only make incremental to tape jobs?

You need to specify a synthesized full backup creation for backup to tape running pointed to forever incremental backup job. The synthesized full backup file is created directly on tape and requires no disk space. Please, take a look Creating Backup to Tape Jobs > Step 4. Choose Media Pools

Should I have a media pool that corresponds for each backup job or should have media pools related to the type of media? The media pool usage is kind of tripping me up.

It’s up to you, actually. Media pool is a defined set of tape media – you can specify the exact scope of tapes for each media pool based on original backup to disk jobs / media retention / schedule or any other criteria.

Is there some location that lists the variables (and examples) one can use when auto naming media sets such as %date%?

You can find this information by going to the customer media pool. There is a corresponding column in the grid that displays media set name for each used tape.

5. Veeam One has me stumped. I cannot for the life of me figure out how to get a simple clean report of all tapes used over the last 7 days (very important for audits!).

Have you seen this type of report?

6. Again, if a tape fails is Veeam able to recover from that job, or should the tapes be recycled to the free media pool for another try?

Yes, you can restore the backup that was actually transferred to tape during the failed job: let’s say you pointed backup to tape job to three backup to disk jobs. Two of these were disabled (so backup was transferred to tape successfully) and the last job was running (so the backup file was locked by the original job). First two backups could be restored, last one - not.

7. Do we need to schedule cleaning jobs?

Unfortunately, there is no such functionality now.

[seadave]

Ugh. Just lost an hour of replying to this when browser timed out. Thanks for replying to my questions. Let's try this again.

Regarding Schedules

We have 9 VM backup jobs that contain ~5 VMs each grouped based on similarity to maximize dedupe. Those work great. Much faster backups than our older agent based Simpana install. We have another 9 Tape backup jobs. Each one of those jobs uses the corresponding VM backup job as its source. Backup Files...Add...Backup Jobs...Job Name. The VM backup jobs run from 7PM to ~1AM each night. We initially setup the tape jobs to run starting at 5AM, but then noticed due to a 50MB/s throughput limit that the jobs were taking longer than 24 hours and thus failing with the corresponding VM backup job fired off again. This shouldn't be the case. If a tape job is running, the source of that job should not be able to interrupt or fail it until it is finished. That should at least be an option. We are currently using an iSCSI to SAS bridge which is why the throughput is low. We are attempting to get a HBA which should at least 2x the throughput, but 50MB/s is what we have now. Strangely enough, this setup worked fine over the last three years with CommVault. I think part of the problem is the creation of the synthetic full on tape mentioned in the next section. This takes a VERY long time and may not be the best approach.

Regarding synthesized full backup creation...

I have my tape tape jobs configured as shown, but I still don't understand. So each time we run a tape job it creates a synthesized full first and then any incrementals? I guess that makes sense, but how does that work with the "After this job" and "As new backup files appear" options under schedule? I assume it doesn't do this every time with those.

Regarding Media Pools

So do I need to create two tape jobs associated with two media pools (F and INCR) for each VM backup job I want to protect? This seems laborious. Still looking for a variable list that I can use in the media set creation. I'd like each tape to be called for example "Exchange 010815 Tape 1" (In this case the variables would be VM Job Name, Date of Tape Backup, Tape Sequence Number).

Regarding Reports...

I've seen that report but I need something more granular. All tapes used over last 7 days for specific backup jobs that I can select. The Veeam One Reporter UI isn't intuitive so I'm trying to get caught up with that, but not as fast as I would like.

[seadave]

So after fighting with the tape drive today, I discovered the job chaining options which I'm sure most people are using. I now have my daily first VM job set to start at 7PM and then each job is triggered in series after that. Eliminates the need to guess how long a job will take and start the next one at a specific time. Duh. Great feature.

So I did the same thing with my tape jobs. When we got Veeam one thing I was looking forward to is being able to backup my VMs on Sat and Sun also as previously we were only doing the M-T Incr, F Full, with Saturday and Sunday writing to tape. I didn't appreciate that Veeam couldn't backup a VM and write backup files to tape for the same job at the same time because of how it merges the full. Again this should have been obvious to me but it wasn't.

So now I'm back to VM backups M - F and then Tape Jobs Saturday through Monday afternoon before the Monday night VM backups start. This is where I have a problem. It would be nice if you could not only start a job after another job as you can now, but also say start a job after the Friday occurrence of another job. That way my tapes would automatically kick off as soon as the last VM backup is done Friday night. Just a suggestion.

So I'm still left with the problem of throughput. I have ~25TB of Fulls that I need to move offsite each week. At 50MB/s of tape, that isn't going to work if I have a 66 hour window. I need to ditch the SAS to iSCSI bridge and get my throughput up to 100MB/s or more with a direct HBA. I think the max my storage can attain is 125MB/s so moving to LTO5 or 6 won't work unless I'm missing something. I feel like I still am as people must be protecting way more data than I am with Veeam.

Questions:

1. Should I be ignoring incrementals and only shooting Fulls to tape each week?
2. Should I rotate the Fulls that I send to tape per job so that I'm not trying to do all the job Fulls each week? (For example:
Week 1
Job A - Full and Incr
Job B - Incr only
Job C - Incr only

Week 2
Job A - Incr only
Job B - Full and Incr
Job C - Incr only

If I do the above, will those Fulls still be valid for restore after a few weeks? Hard to tell if you are opening a window to data loss. Seems obviously that I might with this method.

What is the best strategy? I've been looking forward to some type of online backup, but I'll need at least a Gbps link to be able to match my tape speed. Seems like it is pretty easy to hit the limit regarding the amount of data you can effectively move offsite either by tape or replication unless you increase spending massively.

[Dima P.]

Since you migrating, it is expected that some questions must be answered and we are here to help. We are getting a flawless feedback in exchange

This shouldn't be the case. If a tape job is running, the source of that job should not be able to interrupt or fail it until it is finished. That should at least be an option.

Agree, but I wonder have you set the backup to tape to retry or it does not work for your case?

So each time we run a tape job it creates a synthesized full first and then any incrementals? I guess that makes sense, but how does that work with the "After this job" and "As new backup files appear" options under schedule? I assume it doesn't do this every time with those.

First, make sure you installed patch 1, where the synthesized setting misbehavior was corrected.

Synthesized full – is a setting that set up a creation of the full back up on the specified day, its not linked to the job schedule. Let’s say you set up ‘After this job schedule’ plus synthesized full on Sunday. Backup to tape would pick increments every time its triggered by schedule and on Sunday it will collect all the existing increments plus one increment will be synthesized to full backup and placed on tape. All further incremental restore points will be linked to this new synthesized full backup.

Bare in mind – synthesized full is working only for forever-incremental backup mode without active full’s or synthetic fulls. Once any of said options enabled in the backup to disk properties synthesized schedule is ignored.

So do I need to create two tape jobs associated with two media pools (F and INCR) for each VM backup job I want to protect? This seems laborious.

No. You can push fulls and increments to the single media pool. You can have one single media pool for all you backup to tape jobs, but that would not provide you ease of media management.

In addition, once the backup to tape job starts – it locks the media pool and the drive, preventing other jobs from using it. So its good to have multiple media pools if you want to utilize you tape drives simultaneously.

Still looking for a variable list that I can use in the media set creation. I'd like each tape to be called for example "Exchange 010815 Tape 1" (In this case the variables would be VM Job Name, Date of Tape Backup, Tape Sequence Number).

You can set a custom name of backup to tape job, media pool, media set and each tape media. I’ll check for the available variables and update this thread.

Should I be ignoring incrementals and only shooting Fulls to tape each week?

I would set separate media pools and keep full backups longer than increments. Something like full retention – 1 month and increments – 1 week. As I’ve said before it’s up to you and yes you can

2. Should I rotate the Fulls that I send to tape per job so that I'm not trying to do all the job Fulls each week? … If I do the above, will those Fulls still be valid for restore after a few weeks?

Yes it will be valid and you can do that without any issues.

[Dima P.]

It seems the only variable we have is %date% and it applies only to the name of the media set. So I count it as a good feature request.

[seadave]

It seems the only variable we have is %date% and it applies only to the name of the media set. So I count it as a good feature request.

Thanks for the reply. That would be great. I'd like to see the following:

%date% - Looks like this exists.
%time% - Time of tape creation
%job% - Would insert the Source Job Name via the Backup Files\Name field.
%seq% - Would insert the tape sequence number
%total% - Would insert the total number of tapes in the job sequence

So you could create a string of: %job% - %date% %time% - Tape %seg% of %total%

The last two could be considered redundant as that info is provided in the Tape Infastructure UI, but still nice to have that info hardcoded to media in the event of a rebuild from a DR situation.

Retrying tape jobs is not a viable option in most cases for us due to the time it takes for them to complete. I really need some logic or job option that is equivalent to "If a tape job is in process, do not attempt to run the source job until the tape job is finished". All of my tape jobs are relying on type "VMware Backup" jobs as their source. I have a one for one setup. 9 VMware backups jobs and 9 corresponding tape jobs. Hope that makes sense.

Another problem we are having now is the tapes are attempting to copy ALL files (all fulls and all incrementals) and not generating a synthetic full. What I want is when my tape jobs run Saturday and Sunday, each job looks at the corresponding VM job it uses as a sources and it synthesizes the latest data to create one full on the tape. I don't want all of the incrementals. I only want to be shipping one copy per week of a full for each job using the latest data available. What am I doing wrong?

I have our backup repository selected as the media pool for full backups. For the "Full Backup Schedule" I have that configured "Weekly on selected days: Saturday" which is the same day that our tapes start, but right now our tapes start Saturday and finish Monday AM. I do have "Process incremental backup files" selected because I'm under the impression that needs to be enabled to create the latest Synthetic full. What am I doing wrong?

Another bug we have noticed is that during very long tape jobs; we had one that ran for over 80 hours due to throughput constrains, (I'll call for lack of a better term) ghost "Offline tapes" will occur. If I look in the Tape Infrastructure UI, I still see the TI>Servers>Libraries>Medium Changer>Drives>Tape Drive, but below that tree (at the same level as Servers/Libraries/Vaults) I also see a section called "Offline tapes" that lists the tape drive without the Medium changer as a parent object. If I wait until the existing job is finished, I'm able to right click on the ghost or orphaned tape drive and choose "Remove Drive" which appears to "fix" the bug. I had two instances of orphaned tapes drives this time (we only have one physical in one library). If I choose to "Remove Drive" while a job is running, the job will fail.

This may be related to us using an iSCSI to SAS bridge. Perhaps the communication between the tape server (in this case the same as the Veeam Backup server) times out due to the data stream from the copy process. Not sure.

[seadave]

So we just got our Dell 6Gbps SAS HBA installed. Was a bit of a pain clearing out old config/drivers and reinstalling, but it is now working. WHAT A DIFFERENCE. Don't underestimate your data throughput needs when moving to Veeam. Previously we were only backing up Data and not entire configurations. Now as we are backing up the entire VM, we are protecting WAY more data. Our old 50MB/s cap caused by using a Dell iSCSI to SAS bridge wasn't going to work. With the Dell HBA card, we are getting ~110MB/s which is awesome. I just shot a job to tape and was super fast. I also disabled the tape compression as suggested in the job dialog box. Finally, I disabled the "Process incrementals" option as it appears that ALL of our data was being copied and not just the most recent Full. Still not sure about that option.

Update: job just finished and that was not the correct option. So if I leave "Process incrementals" enabled, it appears to copy ALL of my restore points. If I disable it, it only copies my oldest Full? Very confused by this.

[seadave]

So I just looked and my latest VBK is from last night as one would expect. I don't even see a VBK from 12/21/14 which is what appears on the tape. Why isn't it putting the 1/14/15 VBK if I tell it to NOT process incrementals???

[Dima P.]

Hello seadave,
I'll let the team know about the variables you suggested, for me it looks like a complete feature description, thank you.

I only want to be shipping one copy per week of a full for each job using the latest data available. What am I doing wrong?

Nothing is wrong – in the current implementation the synthesized full backup will work only with process incrementals checked, so that’s by design. In defense, this is a new feature just released in v8 and we are going to adjust it's behavior based on the customer feedback: now the plan is to support synthesized full backup without increments being included. Don’t have an ETA – will follow up shortly.

WHAT A DIFFERENCE. Don't underestimate your data throughput needs when moving to Veeam. Previously we were only backing up Data and not entire configurations. Now as we are backing up the entire VM, we are protecting WAY more data. Our old 50MB/s cap caused by using a Dell iSCSI to SAS bridge wasn't going to work. With the Dell HBA card, we are getting ~110MB/s which is awesome. I just shot a job to tape and was super fast. I also disabled the tape compression as suggested in the job dialog box.

Indeed, you performance output looks good. Thanks for sharing!

[seadave]

To follow up on this, our backups ran over the weekend. All of them; with the exception of one, only copied incrementals. The aberration did create a synthetic full although it failed during the job and required a restart. For some reason it was not able to eject a tape. After I used the library web interface to do so, I restarted the job and it finished on the second attempt. I'm attempting to figure out why that is. As far as I can tell, all of the jobs are configured the same. Which brings me to another request. A job configuration report. This would be simply in row/column format.

The left most column would be job name, followed by each available option and its setting in the following columns to the right. You should be able to pick which jobs you want to compare. This would make it very easy to spot abnormal setting or configurations when troubleshooting things such as this.

When I look at our VeeamOne install it appears that I can audit changes but this isn't int he right format to show me current settings at a glance like the above suggestion would.

[Dima P.]

seadave,

For some reason it was not able to eject a tape.

Any errors in the session log? It might be worth checking with the support team, but possibly this article is a solution Tape requires return to original slot , but it is already filled. Move tape manually, and retry operation

In addition, I realized that you are using tape library directly connected to the backup server, right? There is experimental (called so because its not set as default in the current patch, but we confirmed that it is safe to use) feature that may increase you performance even more, so feel free to test:

To enable alternative data exchange path, create theDataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set value “2” (no quotes)

Here are the details (Performance enhancements part)

And I forwarded your request to VeeamOne team, thank you for the heads up.

[Shestakov]

Hello seadave,

I`m a Report Analyst of Veeam One product and your request looks interesting. However, showing each job option might be too messy and resource-consuming.

Are you interested in "Job Configuration Report", showing configuration of the chosen job or "Jobs` Configuration Comparison Report", showing only differences between several jobs` configuration?

Thanks!

[seadave]

Shestakov-

Thanks for the reply. What I'm looking for is choice. It seems there are a lot of good template reports built into Veeam one, but they allow minimal configuration options. Many times we want to report on only a few VMs, but our choices are limited to "Backup Infrastructure". Tape Backups is one example. That is all I can choose. What if I want a report that shows the results of only 3 of my 50 VMs. That is my situation due to auditors. I don't want to blind them with excess data. Like the old saying goes "Too much information can be a bad thing". I did not like CommVault's complexity or cost, but I did like the reporting. They made it very easy to generate reports.

1. Pick the server instances you want to report on. (backup job, individual VMs, VMWare Group Folders)
2. Pick the type of job(s) you want to report on. (Backup, Tape, Copy)
3. Pick the events associated with those jobs (success, failure, success with error, etc)
4. Pick the format you would like (PDF, Excel, Word, HTML)
5. Pick who/where it should be sent if scheduled (Email, File Share, etc)

Veeam one has some of that, but it much more limited. Also these options should be consistent across ALL reports. Some reports have the right options but are hard coded to the wrong details, etc. I'm sorry to say also that most of the reports to my eye are not laid out in a logical fashion. Again I'll focus on the "Tape Backups" report. It shows EVERY Veeam Backup that is on tape and the tape that each restore point is on. More often we like to see the date for the tape job not the date for the backup job. For instance we had a report on CommVault that showed all tape jobs for the previous 7 days. That is what we used to help us determine which tapes needed to be sent of each week.

[Shestakov]

Thank you for the comprehensive feedback, seadave.
You are right, there always should be balance between simplicity and complexity, but the golden mean is different for prosumers with huge infrastructure and newbies with just few VMs.
By the way, starting from Veeam One v8, there is a Report Builder which allows to merge data generated by separate custom reports into a single document chopping off superfluous info.

1. Pick the server instances you want to report on. (backup job, individual VMs, VMWare Group Folders)
2. Pick the type of job(s) you want to report on. (Backup, Tape, Copy)
3. Pick the events associated with those jobs (success, failure, success with error, etc)
4. Pick the format you would like (PDF, Excel, Word, HTML)
5. Pick who/where it should be sent if scheduled (Email, File Share, etc)
Veeam one has some of that, but it much more limited.

Could you specify what are you missing in Veeam One from that list? Maybe it is just not that easy to find the option, so I can help you with that.

Some reports have the right options but are hard coded to the wrong details, etc. I'm sorry to say also that most of the reports to my eye are not laid out in a logical fashion.

I`ve got your point regarding Tape Backups report.
It would be great if you share with us which reports also seem imperfect and why. Thanks!

[seadave]

The main issue is that the criteria for report generation is not open across the pre-configured reports. I seem to be finding Report B is targeted at the data I want, but the options to configure it as needed are only available in Report A for example. I'll do some checking and be more specific. All reports should allow you to pick a data range for reporting for example.

Going back to my tape issue and also snapshot problems I was having:

1. Make sure your ESXi hosts are patched current. I found this patch:

http://kb.vmware.com/selfservice/micros ... Id=2096282

Summaries and Symptoms
This patch updates the esx-base VIB to resolve the following issues:
• Attempts to vMotion a virtual machine that is swapping memory to disk due to high memory load might cause the virtual machine to stop responding.
• A virtual machine might stop responding after a successful snapshot consolidation.

Definitely seemed to help in our case.

2. Reduce the number of "Concurrent Tasks" in your proxies. Mine was set at 8 for the virtual (VM in my cluster) proxy (8GB RAM) and 12 for the local proxy (32GB RAM). This was killing my EQL PS6X00 storage. Reducing the number of tasks actually allows the jobs to finish faster because the individual VMDKs are processed at 80MB/s vs 40MB/s. Similar to trying to copy multiple sets of files off a CDROM. Faster to do each in succession than at the same time.
3. Make sure you remove any old backup agent software from your VMs.
4. Make sure latest VMWare tools are installed after updating your hosts. Use VUM for this. MUCH FASTER. It also allows you to configure hosts for auto update on reboot. (You can do this from the VM Settings Options also).
5. Pay attention to VMDK size, LUN free space, and how multi VMDK VMs are provisioned. Assign OS and Data volumes to different SCSI cards if warranted.
6. Regarding tape, make sure your interface is as fast as possible. Direct SAS is best. If using a tape server, you won't get more than the bandwidth of the cable connecting the two systems. We had a Dell iSCSI to SAS bridge. It topped out at 50MB/s over 1Ge. (We have very fast switches). Using 6Gbs SAS HBA directly connected, we now get ~120MB/s which is near top end of our LT4HH drive.

[Shestakov]

Thanks for the feedback, seadave!

Actually giving custom settings for the each predefined report is made by purpose to avoid superfluous information.
If you need think some reports need more options/info, please let us know. So, yes examples will help a lot here.

Note that you can choose desired options using our Custom Reports such as VMware Custom Performance and Custom Infrastructure.
Thanks.

[seadave]

Thanks for the feedback, seadave!

Actually giving custom settings for the each predefined report is made by purpose to avoid superfluous information.
If you need think some reports need more options/info, please let us know. So, yes examples will help a lot here.

Note that you can choose desired options using our Custom Reports such as VMware Custom Performance and Custom Infrastructure.
Thanks.

I've started an updated thread related to the concerns I have with the product located here:

http://forums.veeam.com/tape-f29/sugges ... 26571.html