Files to tape is painfully slow

[Hevros]

Hi,

so as part of our testing I am doing some backup to tape jobs and some file to tape jobs.

the backup copy to tape runs at 100MB/s which is awesome but the file to tape runs a 1MB/s and that is from files on the local Veeam backup server.
when I run it to backup files on another server it run at a few KB/s
it took 8 hours to backup 12GB !!

I am using a HP MSL2024 library with a fibre attached LTO 5 drive
at this stage it is unusable.

I presume this was tested in beta, do you have any throughput number from that testing?

[veremin]

Hi, Stephen. Is it possible that a file to tape job, in contrast to, a backup to tape one uses hardware compression by any chance? Also, what are the files/folders that are being copied to a tape media ? .vbks? Thanks.

[Hevros]

Hi,

Yes the file to tape has compression enabled. the files are just a mix of file server type data.
excel, word & PDf's and the like. they are not Veeam backup files.

what I want to sort out is if I have a customer with a single physical file server and the rest are VM's can I use Veeam for a flat file backup of the file server.

[Dima P.]

Stephen,

Is it possible to do the following comparison:
- Test backup to tape job
- Test file to tape job, on a backup files you used during the backup to tape job

This would be much more honest comparison of speed as in this case you would be transferring to tape the same files? I am thinking that this could be an indexing issue - in case you are pushing too many single files during files to tape.
Thank you.

[Hevros]

Hi,

ok so I did a backup copy to tape and a file backup to tape of the same 200GB VBK file
both of them ran at 100MB/s

[Dima P.]

Great,
So it looks like the issue arises only when backing up a bulk of small files using the file to tape job.

From this point I believe you need to contact with our support team, please open a support case and attach Veeam logs + Veeam service logs. Please update this thread with the case number so we could provide deep analysis of this problem. Thank you!

[Dima P.]

Stephen,

I just confirmed with R&D that low transfer rate problems with running file to tape job on a large number of small files is a known issue, and we will look at addressing it in future releases. Thank you and sorry for an inconvenience!

[TommyB]

Hi,

I have the same issue here - a backup rate of 450 kb/s over a 1Gbit Network is simply not acceptable for any kind of useful backup (14 Hours for 7.5 GB of data and this is only 18% of the whole job...)

Is there already any date for the next release?

[Dima P.]

Tommy,

Unfortunately , there is no ETA at the moment.

[TommyB]

I'm not surprised by that (I'm a software developer as well).

First rule: "Never publish a release date"

[horst]

I think, I have the same issue, already written here: http://forums.veeam.com/viewtopic.php?f=29&t=17900

[TommyB]

I think, I have the same issue, already written here: http://forums.veeam.com/viewtopic.php?f=29&t=17900

Your backup job is really fast - I have transfer rates of 11 MB/s, even if there are huge files like vhd files from Microsoft server backup.

The backup client on the source machine is idling around 5% processor load, maximum throughput on the LAN is around 40-50 Mbit/s (which is 4-5% on a 1Gbit LAN).

Indeed there is a BIG need for optimization.

[sierdw]

Hi all,

As a new VEEAM user (v7.0) I also have some serious File to Tape performance issues. My backup has been running for 16 hours and has only processed 4.3 Gb

Backup of VM's is performing normal.

Any tips? Hardware compression is already disabled but doesn't make a difference.

[foggy]

Sierd, as stated above, there is a known issue with slow transfer of a large number of smaller files (and you are copying more than 3 million) using file to tape job and this will be addressed in one of the following updates.

[richhartdr]

Just chiming in...same issue here....backups to tape about 40MB/s, while files to tape like above examples under 1 MB/s...nice to know a fix is coming in the next version!

[richhartdr]

LOL, guess I'll quit working on my powershell script to get around the files2tape not supporting folder exclusions...

http://forums.veeam.com/viewtopic.php?f ... 324#p88324

...since ~1MB/s is obviously not doable in production...looks like I'm waiting for the next version <fingers crossed>.

PS - Veeam, please get us current documentation for v7 and tape support specifically...this trial and error powershelling is not fun.

[Gostev]

you are copying more than 3 million

Wow o.O, keep in mind we recommend no more than 10000 files per job with 7.0, this is explained right in the dashboard displayed when you click on Tape node. Remember that Veeam tape backup engine is currently optimized for moving small number of large files (such as VM backup files), rather than millions of small files.

[richhartdr]

Remember that Veeam tape backup engine is currently optimized for moving few large files (such as VM backups) to tape, rather than millions of small files.

Would you go as far as to say, Veeam's not optimized to handle small files at all? I have much less (under 1000) files and my performance is no different (<1Mb/s). Yet, similarly, .vbk backups and my backup to tape jobs cruise along at 40MB/s...not great, but that's my tape hardware limitation.

[Gostev]

That definitely is correct statement too, but I guess my main point there was that we did not even test the product with millions of files, as again, the primary use case for this initial release of tape support is very different. For example, I don't know if our configuration database queries will scale to millions of files, etc.

Remember, v7 is 1.0 release of tape support aimed at providing ability to copy VM backups to tape. File to Tape is merely a "side" functionality that we were able to deliver almost "for free" in terms of R&D. In fact, this functionality is literally free (available in Free Edition).

We'll be improving this part though, as (unexpectedly) there seems to be strong demand... just cannot have everything at once

[richhartdr]

<thumbs up> files to tape...got my support guys, I'd LOVE to get to ONE backup software, bye bye Symantec!

[stjg@sams.ac.uk]

Yeah, experiencing the same thing - lots of small files just make a Tape job very slow. It would be nice if this could be fixed.

However I am having to use the free version on my archive server (which has the SCSI card and tape libraries attached) just to take advantage of this: my licenced production veeam software is on a Hyper-V virtual machine that uses a backup proxy that has a direct link to the SAN and stores the actual backup files locally : the Veeam machine can't have a directly attached tape library: it would be very useful if Tape jobs could be done through a backup proxy too.

[Gostev]

it would be very useful if Tape jobs could be done through a backup proxy too

We will support tape devices connected to any Windows server in the short term.

[joelc67]

I have experienced similarly SLOW file to tape performance. We have three large file servers with approximately 100k in files on each environment. If I attempt to backup the file level on each server it goes twelve hours backing up aprox. 120GB of data. If I just backup the VM's themselves it runs a x100 faster. I tried Veeam support. However, the product is not good at backing up 1000s of files. IMHO that makes the file level backups worthless to me. I have had to turn to Backup Exec 2010 (old license) to run file level backups. It is not always practical for me to backup VMs using Veeam 7.0 when archiving files from numerous projects week to week; and having to provide ample space on my backup server for VM's, just so I can later run backup copies to tape for off-site archival purposes is not an efficient use of our server space.

I hope there are plans to change this in a new patch. However, no one could tell me that on Veeam Support.

[johndoe10110]

Just subscribing to this thread (even though there prob won't be any more updates posted in here). I, like many others, look forward to future updates on Files to Tape.

[fabian.muecher]

Hello,

I have a client whose file server amounts to 20 TB of productive data. Since they are in advertising I am not able to convince them to scale this down; even archiving everything older than one year would not reduce the size more than 10-15%.
At the moment an agent-based BackupExec job saves the files directly to tape - but since I personally dislike Symantec I would like to switch to Veeam.

A single VM-based backup job seems to be out of the question, since adding an additional storage matching the size of the productivity storage just to be able to use Veeam is difficult to convey.
With the file based job I don't seem to be able to get performance above 5 MB/s when ignoring the 10k file count limit. A performance test (using a filter) with only a couple of hundred iso files gives me about 80-90 MB/s.

Has anyone made similar experiences?
I feel like it seems naive from Veeam to expect file based jobs never to exceed 10.000 files. If they offer this feature, it should be enterprise compatible.

How would you go about backing this up? Split it into multiple jobs to match the available backup disk storage and then replicate to tape? The VMDK are in increments of 2 TB.
Is there any info on Veeam improving file based jobs to handle a realistic amount of files?


Happy for any suggestions.

[veremin]

Hi, Fabian, please be aware that this was 1.0 version of tape functionality that was optimized specifically for handling backup files produced by VB&R, thus, there is such a limitation (<10000 files).

However, we consider improving current mechanism in one of the next releases.

Thanks.

[Hevros]

We'll be improving this part though, as (unexpectedly) there seems to be strong demand... just cannot have everything at once

Just to try to give you an understanding of why there is a demand for this function.

A lot of our customers still have one or two physical servers that they cannot or will not make virtual.
We try to push Veeam to these customers but they do not want two backup solutions, they simply say our current backup can do virtual aswell so why change.
now we all know the good things Veeam has to offer but they do not listen when in involves two solutions.

However if we can use Veeam to do a file copy to tape for the last physical file server or the last physical SQL server via maintinance plan backups copied to tape by veeam, then we can offer the customer one solution and they will then buy Veeam.

So when Veeam say it will be addressed in future releases, is that patch release or version release?

[johndoe10110]

I imagine it'll be a version release. It's going to be a while, unfortunately.

[veremin]

Yes, most likely the current file to tape mechanism will be reviewed in the next product release. Thanks.

[Gostev]

Just to try to give you an understanding of why there is a demand for this function.

A lot of our customers still have one or two physical servers that they cannot or will not make virtual.
We try to push Veeam to these customers but they do not want two backup solutions, they simply say our current backup can do virtual aswell so why change.
now we all know the good things Veeam has to offer but they do not listen when in involves two solutions.

However if we can use Veeam to do a file copy to tape for the last physical file server or the last physical SQL server via maintinance plan backups copied to tape by veeam, then we can offer the customer one solution and they will then buy Veeam.

Thank you. Here is what I do not get though. What about the same customers who have no tape and no desire to acquire one, which is approximately 66% of small customers like you mention. Why spend time on the solution that will only work for a smaller part of customers?

Secondly, have you considered using Cloud Edition for file level backup with these type of customers instead?