Discussions related to exporting backups to tape and backing up directly to tape.
Hevros
Veeam Vanguard
Posts: 26
Liked: 1 time
Joined: Jan 17, 2013 5:09 pm
Full Name: Stephen Seagrave
Contact:

Files to tape is painfully slow

Post by Hevros »

Hi,

so as part of our testing I am doing some backup to tape jobs and some file to tape jobs.

the backup copy to tape runs at 100MB/s which is awesome but the file to tape runs a 1MB/s and that is from files on the local Veeam backup server.
when I run it to backup files on another server it run at a few KB/s
it took 8 hours to backup 12GB !!

I am using a HP MSL2024 library with a fibre attached LTO 5 drive
at this stage it is unusable.

I presume this was tested in beta, do you have any throughput number from that testing?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Files to tape is painfully slow

Post by veremin »

Hi, Stephen. Is it possible that a file to tape job, in contrast to, a backup to tape one uses hardware compression by any chance? Also, what are the files/folders that are being copied to a tape media ? .vbks? Thanks.
Hevros
Veeam Vanguard
Posts: 26
Liked: 1 time
Joined: Jan 17, 2013 5:09 pm
Full Name: Stephen Seagrave
Contact:

Re: Files to tape is painfully slow

Post by Hevros »

Hi,

Yes the file to tape has compression enabled. the files are just a mix of file server type data.
excel, word & PDf's and the like. they are not Veeam backup files.

what I want to sort out is if I have a customer with a single physical file server and the rest are VM's can I use Veeam for a flat file backup of the file server.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Files to tape is painfully slow

Post by Dima P. »

Stephen,

Is it possible to do the following comparison:
- Test backup to tape job
- Test file to tape job, on a backup files you used during the backup to tape job

This would be much more honest comparison of speed as in this case you would be transferring to tape the same files? I am thinking that this could be an indexing issue - in case you are pushing too many single files during files to tape.
Thank you.
Hevros
Veeam Vanguard
Posts: 26
Liked: 1 time
Joined: Jan 17, 2013 5:09 pm
Full Name: Stephen Seagrave
Contact:

Re: Files to tape is painfully slow

Post by Hevros »

Hi,

ok so I did a backup copy to tape and a file backup to tape of the same 200GB VBK file
both of them ran at 100MB/s
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Files to tape is painfully slow

Post by Dima P. »

Great,
So it looks like the issue arises only when backing up a bulk of small files using the file to tape job.

From this point I believe you need to contact with our support team, please open a support case and attach Veeam logs + Veeam service logs. Please update this thread with the case number so we could provide deep analysis of this problem. Thank you!
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Files to tape is painfully slow

Post by Dima P. »

Stephen,

I just confirmed with R&D that low transfer rate problems with running file to tape job on a large number of small files is a known issue, and we will look at addressing it in future releases. Thank you and sorry for an inconvenience!
TommyB
Expert
Posts: 123
Liked: 16 times
Joined: Aug 28, 2013 9:46 am
Full Name: Thomas Braun
Location: Germany.Europe.Terra.Sol.Milkyway.Localgroup.Virgo
Contact:

Re: Files to tape is painfully slow

Post by TommyB »

Hi,

I have the same issue here - a backup rate of 450 kb/s over a 1Gbit Network is simply not acceptable for any kind of useful backup (14 Hours for 7.5 GB of data and this is only 18% of the whole job...)

Is there already any date for the next release?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Files to tape is painfully slow

Post by Dima P. »

Tommy,

Unfortunately , there is no ETA at the moment.
TommyB
Expert
Posts: 123
Liked: 16 times
Joined: Aug 28, 2013 9:46 am
Full Name: Thomas Braun
Location: Germany.Europe.Terra.Sol.Milkyway.Localgroup.Virgo
Contact:

Re: Files to tape is painfully slow

Post by TommyB »

I'm not surprised by that (I'm a software developer as well).

First rule: "Never publish a release date" :mrgreen:
horst
Enthusiast
Posts: 38
Liked: 4 times
Joined: Aug 27, 2013 6:55 am
Location: GER
Contact:

Re: Files to tape is painfully slow

Post by horst »

I think, I have the same issue, already written here: http://forums.veeam.com/viewtopic.php?f=29&t=17900
TommyB
Expert
Posts: 123
Liked: 16 times
Joined: Aug 28, 2013 9:46 am
Full Name: Thomas Braun
Location: Germany.Europe.Terra.Sol.Milkyway.Localgroup.Virgo
Contact:

Re: Files to tape is painfully slow

Post by TommyB »

horst wrote:I think, I have the same issue, already written here: http://forums.veeam.com/viewtopic.php?f=29&t=17900
Your backup job is really fast - I have transfer rates of 11 MB/s, even if there are huge files like vhd files from Microsoft server backup.

The backup client on the source machine is idling around 5% processor load, maximum throughput on the LAN is around 40-50 Mbit/s (which is 4-5% on a 1Gbit LAN).

Indeed there is a BIG need for optimization.
sierdw
Lurker
Posts: 1
Liked: never
Joined: Sep 11, 2013 12:41 pm
Full Name: Sierd Weidema
Contact:

Re: Files to tape is painfully slow

Post by sierdw »

Hi all,

As a new VEEAM user (v7.0) I also have some serious File to Tape performance issues. My backup has been running for 16 hours and has only processed 4.3 Gb :?
Image
Backup of VM's is performing normal.

Any tips? Hardware compression is already disabled but doesn't make a difference.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Files to tape is painfully slow

Post by foggy »

Sierd, as stated above, there is a known issue with slow transfer of a large number of smaller files (and you are copying more than 3 million) using file to tape job and this will be addressed in one of the following updates.
richhartdr
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 07, 2010 9:34 pm
Full Name: Don Richhart
Contact:

Re: Files to tape is painfully slow

Post by richhartdr »

Just chiming in...same issue here....backups to tape about 40MB/s, while files to tape like above examples under 1 MB/s...nice to know a fix is coming in the next version!
richhartdr
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 07, 2010 9:34 pm
Full Name: Don Richhart
Contact:

Re: Files to tape is painfully slow

Post by richhartdr »

LOL, guess I'll quit working on my powershell script to get around the files2tape not supporting folder exclusions...

http://forums.veeam.com/viewtopic.php?f ... 324#p88324

...since ~1MB/s is obviously not doable in production...looks like I'm waiting for the next version <fingers crossed>.

PS - Veeam, please get us current documentation for v7 and tape support specifically...this trial and error powershelling is not fun.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Files to tape is painfully slow

Post by Gostev »

foggy wrote:you are copying more than 3 million
Wow o.O, keep in mind we recommend no more than 10000 files per job with 7.0, this is explained right in the dashboard displayed when you click on Tape node. Remember that Veeam tape backup engine is currently optimized for moving small number of large files (such as VM backup files), rather than millions of small files.
richhartdr
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 07, 2010 9:34 pm
Full Name: Don Richhart
Contact:

Re: Files to tape is painfully slow

Post by richhartdr »

Gostev wrote:Remember that Veeam tape backup engine is currently optimized for moving few large files (such as VM backups) to tape, rather than millions of small files.
Would you go as far as to say, Veeam's not optimized to handle small files at all? I have much less (under 1000) files and my performance is no different (<1Mb/s). Yet, similarly, .vbk backups and my backup to tape jobs cruise along at 40MB/s...not great, but that's my tape hardware limitation.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Files to tape is painfully slow

Post by Gostev »

That definitely is correct statement too, but I guess my main point there was that we did not even test the product with millions of files, as again, the primary use case for this initial release of tape support is very different. For example, I don't know if our configuration database queries will scale to millions of files, etc.

Remember, v7 is 1.0 release of tape support aimed at providing ability to copy VM backups to tape. File to Tape is merely a "side" functionality that we were able to deliver almost "for free" in terms of R&D. In fact, this functionality is literally free (available in Free Edition).

We'll be improving this part though, as (unexpectedly) there seems to be strong demand... just cannot have everything at once ;)
richhartdr
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 07, 2010 9:34 pm
Full Name: Don Richhart
Contact:

Re: Files to tape is painfully slow

Post by richhartdr » 2 people like this post

<thumbs up> :) files to tape...got my support guys, I'd LOVE to get to ONE backup software, bye bye Symantec! :)
stjg@sams.ac.uk
Novice
Posts: 3
Liked: never
Joined: Mar 05, 2013 1:46 pm
Full Name: Steve Gontarek
Contact:

Re: Files to tape is painfully slow

Post by stjg@sams.ac.uk »

Yeah, experiencing the same thing - lots of small files just make a Tape job very slow. It would be nice if this could be fixed.

However I am having to use the free version on my archive server (which has the SCSI card and tape libraries attached) just to take advantage of this: my licenced production veeam software is on a Hyper-V virtual machine that uses a backup proxy that has a direct link to the SAN and stores the actual backup files locally : the Veeam machine can't have a directly attached tape library: it would be very useful if Tape jobs could be done through a backup proxy too.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Files to tape is painfully slow

Post by Gostev »

stjg@sams.ac.uk wrote:it would be very useful if Tape jobs could be done through a backup proxy too
We will support tape devices connected to any Windows server in the short term.
joelc67
Novice
Posts: 5
Liked: never
Joined: Sep 16, 2013 2:48 pm
Full Name: Joel Chambers
Contact:

Re: Files to tape is painfully slow

Post by joelc67 »

I have experienced similarly SLOW file to tape performance. We have three large file servers with approximately 100k in files on each environment. If I attempt to backup the file level on each server it goes twelve hours backing up aprox. 120GB of data. If I just backup the VM's themselves it runs a x100 faster. I tried Veeam support. However, the product is not good at backing up 1000s of files. IMHO that makes the file level backups worthless to me. I have had to turn to Backup Exec 2010 (old license) to run file level backups. It is not always practical for me to backup VMs using Veeam 7.0 when archiving files from numerous projects week to week; and having to provide ample space on my backup server for VM's, just so I can later run backup copies to tape for off-site archival purposes is not an efficient use of our server space.

I hope there are plans to change this in a new patch. However, no one could tell me that on Veeam Support. :|
johndoe10110
Influencer
Posts: 22
Liked: 5 times
Joined: Oct 23, 2013 12:49 pm
Full Name: John Dooe
Contact:

Re: Files to tape is painfully slow

Post by johndoe10110 »

Just subscribing to this thread (even though there prob won't be any more updates posted in here). I, like many others, look forward to future updates on Files to Tape.
fabian.muecher
Novice
Posts: 3
Liked: never
Joined: Nov 01, 2013 1:07 pm
Contact:

[MERGED] Best practice w/ large file quantities & sizes

Post by fabian.muecher »

Hello,

I have a client whose file server amounts to 20 TB of productive data. Since they are in advertising I am not able to convince them to scale this down; even archiving everything older than one year would not reduce the size more than 10-15%.
At the moment an agent-based BackupExec job saves the files directly to tape - but since I personally dislike Symantec I would like to switch to Veeam.

A single VM-based backup job seems to be out of the question, since adding an additional storage matching the size of the productivity storage just to be able to use Veeam is difficult to convey.
With the file based job I don't seem to be able to get performance above 5 MB/s when ignoring the 10k file count limit. A performance test (using a filter) with only a couple of hundred iso files gives me about 80-90 MB/s.

Has anyone made similar experiences?
I feel like it seems naive from Veeam to expect file based jobs never to exceed 10.000 files. If they offer this feature, it should be enterprise compatible.

How would you go about backing this up? Split it into multiple jobs to match the available backup disk storage and then replicate to tape? The VMDK are in increments of 2 TB.
Is there any info on Veeam improving file based jobs to handle a realistic amount of files?


Happy for any suggestions.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Files to tape is painfully slow

Post by veremin »

Hi, Fabian, please be aware that this was 1.0 version of tape functionality that was optimized specifically for handling backup files produced by VB&R, thus, there is such a limitation (<10000 files).

However, we consider improving current mechanism in one of the next releases.

Thanks.
Hevros
Veeam Vanguard
Posts: 26
Liked: 1 time
Joined: Jan 17, 2013 5:09 pm
Full Name: Stephen Seagrave
Contact:

Re: Files to tape is painfully slow

Post by Hevros »

Gostev wrote:We'll be improving this part though, as (unexpectedly) there seems to be strong demand... just cannot have everything at once ;)
Just to try to give you an understanding of why there is a demand for this function.

A lot of our customers still have one or two physical servers that they cannot or will not make virtual.
We try to push Veeam to these customers but they do not want two backup solutions, they simply say our current backup can do virtual aswell so why change.
now we all know the good things Veeam has to offer but they do not listen when in involves two solutions.

However if we can use Veeam to do a file copy to tape for the last physical file server or the last physical SQL server via maintinance plan backups copied to tape by veeam, then we can offer the customer one solution and they will then buy Veeam.

So when Veeam say it will be addressed in future releases, is that patch release or version release?
johndoe10110
Influencer
Posts: 22
Liked: 5 times
Joined: Oct 23, 2013 12:49 pm
Full Name: John Dooe
Contact:

Re: Files to tape is painfully slow

Post by johndoe10110 »

I imagine it'll be a version release. It's going to be a while, unfortunately.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Files to tape is painfully slow

Post by veremin »

Yes, most likely the current file to tape mechanism will be reviewed in the next product release. Thanks.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Files to tape is painfully slow

Post by Gostev »

Hevros wrote:Just to try to give you an understanding of why there is a demand for this function.

A lot of our customers still have one or two physical servers that they cannot or will not make virtual.
We try to push Veeam to these customers but they do not want two backup solutions, they simply say our current backup can do virtual aswell so why change.
now we all know the good things Veeam has to offer but they do not listen when in involves two solutions.

However if we can use Veeam to do a file copy to tape for the last physical file server or the last physical SQL server via maintinance plan backups copied to tape by veeam, then we can offer the customer one solution and they will then buy Veeam.
Thank you. Here is what I do not get though. What about the same customers who have no tape and no desire to acquire one, which is approximately 66% of small customers like you mention. Why spend time on the solution that will only work for a smaller part of customers?

Secondly, have you considered using Cloud Edition for file level backup with these type of customers instead?
Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests