Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
sentenza
Novice
Posts: 5
Liked: 3 times
Joined: May 27, 2015 7:41 pm
Full Name: Olivier Tarnus
Contact:

Feature request: parallel tape jobs to same media pool

Post by sentenza »

Dear Veeam community,

Here is my "official" feature request: Authorize multiple jobs to request tapes from the same media pool. This comes after weeks of investigation with support (#00850675), and seems the minimal requirement to support "big" tape jobs, ie: more than 12TB in less than 24 hours.

Problem: When a job starts, it locks the media pool and will only release it when finished. Another job that could run on another drive in the same library can not request tapes from the same media pool until first jobs is finished, greatly limiting the throughput to only one drive. The only alternative is to add more media pools, adding far too much complexity and fragmentation in the library.

Why is this needed: As Veeam is not implementing tape multi-streaming (ie: writing the same job to multiple tapes in parallel), the only alternative for "big data volumes" is to run multiple jobs in parallel on individual drives. Managing more jobs already adds complexity, but fragmenting the media pools decrease efficiency dramatically and totally "disconnect" the pools from what they should be (ie: Daily, Weekly, Monthly...).

Expected result: A job should not lock the media pool but only the tape it is currently using. Other jobs should be able to claim another (free) tape in the same pool.

My experience on this: I have totally changed my strategy because of tape issues...

6 month ago, I had 3-4 jobs backing up 400 VMs to 2 big repositories. It was easy to manage and error prone: basically we did backup everything, and were sure no VM was left away. And then we started to implement tape "for real"...

Fast forward 6 months: I have 15 jobs, backing up to 7-8 repositories. We have also added copy jobs and their own repositories as a "buffer", because tape jobs were locking the main repositories thus preventing production backups to run. Everything is now more complex, our storage usage as exploded and it definitely requires more time to manage, but my VM count and volume increased only by 5% and this makes it hard to explain to management... And still can't come up with a viable tape strategy...

We need a solution, as 12TB in 24 hours (basically 1 LTO6 drive full speed) does not seem "that" big in my opinion and set the bar quite low for tape users.

Olivier
justind
Enthusiast
Posts: 32
Liked: 4 times
Joined: Nov 09, 2014 4:20 am
Location: Australia
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by justind »

+1 vote from me

Being able to share media pools would be great. Currently I have 18 Media pools that I could reduce down to 3 with shared pools
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

Hello guys,
I can’t provide you any details on the next major version feature set, however, I want to encourage you to wait for it as the issue you described will be resolved with some additional really cool new functionality :wink:
robpare
Lurker
Posts: 1
Liked: never
Joined: Sep 16, 2014 9:57 am
Full Name: Rob
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by robpare »

+1 from me as well and I look forward to see what comes in the next version.

We've just started setting up tape media pools and have been caught out by media pools being locked when in use by one job. Sentenza captures the problem more perfectly than I think I'd have done.

We can come up with workarounds for this but I'd much rather go back to having a small number of media pools - we might get away with just two for weekly/monthly if the lock happened per tape rather than at the pool level.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

So you would like to point multiple tape jobs to the singe media pool and let this tape jobs run in parallel, right?
gosnells
Influencer
Posts: 17
Liked: 2 times
Joined: Jul 23, 2012 4:28 am
Full Name: Mike Smith
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by gosnells »

+1
being able to fully utilise an LTO6 tape library with two drives is a must. Surely tape support was added for / aimed at enterprise users, and most enterprise users do not muck around with a single drive. Splitting the tape backup jobs into multiple media pools to get concurrency is crazy.
bvrz
Lurker
Posts: 1
Liked: never
Joined: Jun 15, 2015 8:40 am
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by bvrz »

hello guys,

when comes the next major version Feature with these addin?
thanks
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

bvrz,
There is no ETA at the moment.

Mike,
Are you ok with a possible tape media overrun?
Gerwin1
Novice
Posts: 4
Liked: never
Joined: Jul 16, 2013 9:34 am
Full Name: Gerwin van de Bunt
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Gerwin1 »

+1 for me to.
I have a tape drive with 3 LTO 6 drives. Now we have 3 media pools. When I need 100 GB additional tape space in media pool 1 and I have space left on a tape in media pool 2. It is a pain in the ass. Most of the time I add an extra tape to the pool wich is insane because I only need 100GB wich is available on media pool 2. It would be an nice faeture to have 1 medial pool with 3 drives (atthaced to 3 tape servers), so that tapes are use more effecient.
I don't have a problem with using seperate tape jobs for each drive.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

Gerwin,
Thanks! Have you considered the ability to run tape jobs simultaneously - one per drive only or multi-streaming whithin one tape job is also a possibility? Thanks
aaron@ARB
Expert
Posts: 138
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by aaron@ARB »

+1 from me too.. Its so frustrating to have 2 LTO6 drives and only really able to use one.

but PLEASE put some proper thought into it and give a lot of customisation options in relation to such an option as there ARE times that you want to lock the media pool so jobs will run in serial, but as said above, BIG data jobs would benefit from this functionality
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

Aaron,
+1
To which one: job multi-streaming or backup files within one job multi-streaming? :D
give a lot of customisation options in relation to such an option
I was thinking about tape jobs and files multistreaming, and settings that would allow to tune the actunal number of drives that could be used for multi-streaming within one media pool. Any other customization suggestions are highly appreciated
aaron@ARB
Expert
Posts: 138
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by aaron@ARB »

Make that' +2 then as both are good

I really hate to say this but check out backup execs tape management. They have been doing tapes for eons, they have it working pretty well from THAT perspective. (Runs into my post about tape views too). Multi streaming and multi files are both great ideas. Tape is still something complex to get right and us with auto loaders have them for a reason. I for one have to manage 12-14tb out to tape. It's a struggle, anything that can be done to stream that data out to as many drives as possible is an advantage.
michaelryancook
Expert
Posts: 116
Liked: 14 times
Joined: Nov 26, 2013 6:13 pm
Full Name: Michael Cook
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by michaelryancook »

I would say having both options would be great. Multiple files to multiple drives within the same job or multiple jobs to the same pool at the same time.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

Thanks guys. Your feedback makes the picture less blurry!
guckux
Lurker
Posts: 2
Liked: never
Joined: Aug 06, 2015 2:23 pm
Full Name: Stefan Huerter
Contact:

[MERGED] Tape backup - multiples/ parallel streams supporte

Post by guckux »

Does Veeam support multiples/parallel streams from different backup jobs (mixed: full and incremental and copy2tape) to one tape?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Shestakov »

Hello Stefan,
Parallel processing between multiple tape drives becomes available in upcoming v9 release along with other great tape features. Please take a look.
Thanks!
guckux
Lurker
Posts: 2
Liked: never
Joined: Aug 06, 2015 2:23 pm
Full Name: Stefan Huerter
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by guckux »

Hello Shestakov

that was not the question - I regognized in this discussion, that there want to be the feature, to have one backup-job to multiple tapes.
Due to my lack of knowledge of Veeam (I am in an evaluationg process, combining with data domain, comparing with EMC networker) I am learning and asking for "Features, which are standards in networker". But also, there is a entirely different concept in these products.

For me, it is important, what is NOW supported, and not, what is announced or so... these are no facts, which are responsible...

So, it should be better, to ask once more with more words:
Is it possible to save more than one VM to ONE media-pool - we have a few vCenter in production...

And - is it possible to save with one Tape-Server more than one vCenter?

So far - I understand, that ONE backup job "exclusivly" use ONE media-pool - also ONE tape...?!
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Shestakov »

Stefan,
Thanks for the clarification.
guckux wrote:So far - I understand, that ONE backup job "exclusivly" use ONE media-pool - also ONE tape...?!
That is not correct. You can add as much tapes as you want to a media pool as well as just make it add tapes from Free Pool automatically when needed.
You can also set several jobs to use same media pool. All this applies to the current version and works fine.

And as I said, starting from the upcoming version parallel processing between tape drives will be available.
Thanks!
sentenza
Novice
Posts: 5
Liked: 3 times
Joined: May 27, 2015 7:41 pm
Full Name: Olivier Tarnus
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by sentenza »

Up...

I just want to confirm that this feature(s) is still greatly missing. Any confirmation/news/rumors of what will come in v9?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by foggy »

Olivier, tape functionality enhancements coming in v9 are described in this blog post. Thanks.
sentenza
Novice
Posts: 5
Liked: 3 times
Joined: May 27, 2015 7:41 pm
Full Name: Olivier Tarnus
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by sentenza » 2 people like this post

Hello,

This looks really promising. I was at one of the Veeam Tour events this morning and discussed this with the Veeam engineers present. Things are not going to be 100% perfect for tape in v9, but the improvements that are coming are definitely huge and I'm sure this will help a lot of customers, particularly on the performance side.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

Music to my ears, thanks Olivier. Do not forget to share the thoughts about missing features and your personal use cases – it will help us a lot when we are going to discuss the next version’s improvements. Cheers!
einhirn
Enthusiast
Posts: 53
Liked: 18 times
Joined: Feb 02, 2015 1:51 pm
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by einhirn »

Sorry for hijacking this thread, but I found it when looking for someone requesting the same feature as me.

I just found out that with v9 I can start multiple jobs pointing to one Media pool, but I can't make it to run more than one job with parallel processing.
I tried to accomplish that by running two jobs pointing to the same media pool and set it to use 3 of 6 drives. But obviously the count of 3 drives is a total for the media pool instead a "per job" limit for the media pool. I can't load-balance two jobs to one pool, they'll always run one after another.

The reason I was trying to do that, is that one job doesn't seem to run faster with parallel processing to more drives so I just wanted to run more jobs in parallel.

Hopefully v9.5 will change a lot in terms of tape performance...
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by Dima P. »

einhirn,
the count of 3 drives is a total for the media pool instead a "per job" limit for the media pool
Can you elaborate? Parallel processing allows to run several tape jobs simultaneously if you have enough tapes assigned to this media pool.
v9.5 will change a lot in terms of tape performance
You will be surprised :wink:

P.S. Tape Parallel Processing - good read
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Feature request: parallel tape jobs to same media pool

Post by mkretzer »

Will there be an option in V10 to stream multiple backup files to the same drive? We really have problems keeping our LTO-7 Drive streaming - the storage could deliver more but not in one stream. Furthermore, if we have more than one storage we are always limited by the individual storages...
Post Reply

Who is online

Users browsing this forum: No registered users and 28 guests