Feature request: parallel tape jobs to same media pool

[sentenza]

Dear Veeam community,

Here is my "official" feature request: Authorize multiple jobs to request tapes from the same media pool. This comes after weeks of investigation with support (#00850675), and seems the minimal requirement to support "big" tape jobs, ie: more than 12TB in less than 24 hours.

Problem: When a job starts, it locks the media pool and will only release it when finished. Another job that could run on another drive in the same library can not request tapes from the same media pool until first jobs is finished, greatly limiting the throughput to only one drive. The only alternative is to add more media pools, adding far too much complexity and fragmentation in the library.

Why is this needed: As Veeam is not implementing tape multi-streaming (ie: writing the same job to multiple tapes in parallel), the only alternative for "big data volumes" is to run multiple jobs in parallel on individual drives. Managing more jobs already adds complexity, but fragmenting the media pools decrease efficiency dramatically and totally "disconnect" the pools from what they should be (ie: Daily, Weekly, Monthly...).

Expected result: A job should not lock the media pool but only the tape it is currently using. Other jobs should be able to claim another (free) tape in the same pool.

My experience on this: I have totally changed my strategy because of tape issues...

6 month ago, I had 3-4 jobs backing up 400 VMs to 2 big repositories. It was easy to manage and error prone: basically we did backup everything, and were sure no VM was left away. And then we started to implement tape "for real"...

Fast forward 6 months: I have 15 jobs, backing up to 7-8 repositories. We have also added copy jobs and their own repositories as a "buffer", because tape jobs were locking the main repositories thus preventing production backups to run. Everything is now more complex, our storage usage as exploded and it definitely requires more time to manage, but my VM count and volume increased only by 5% and this makes it hard to explain to management... And still can't come up with a viable tape strategy...

We need a solution, as 12TB in 24 hours (basically 1 LTO6 drive full speed) does not seem "that" big in my opinion and set the bar quite low for tape users.

Olivier

[justind]

+1 vote from me

Being able to share media pools would be great. Currently I have 18 Media pools that I could reduce down to 3 with shared pools

[Dima P.]

Hello guys,
I can’t provide you any details on the next major version feature set, however, I want to encourage you to wait for it as the issue you described will be resolved with some additional really cool new functionality

[robpare]

+1 from me as well and I look forward to see what comes in the next version.

We've just started setting up tape media pools and have been caught out by media pools being locked when in use by one job. Sentenza captures the problem more perfectly than I think I'd have done.

We can come up with workarounds for this but I'd much rather go back to having a small number of media pools - we might get away with just two for weekly/monthly if the lock happened per tape rather than at the pool level.

[Dima P.]

So you would like to point multiple tape jobs to the singe media pool and let this tape jobs run in parallel, right?

[gosnells]

+1
being able to fully utilise an LTO6 tape library with two drives is a must. Surely tape support was added for / aimed at enterprise users, and most enterprise users do not muck around with a single drive. Splitting the tape backup jobs into multiple media pools to get concurrency is crazy.

[bvrz]

hello guys,

when comes the next major version Feature with these addin?
thanks

[Dima P.]

bvrz,
There is no ETA at the moment.

Mike,
Are you ok with a possible tape media overrun?

[Gerwin1]

+1 for me to.
I have a tape drive with 3 LTO 6 drives. Now we have 3 media pools. When I need 100 GB additional tape space in media pool 1 and I have space left on a tape in media pool 2. It is a pain in the ass. Most of the time I add an extra tape to the pool wich is insane because I only need 100GB wich is available on media pool 2. It would be an nice faeture to have 1 medial pool with 3 drives (atthaced to 3 tape servers), so that tapes are use more effecient.
I don't have a problem with using seperate tape jobs for each drive.

[Dima P.]

Gerwin,
Thanks! Have you considered the ability to run tape jobs simultaneously - one per drive only or multi-streaming whithin one tape job is also a possibility? Thanks

[aaron@ARB]

+1 from me too.. Its so frustrating to have 2 LTO6 drives and only really able to use one.

but PLEASE put some proper thought into it and give a lot of customisation options in relation to such an option as there ARE times that you want to lock the media pool so jobs will run in serial, but as said above, BIG data jobs would benefit from this functionality

[Dima P.]

Aaron,

+1

To which one: job multi-streaming or backup files within one job multi-streaming?

give a lot of customisation options in relation to such an option

I was thinking about tape jobs and files multistreaming, and settings that would allow to tune the actunal number of drives that could be used for multi-streaming within one media pool. Any other customization suggestions are highly appreciated

[aaron@ARB]

Make that' +2 then as both are good

I really hate to say this but check out backup execs tape management. They have been doing tapes for eons, they have it working pretty well from THAT perspective. (Runs into my post about tape views too). Multi streaming and multi files are both great ideas. Tape is still something complex to get right and us with auto loaders have them for a reason. I for one have to manage 12-14tb out to tape. It's a struggle, anything that can be done to stream that data out to as many drives as possible is an advantage.

[michaelryancook]

I would say having both options would be great. Multiple files to multiple drives within the same job or multiple jobs to the same pool at the same time.

[Dima P.]

Thanks guys. Your feedback makes the picture less blurry!

[guckux]

Does Veeam support multiples/parallel streams from different backup jobs (mixed: full and incremental and copy2tape) to one tape?

[Shestakov]

Hello Stefan,
Parallel processing between multiple tape drives becomes available in upcoming v9 release along with other great tape features. Please take a look.
Thanks!

[guckux]

Hello Shestakov

that was not the question - I regognized in this discussion, that there want to be the feature, to have one backup-job to multiple tapes.
Due to my lack of knowledge of Veeam (I am in an evaluationg process, combining with data domain, comparing with EMC networker) I am learning and asking for "Features, which are standards in networker". But also, there is a entirely different concept in these products.

For me, it is important, what is NOW supported, and not, what is announced or so... these are no facts, which are responsible...

So, it should be better, to ask once more with more words:
Is it possible to save more than one VM to ONE media-pool - we have a few vCenter in production...

And - is it possible to save with one Tape-Server more than one vCenter?

So far - I understand, that ONE backup job "exclusivly" use ONE media-pool - also ONE tape...?!

[Shestakov]

Stefan,
Thanks for the clarification.

So far - I understand, that ONE backup job "exclusivly" use ONE media-pool - also ONE tape...?!

That is not correct. You can add as much tapes as you want to a media pool as well as just make it add tapes from Free Pool automatically when needed.
You can also set several jobs to use same media pool. All this applies to the current version and works fine.

And as I said, starting from the upcoming version parallel processing between tape drives will be available.
Thanks!

[sentenza]

Up...

I just want to confirm that this feature(s) is still greatly missing. Any confirmation/news/rumors of what will come in v9?

[foggy]

Olivier, tape functionality enhancements coming in v9 are described in this blog post. Thanks.

[sentenza]

Hello,

This looks really promising. I was at one of the Veeam Tour events this morning and discussed this with the Veeam engineers present. Things are not going to be 100% perfect for tape in v9, but the improvements that are coming are definitely huge and I'm sure this will help a lot of customers, particularly on the performance side.

[Dima P.]

Music to my ears, thanks Olivier. Do not forget to share the thoughts about missing features and your personal use cases – it will help us a lot when we are going to discuss the next version’s improvements. Cheers!

[einhirn]

Sorry for hijacking this thread, but I found it when looking for someone requesting the same feature as me.

I just found out that with v9 I can start multiple jobs pointing to one Media pool, but I can't make it to run more than one job with parallel processing.
I tried to accomplish that by running two jobs pointing to the same media pool and set it to use 3 of 6 drives. But obviously the count of 3 drives is a total for the media pool instead a "per job" limit for the media pool. I can't load-balance two jobs to one pool, they'll always run one after another.

The reason I was trying to do that, is that one job doesn't seem to run faster with parallel processing to more drives so I just wanted to run more jobs in parallel.

Hopefully v9.5 will change a lot in terms of tape performance...

[Dima P.]

einhirn,

the count of 3 drives is a total for the media pool instead a "per job" limit for the media pool

Can you elaborate? Parallel processing allows to run several tape jobs simultaneously if you have enough tapes assigned to this media pool.

v9.5 will change a lot in terms of tape performance

You will be surprised

P.S. Tape Parallel Processing - good read

[mkretzer]

Will there be an option in V10 to stream multiple backup files to the same drive? We really have problems keeping our LTO-7 Drive streaming - the storage could deliver more but not in one stream. Furthermore, if we have more than one storage we are always limited by the individual storages...