Tape logic when dealing with libraries and multiple drives

[aaron@ARB]

Veeam,

Can you please help me understand the logic you use when dealing with tape libraries and multiple tape drives? Consider the following scenario for a moment please. I will change the name of the jobs to protect the guilty. When I do full backups I have multiplexing turned on so that I can use both the drives in the Dell TL2000 library that I have. During the week for the incrementals, I have it turned off so that I ensure that tapes are full and not left half empty. I will demonstrate the below with 1 job to demonstrate my point.

Day 1: Multiplexing ON: Multiple Active full backups jobs for multiple VM's run and write these files to tape via dedicated tape jobs.
Day 2: Multiplexing OFF: Daily backup runs on Library 1 it picks CJH018L6 in Drive1. After backup CJH018L6 has 350gb/2.3tb free

I notice that there is CJH019L6 which has 2.2tb/2.3tb free. Knowing that my backups take about 400gb a day, if i left Veeam to its own devices it would fill CJH018L6 and grab another spare tape which would only have 50gb used. Trying to help Veeam out here, I eject CJH018L6 from Drive1 and load CJH019L6 into Drive1 which has 2.2tb free

To make sure that it is going to use the tape that is in there, i run the backup job to create another VIB and then run the subsequent tape job again. Instead of writing to CJH019L6 which is in Drive1 it loads CJH018L6 into Drive2 and then completes the backup completely ignoring CJH019L6.

Day 2 job log summary, you can see it used Drive 1 with tape CJH018L6

Code: Select all

23/12/2019 11:34:40 PM :: Building source backup files list started at 23/12/2019 11:34:40 PM  
23/12/2019 11:34:41 PM :: New backup P2-ARBBackup-AccountsPSD2019-12-23T232939_B4B3.vib will be placed into the media set   
23/12/2019 11:34:41 PM :: Source backup files detected. VIB: 1  
23/12/2019 11:34:41 PM :: Queued for processing at 23/12/2019 11:34:41 PM  
23/12/2019 11:34:41 PM :: Required backup infrastructure resources have been assigned  
23/12/2019 11:34:42 PM :: Waiting for tape infrastructure resource availability  
23/12/2019 11:50:32 PM :: Using tape library IBM 3573-TL C.30  
23/12/2019 11:50:35 PM :: Drive 1 (Server: ARBBACKUP, Library: IBM 3573-TL C.30, Drive ID: Tape1) locked successfully  
23/12/2019 11:50:38 PM :: Current tape is CJH018L6  
23/12/2019 11:50:38 PM :: New tape backup session started, encryption: disabled  
23/12/2019 11:50:42 PM :: Processing incremental backups started at 23/12/2019 11:34:38 PM  
23/12/2019 11:50:43 PM :: Processing P2-ARBBackup-AccountsPSD2019-12-23T232939_B4B3.vib  
23/12/2019 11:50:59 PM :: 0 folders and 1 files have been backed up  
23/12/2019 11:50:59 PM :: Busy: Source 11% > Proxy 17% > Network 14% > Target 99%  
23/12/2019 11:50:59 PM :: Primary bottleneck: Target  
23/12/2019 11:50:59 PM :: Network traffic verification detected no corrupted blocks  
23/12/2019 11:50:59 PM :: Processing finished at 23/12/2019 11:50:59 PM  

I then eject CJH018L6 from Drive 1 and load CJH019L6 instead. CJH019L6 already has some VIB's on it, so it is already in the correct media set etc.

Code: Select all

24/12/2019 10:07:06 AM :: Building source backup files list started at 24/12/2019 10:07:06 AM  
24/12/2019 10:07:06 AM :: New backup P2-ARBBackup-AccountsPSD2019-12-24T100307_40D7.vib will be placed into the media set   
24/12/2019 10:07:06 AM :: Source backup files detected. VIB: 1  
24/12/2019 10:07:06 AM :: Queued for processing at 24/12/2019 10:07:06 AM  
24/12/2019 10:07:06 AM :: Required backup infrastructure resources have been assigned  
24/12/2019 10:07:07 AM :: Using tape library IBM 3573-TL C.30  
24/12/2019 10:07:10 AM :: Drive 2 (Server: ARBBACKUP, Library: IBM 3573-TL C.30, Drive ID: Tape0) locked successfully  
24/12/2019 10:07:11 AM :: Loading tape CJH018L6 from Slot 3 to Drive 2 (Server: ARBBACKUP, Library: IBM 3573-TL C.30, Drive ID: Tape0)  
24/12/2019 10:08:03 AM :: Current tape is CJH018L6  
24/12/2019 10:08:04 AM :: New tape backup session started, encryption: disabled  
24/12/2019 10:09:07 AM :: Processing incremental backups started at 24/12/2019 10:07:03 AM  
24/12/2019 10:09:09 AM :: Processing P2-ARBBackup-AccountsPSD2019-12-24T100307_40D7.vib  
24/12/2019 10:09:17 AM :: 0 folders and 1 files have been backed up  
24/12/2019 10:09:18 AM :: Busy: Source 11% > Proxy 7% > Network 11% > Target 99%  
24/12/2019 10:09:18 AM :: Primary bottleneck: Target  
24/12/2019 10:09:18 AM :: Network traffic verification detected no corrupted blocks  
24/12/2019 10:09:18 AM :: Processing finished at 24/12/2019 10:09:18 AM  

So we can see that Veeam loads CJH018L6 into Drive2 rather than using CJH019L6 in Drive1. Why?

So, I would like to know... What causes this logic? Does Veeam try to keep writing to the same tape it used during its last run instead of realising that there is a tape already in drive which is of the correct media set?

I logged this with support a little while ago but got no joy no-one really knew what was going on with any level of confidence.

Cheers,
Aaron

[HannesK]

Hello,
can you please post the case number that we can have a look at it?

Does Veeam try to keep writing to the same tape it used during its last run

yes, that's how a media set works. The idea of "manual help" does not exist, because this only works in very small environments.

Best regards,
Hannes

[aaron@ARB]

It was long since closed so I dont have it to hand but i only raise it as it happened again and rather than log it i thought I would ask on here as often you guys are closer to engineering and sometimes get more specific answers.

yes, that's how a media set works. The idea of "manual help" does not exist, because this only works in very small environments.

I was kind of hoping that the log would have been to just check to see that the drive had a tape of the right media set in it and use that as opposed to go looking for the last one it used. Often these would be the same thing as the tape it used last would often still be in the drive, but it would he handy to be able to influence / give it a little guidance every now and then.

Great example is at the moment.. I have 2 tape drives... a job is stalled on "waiting for tape". There is a tape with 1.6tb free sitting in the other drive, yet Veeam won't use it. I get that part as I don't have multiplexing turned on and even if I did, I dont suspect the logic would allow it to jump to the next drive and start using that. So what I have done is eject the tape from Drive 1 back to slot 19, yet the system will not pick realise that it can use it, it wants a blank tape instead as shown by the task log

Code: Select all

[27.12.2019 09:03:54] <22> Info     Selecting media pool: Media Pool, id: 43ff2e56-0452-43ca-b608-4e6f9b3cc744, name: 'Week4', description: 'Created by ENSAT\administrator at 23/06/2015 11:52:14 AM.', type: UserSimple, lockId: , options: '<?xml version="1.0" encoding="utf-16"?><MediaPoolOptions xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><MoveFromFree>true</MoveFromFree><LibrarySwitchPolicy><Type>Failover</Type><LibraryIsOffline>true</LibraryIsOffline><NoAvailableMedia>true</NoAvailableMedia><AllDrivesAreBusy>false</AllDrivesAreBusy></LibrarySwitchPolicy><UseWormTapesOnly>false</UseWormTapesOnly><MediaSetName>Week4</MediaSetName><MediaSetCreationOptions><Type>Never</Type><DailyType>Everyday</DailyType><DailyDays><DayOfWeek>Monday</DayOfWeek><DayOfWeek>Tuesday</DayOfWeek><DayOfWeek>Wednesday</DayOfWeek><DayOfWeek>Thursday</DayOfWeek><DayOfWeek>Friday</DayOfWeek></DailyDays><MonthlyTicks>432000000000</MonthlyTicks><DailyTicks>432000000000</DailyTicks><MonthlyType>First</MonthlyType><MonthlyDayOfWeek>Sunday</MonthlyDayOfWeek><MonthlyDay>1</MonthlyDay></MediaSetCreationOptions><GfsDailyOptions><RetentionPeriod><Type>Days</Type><Value>14</Value></RetentionPeriod><MediaSetName>Daily media set #%id% %date%</MediaSetName><AppendBackupFiles>true</AppendBackupFiles><AddTapesFromMediaPool>true</AddTapesFromMediaPool><VaultId xsi:nil="true" /></GfsDailyOptions><OverwriteRetention>Period</OverwriteRetention><OverwritePeriod><Type>Weeks</Type><Value>3</Value></OverwritePeriod><UseEncryption>false</UseEncryption><ParallelProcessingOptions><Enabled>false</Enabled><TaskParalleling>false</TaskParalleling><DriveCount>2</DriveCount></ParallelProcessingOptions></MediaPoolOptions>', vault: 'N/A'
[27.12.2019 09:03:54] <22> Info     [CTapeMediaPool] Locking media pool 'Week4'
[27.12.2019 09:03:54] <22> Info         [CTapeMediaPool] Media pool 'Week4' is successfully locked
[27.12.2019 09:03:54] <22> Info     [CTapeMediaPoolLocker] Media pool Week4 locked successfully
[27.12.2019 09:03:54] <22> Info     [CTapeMediumForWriteFindAlg] Searching next tape medium to write to in media pool 'Week4', media set '1576966684': [MediaPoolAvailableTapes: All: 15, Available: 0, AvailableForOverwrite: 0, FreeTapes: 0,  IsNewMediaSet: False] 
[27.12.2019 09:03:54] <22> Info     [CTapeMediumForWriteFindAlg] Last modified tape medium found: [TapeMedia CJH177L6/adee60cf-2fa0-46ac-a0e8-c0a4a0bceced, 2499999170560b/4851761152b (Protected: False, IsActive: False, Retired: False, IsFree: False, IsFull: True, IsOnline: False, IsWriteProtected: False, IsWorm: False),  MediaSetId: 1576966684, MediaSetType: Simple]
[27.12.2019 09:03:54] <22> Info     [TapeMedium] Capturing tape medium from media pool 'Free' to media pool 'Week4'
[27.12.2019 09:03:54] <22> Info     [TapeMedium] No tape medium has been captured
[27.12.2019 09:03:54] <22> Info     Unlocking media pool 'Week4'
[27.12.2019 09:03:54] <22> Info         Media pool 'Week4' is successfully unlocked

What would be more advantageous (and more tape efficient I believe) is something like

Code: Select all

  Select tapes from Mediaset that are not full
     Return Tapes
 If No free tapes, return one from free media pool    

That way any used tapes that are already in the mediaset will be used as opposed to just chewing up free tapes as is what is going to have to happen to me now.

just some thoughts.

[soncscy]

Hi Aaron,

What are your full media pool settings? Your use of "multiplex" here is a little off-putting since I associate multiplexing as multiple streams on a single file, but really Veeam does processing in parallel (hence, Parallel Processing).

With parallel processing, each drive gets its own mediaset, a logical collection of tapes, and this might be where you're getting a bit of a wrench in the works. IIRC, tapes are restricted to their own mediaset, and the only foreign tapes that can be introduced are those from the Free Media Pool. (one of my colleagues mentioned there is some key she got from Veeam Support for GFS media pools which makes this an exception, but I didn't look into it)

So if I get this right from the log, your Mediapool doesn't have free tapes:

MediaPoolAvailableTapes: All: 15, Available: 0, AvailableForOverwrite: 0, FreeTapes: 0, IsNewMediaSet: False

That's a lot of zeroes. In order to keep the media sets consistent and the tapes logically consistent, you need to use tapes within the media pool, and from there, it depends on your mediaset settings. If you are setting to Append to the previous tape, then Veeam wants to use the previous tape if it is available and has space left. If you create new mediasets at a given interval, then Veeam will always want a new tape at the start of a new said interval, and this is purely organizational.

What is your target tape-out goal? Likely it's doable and just your media pool settings (mediaset creation) needs to be adjusted.

As best I know, just hot-swapping whatever tape isn't possible, so you're not going to get that, but if these tapes are correctly organized into a said pool, then you should be able to do what you want and not have to worry about the need to hot-swap a tape in; your backups should be predictable (my favorite part about tape) and you can meet your backup requirements.

[aaron@ARB]

Apologies, yes we are talking about Parallel Processing.

Correct, there were deliberately no free tapes available but this is to try and ensure that Veeam utilises the tapes that it has available before it goes and grabs new tapes. Example being that as i write this, I have 2 tapes each with only about 250/300gb stored on them as the logic will grab new tapes often before it checks if there is free space on another tape. As soon as it gets into the 'waiting on tape' it will only accept a free tape as opposed to having a better look around for a tape that is already in the media pool that has some space on it. Either through being in the other drive, or closer to what i would consider possible, ejected back into its slot. It's just the logic would be better suited in my situation that at the 'waiting for tape' stage, that the system checks for tapes with available space as opposed to just wanting a free/blank tape.