extra tape being used - cant figure out why

[gosnells]

Hi,

We have an Tape backup job that takes weekend active fulls and writes them off to LTO-7.
It has been in play for quite some time, and working ok. Circa 16-17TB of active full data writing out to 4 tapes.

My quandary is that the tape job has recently pushed out to 5 tapes and I cant work out why.

Quantum tape library. One drive enabled. One single media pool.
LT0-7 drive. No h/w compression enabled.
LTO-7 tapes @ 5.5TB each tape
4 tapes == 22TB capacity
5 tapes == 27.5TB capacity

For the tape backup job, Total size, Data read and Transferred all == 17.9TB

However, if looking at tape info within Veeam, and summing up all the free space for the individual tapes used for that job run, the total tape capacity used is circa 25TB.

A couple of weeks back, we were still on 4 tapes, and Total size, Data read and Transferred were all at about 16.7TB. Tape 4 was barely used, showing 5.0TB of free space.
Now with the new 5 tape media set, tapes 1to4 have <3GB free on each, and tape 5 is showing ~2.5TB of free space. Nonetheless, 25TB is still way more than the reported <18TB of VBK data being written. Are there any known inefficiencies with writing large files, or something else I'm missing ?

The stupid thing is that the active fulls only pushed out to 4 tapes <3 months ago, and whilst there may be a bit of data growth, it certainly isn't 5TB worth (of Veeam compressed/deduped data).

Any ideas ?

[Dima P.]

Hello Mike,

Is that brand new media? Thanks!

[gosnells]

Hi,

Nope, other than the additional 5th tape being used, the existing tapes are used.
At the time of the backup, for the tapes being re-used, retention has expired, and these are being overwritten. All show 5.5TB capacity free.

I can't understand where the inefficiencies are coming from. If Veeam is calculating total size, reading and transferrring/writing, why does Veeam own individual tape stats show the tapes having written at lot more data.
For one of our VMs, the generated VBK is huge >7TB VBK file, but afaik this isn't a new thing.

e.g. for our latest backup job

Total Size 18.3TB
Data Read 18.3TB
Transferred 18.3TB

name capacity free
0000021L7 5.5TB 2.3GB
0000022L7 5.5TB 4.0GB
0000023L7 5.5TB 2.1GB
0000028L7 5.5TB 3.4GB
0000024L7 5.5TB 3.0TB

total capacity 5 x 5.5TB 27.5TB
total free 3.0115TB
total used 27.5 - 3.0115TB == 24.4885TB

Where's the 30+% overhead coming from ?

Individual tape info:
The worst I could find is Wear: 0 read cycles, 22 write cycles
Can Veeam report any more detailed tape error rate info ?

For completeness, there is no parallel processing in play, just the one tape drive being used, one tape after another in that job.

Any feedback much appreciated.

[soncscy]

Hey Mike,

last time we had disappearing space with our tape unit, it was a bum write header. A bit of investigation with the IBM tool to do a full write test without compression showed that the drive stopped at ~70% of capacity.

Try testing on a tape without any important data on it and see how much you write.

[Dima P.]

In addition to Harvey's comment: tape media being continuously reused may lead to it's degradation, so some portions of 'tape tracks' will be skipped. As a result - you need an extra tape. The best way to identify the real reason is to use native test tool from your library vendor.

[gosnells]

Hi,

Cheers for the feedback.
ITDT FullWrite testing since done - successful run using new/barely-used test tape (not part of any media set)- full tape capacity 5740024 MB (incomp).
I'm not in a position to run some full write tests utilising tapes from existing media pool. These dont appear to be anywhere near the 250 backup cycles these should be good for (based on LTO7 specs). I can however add/substitute new tapes to rule that out. I just thought it was very strange that the jobs have pushed out a good 6TB extra per week.

I cannot see any useful tape details in Veeam other than read/write cycles.
Are there any further tape stats buried in logs in ProgramData ?

I'm assuming that the unlicensed "Reports - Media Integrity Analysis" on the i40 Tape Library we have here, would be the individual tape media stats info I've been blindly searching for.

[Dima P.]

Mike,

You can raise a support case and ask our folks to investigate the tape usage. If you do, please share your case ID for future reference. Cheers!