After the 9.5 u4a update I can't control my tape unit

[aaron@ARB]

More tapes loaded on the weekend and this is what happened, so its not isolated to just the tapes that were present during the upgrade (which was previously thought/communicated) the problem is ongoing.

I am not going to offline and delete them as they have data on them that I dont want to go through the process of re-cataloging, I'm just going to find where the tapes currently sit in the loader and manually update the DB to match I think.

Is there any time frame on a solid fix?

Code: Select all

name		barcode		original_slot
CJH047L6	CJH047L6	1571729563
CJH049L6	CJH049L6	1571729563
CJH167L6	CJH167L6	1571729563
CJH170L6	CJH170L6	1571729563
CJH169L6	CJH169L6	1571729563
CJH044L6	CJH044L6	1571729563
CJH048L6	CJH048L6	1571729563
CJI449L6	CJI449L6	1571729563
CJH050L6	CJH050L6	1571729563

[dariusz.tyka]

Hi,

I've read latest words from Gostev regarding this issue and missing .NET updates. This Veeam server was indeed missing some .NET updates.
My question is - if I install all required updates will this problem disappear or do I still to install this fix from Veeam when it is available?

[mcz]

As far as I understood, updates should be enough. Because if all .NET updates were installed, you wouldn't see the issue. The JIT-compiler itself has this bug and that's why there is no need to change the code or any part that gets interpreted by the compiler.

Please correct me if I'm wrong.

[Avi Milbaum]

Hello Avi,

Have your tried the workaround proposed in this thread? Thanks!

Case # 03556160
Yes
eject the tape from library, making them offline, remove tape from catalog, insert tape again and perform library inventory - and then retry the job finish with no eject issue

[pob579]

I had the case opened for this 3 weeks ago... but found the solution by myself when ejected the tape on Library side.
Other fixes not helped.

[Gostev]

As far as I understood, updates should be enough. Because if all .NET updates were installed, you wouldn't see the issue. The JIT-compiler itself has this bug and that's why there is no need to change the code or any part that gets interpreted by the compiler.

That is correct.

[ers_kentwick]

Ok, I'm confused. I'm seeing some posts saying "the bug is in the .NET 4.6" and other posts saying "Update .NET to 4.6". Which way is it? The Veeam 4a install process installed .NET 4.6 on our Veeam server while our tape server is at .NET 4.5.2. Do we update to .NET 4.7.x? Or what?

[aaron@ARB]

Once this is all finalized and such like, it is also worth considering the way in which support is provided as this issue was closed out by my local engineer last week saying basically the fix is the actual solution and when it happened against yesterday I re-opened it and are awaiting a response.

I get more information from the forum and the community digest from yesterday than I have from support to this point. That is not really the way it should work (IMHO anyway), it feels like (and this is not an isolated issue) that the support experience appears a little disjointed when compared to what Dima and Gostev can report. Do the worldwide support teams speak to each other or have the ability to share their experiences so as to provide a cohesive experience to the userbase?

Just my 0.2c worth.

[Gostev]

@Aaron myself (as the lead PM) and Dmitry (PM for tape) have an unfair advantage of being connected directly into the dev team. Because of that, we get the information much faster than support, like literally the moment devs figure out something. In addition, as you see we have tools in place to instantly share all the important information with our users, and we put a lot of effort in actually doing so.

All of that may indeed leave than impression that we PMs are on top of things while our technical support is not. But in reality, if myself and my team were not so obsessed with keeping our users and partners posted on everything important that we get to know, we would not be any different from any other vendor. Meaning, you would get all the information strictly from support, following normal communication delays (from R&D to support leadership, and then from support leadership to every support engineer).

[Gostev]

Ok, I'm confused. I'm seeing some posts saying "the bug is in the .NET 4.6" and other posts saying "Update .NET to 4.6". Which way is it? The Veeam 4a install process installed .NET 4.6 on our Veeam server while our tape server is at .NET 4.5.2. Do we update to .NET 4.7.x? Or what?

Don't even worry about .NET versions. The only thing you need to do is make sure Windows OS on all Veeam servers is up to date (by checking for and installing all available Windows Updates). Doing this will patch the .NET 4.6 bug that is causing the issue with tape ejection in Update 4a.

[atc_stockton]

Don't even worry about .NET versions. The only thing you need to do is make sure Windows OS on all Veeam servers is up to date (by checking for and installing all available Windows Updates). Doing this will patch the .NET 4.6 bug that is causing the issue with tape ejection in Update 4a.

Our VEEAM server and Tape Server are both up to date. Still not able to eject. This just started happening last week. I upgraded to 4A over a month ago, no issues until now.

[lyapkost]

Hi Scott. Since the problem's appeared, incorrect data was written to the DB and your drive fails to eject tape to nonexistent 10-gidit slot. Please contact support to fix it and double check whether all the required updates are there. You may refer to the case #03497479.

[atc_stockton]

Perfect, thank you!

[Moebius]

I too had some tape issues immediately after upgrading to v9.5 u4a from 9.5 u2.
I'm using GFS tape backups that run on the weekends. The job runs on the first weekend produced errors never seen before (from being unable to unload/eject tapes, to a "Transmission pipeline hanged" error. (Case ID 03525846). Don't know if this is actually related to this thread.

Without really changing anything, the next two weekends ran without errors. I have no clues about what it was. I'm still waiting to see if this can classify as a one-time, upgrade-related issue that fixed itself

The tape jobs have run with no errors since.
However, some other user reports that errors re-surfaced after several successful runs, and according to support this is due to wrong entries written in the database upon upgrading. So if I get this straight this could be a silent problem that will not appear until the badly referenced tapes are used.

So to wrap this up: what am I supposed to do in my case, other than updating .NET? Wait for errors to appear or do something proactively?

[Dima P.]

Lucio,

I'd recommend you to raise a case and update .Net components based on the comments from our support team (they will make sure that you wont face any side affects or related issues during the upgrade procedure). Cheers!

[Compmor]

We also have had no specific problems over 28 years owning here so wondering about the question?

[fkozlik]

Hi,

I installed a few weeks ago B&R 9.5 U4a and we have exact the same situation.
One of the differences, we have a HP StoreEver MSL2024 with LTO5 drives and barcode reader ... (these info just to show that this happens with automatic too)

However: every single Tape we used had wrong slot numbers as "original slot" and we found out with the fresh tapes we get away with erasing them (short is enough) and the the slot was working.

and now to some "funny" stuff:
After some tape rotation, we put in some tapes we used before and we got the same problem again: wrong slot numbers at the eject procedure.

After looking into the db from Veeam I found in dbo.Tape.tape_mediums that there is the wrong slot set at "original_slot" - even the right one is at "location_address" right next to it.
I can try hundreds of things but "original_slot" won't change to the number it was scanned after switching the tapes.
When this happens, than it is possible to load the tape, but not to eject it afterwards!

Now what I found out (just to help others with the same problem - the library is not 5 walking minutes away ...) load the tapes, erase them and then use HP Tape Tools with the "eject utility". After switching back to online the tape library need to scan itself and then the tape is in the right slot! Also in the veeam db!
Sadly, only the tape which was "emergency ejected"

So, my conclusion to this: Maybe it would be easy for veeam, if there would be a "rescan slots" button anywhere or a powershell command for scanning all slots with the tapes and writing the found slots into the "original:slot"
It may a little stupid program which can do that after swapping tapes and there wouldn't be any big problem for anyone...

For sure, anyone else could do it too, but it would be nice, if the code-writers of Veeam would do this for their created database.

After a long search on the internet I can tell you: I'm not alone with this problem and always telling people "mage a call" won't help on the long way!

[fkozlik]

...meant "make a call"

why I've written "mage" I don't know

[owenrock84]

I try re-add tape server, it solved my tape drive issue and backup is working now.

[Didi7]

Hi guys, with all those hassles regarding tape maintenance and management, it would be interesting to know, if all those bugs have been removed with VBR 9.5 U4b, which will soon be released.

Wherever I use single tape units or tape libraries in Veeam B&R environments, I avoided to install VBR 9.5 U4a so far!

Regards,
Didi7

[Gostev]

Hello, the bug discussed in this topic sits in .NET, so it cannot be fixed from Veeam side in Update 4b. Thanks!

[Didi7]

Are those bugs still present, if Net Framework 4.8 is used?

[Dima P.]

Didi7,

Veeam B&R and other products rely on v4.6, so even if you have latest .Net version installed, v4.6 is going to be installed during product installation.

[Didi7]

I see! So the bug is still present, because NET Framework 4.6 hasn't been fixed so far, right?

Regards,
Didi7

[Dima P.]

This bug is related to .Net v4.6 but can be cured by installing the latest Windows updates.

...we were stuck thinking we're dealing with stack corruption, as only that could explain invalid function output. So we had to engage our most hardcore researchers – folks who are really good with disassembler and analyzing the machine code. And after many hours, they've nailed the issue: it appeared to be a bug in JIT (Just In Time) compilation logic of .NET Framework 4.6! A small change in the source code sent the JIT compiler on the path that contains the bug, resulting in invalid optimizations applied to the modified function and the invalid machine code created...

[Didi7]

All those incidents people are reporting here don't make it very trustworthy to install 9.5 U4a (or higher versions) in environments with tape usage.

Best regards,
Didi7

[Dima P.]

Didi7,

Unfortunately the original issue reported in this thread was caught on the unpatched Windows machine and .Net framework but not by Veeam code itself. Overall every Update contains fixes for most critical reported issues to our and provides even better quality than initial release.

[Didi7]

Hi Dima P.,

thanks for letting me know. There is no hurry to update to VBR 9.5 U4a. In environments, where we do not use tape storage with Veeam software, we already sucessfully use VBR 9.5 U4a.

We will wait for VBR 9.5 U4b and will test one environment with tape library usage. We always keep our VBR-servers up-to-date, so if this updated environment (VBR 9.5 U4b) keeps stable, I will rollout this version into all environments.

Regards,
Didi7

[Dima P.]

Didi7,

Sounds like a perfect plan: Update 4b is literary around the corner and you'll save some time if you go strait to U4b bypassing U4a. Since our updates are cumulative and latest version always contain fixes from previous versions you are getting all the needed fixes with one installation instead of two. Cheers!

[dive7]

Is this still an issue in 9.5u4b? Updated last night and had some issues, trying now to eject a tape and prepare the library for weekly backups. Every time I eject the tape the library shows offline in Veeam and the tape is not ejected. I am going to update my existing case # 03693853 with this question as well.