Discussions specific to tape backups
aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 12, 2019 11:09 pm

More tapes loaded on the weekend and this is what happened, so its not isolated to just the tapes that were present during the upgrade (which was previously thought/communicated) the problem is ongoing.

I am not going to offline and delete them as they have data on them that I dont want to go through the process of re-cataloging, I'm just going to find where the tapes currently sit in the loader and manually update the DB to match I think.

Is there any time frame on a solid fix?

Code: Select all

name		barcode		original_slot
CJH047L6	CJH047L6	1571729563
CJH049L6	CJH049L6	1571729563
CJH167L6	CJH167L6	1571729563
CJH170L6	CJH170L6	1571729563
CJH169L6	CJH169L6	1571729563
CJH044L6	CJH044L6	1571729563
CJH048L6	CJH048L6	1571729563
CJI449L6	CJI449L6	1571729563
CJH050L6	CJH050L6	1571729563

dariusz.tyka
Influencer
Posts: 18
Liked: 2 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by dariusz.tyka » May 13, 2019 7:46 am

Hi,

I've read latest words from Gostev regarding this issue and missing .NET updates. This Veeam server was indeed missing some .NET updates.
My question is - if I install all required updates will this problem disappear or do I still to install this fix from Veeam when it is available?

mcz
Expert
Posts: 308
Liked: 58 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by mcz » May 13, 2019 9:03 am 1 person likes this post

As far as I understood, updates should be enough. Because if all .NET updates were installed, you wouldn't see the issue. The JIT-compiler itself has this bug and that's why there is no need to change the code or any part that gets interpreted by the compiler.

Please correct me if I'm wrong.

Avi Milbaum
Novice
Posts: 5
Liked: never
Joined: Jul 17, 2016 3:15 pm
Full Name: Avi Milbaum
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Avi Milbaum » May 13, 2019 1:49 pm

Dima P. wrote:
May 12, 2019 11:29 am
Hello Avi,

Have your tried the workaround proposed in this thread? Thanks!
Case # 03556160
Yes
eject the tape from library, making them offline, remove tape from catalog, insert tape again and perform library inventory - and then retry the job finish with no eject issue

pob579
Influencer
Posts: 22
Liked: 1 time
Joined: Oct 10, 2017 5:46 pm
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by pob579 » May 13, 2019 1:55 pm

I had the case opened for this 3 weeks ago... but found the solution by myself when ejected the tape on Library side.
Other fixes not helped.

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Gostev » May 13, 2019 2:13 pm

mcz wrote:
May 13, 2019 9:03 am
As far as I understood, updates should be enough. Because if all .NET updates were installed, you wouldn't see the issue. The JIT-compiler itself has this bug and that's why there is no need to change the code or any part that gets interpreted by the compiler.
That is correct.

ers_kentwick
Influencer
Posts: 12
Liked: 1 time
Joined: Jul 26, 2016 9:15 pm
Full Name: Kent Wick
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by ers_kentwick » May 13, 2019 3:12 pm 1 person likes this post

Ok, I'm confused. I'm seeing some posts saying "the bug is in the .NET 4.6" and other posts saying "Update .NET to 4.6". Which way is it? The Veeam 4a install process installed .NET 4.6 on our Veeam server while our tape server is at .NET 4.5.2. Do we update to .NET 4.7.x? Or what?

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 13, 2019 10:37 pm

Once this is all finalized and such like, it is also worth considering the way in which support is provided as this issue was closed out by my local engineer last week saying basically the fix is the actual solution and when it happened against yesterday I re-opened it and are awaiting a response.

I get more information from the forum and the community digest from yesterday than I have from support to this point. That is not really the way it should work (IMHO anyway), it feels like (and this is not an isolated issue) that the support experience appears a little disjointed when compared to what Dima and Gostev can report. Do the worldwide support teams speak to each other or have the ability to share their experiences so as to provide a cohesive experience to the userbase?

Just my 0.2c worth.

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Gostev » May 13, 2019 11:38 pm 2 people like this post

@Aaron myself (as the lead PM) and Dmitry (PM for tape) have an unfair advantage of being connected directly into the dev team. Because of that, we get the information much faster than support, like literally the moment devs figure out something. In addition, as you see we have tools in place to instantly share all the important information with our users, and we put a lot of effort in actually doing so.

All of that may indeed leave than impression that we PMs are on top of things while our technical support is not. But in reality, if myself and my team were not so obsessed with keeping our users and partners posted on everything important that we get to know, we would not be any different from any other vendor. Meaning, you would get all the information strictly from support, following normal communication delays (from R&D to support leadership, and then from support leadership to every support engineer).

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Gostev » May 13, 2019 11:44 pm

ers_kentwick wrote:
May 13, 2019 3:12 pm
Ok, I'm confused. I'm seeing some posts saying "the bug is in the .NET 4.6" and other posts saying "Update .NET to 4.6". Which way is it? The Veeam 4a install process installed .NET 4.6 on our Veeam server while our tape server is at .NET 4.5.2. Do we update to .NET 4.7.x? Or what?
Don't even worry about .NET versions. The only thing you need to do is make sure Windows OS on all Veeam servers is up to date (by checking for and installing all available Windows Updates). Doing this will patch the .NET 4.6 bug that is causing the issue with tape ejection in Update 4a.

atc_stockton
Novice
Posts: 3
Liked: never
Joined: May 15, 2019 8:57 pm
Full Name: Scott Stockton
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by atc_stockton » May 16, 2019 2:00 pm

Gostev wrote:
May 13, 2019 11:38 pm
Don't even worry about .NET versions. The only thing you need to do is make sure Windows OS on all Veeam servers is up to date (by checking for and installing all available Windows Updates). Doing this will patch the .NET 4.6 bug that is causing the issue with tape ejection in Update 4a.
Our VEEAM server and Tape Server are both up to date. Still not able to eject. This just started happening last week. I upgraded to 4A over a month ago, no issues until now.

lyapkost
Veeam Software
Posts: 178
Liked: 28 times
Joined: Nov 27, 2015 2:26 pm
Full Name: Konstantin
Location: Saint Petersburg
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by lyapkost » May 16, 2019 3:55 pm 1 person likes this post

Hi Scott. Since the problem's appeared, incorrect data was written to the DB and your drive fails to eject tape to nonexistent 10-gidit slot. Please contact support to fix it and double check whether all the required updates are there. You may refer to the case #03497479.

atc_stockton
Novice
Posts: 3
Liked: never
Joined: May 15, 2019 8:57 pm
Full Name: Scott Stockton
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by atc_stockton » May 16, 2019 4:00 pm

Perfect, thank you!

Moebius
Veeam ProPartner
Posts: 167
Liked: 21 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Moebius » May 21, 2019 10:14 am

Moebius wrote:
May 07, 2019 1:05 pm
I too had some tape issues immediately after upgrading to v9.5 u4a from 9.5 u2.
I'm using GFS tape backups that run on the weekends. The job runs on the first weekend produced errors never seen before (from being unable to unload/eject tapes, to a "Transmission pipeline hanged" error. (Case ID 03525846). Don't know if this is actually related to this thread.

Without really changing anything, the next two weekends ran without errors. I have no clues about what it was. I'm still waiting to see if this can classify as a one-time, upgrade-related issue that fixed itself :-)
The tape jobs have run with no errors since.
However, some other user reports that errors re-surfaced after several successful runs, and according to support this is due to wrong entries written in the database upon upgrading. So if I get this straight this could be a silent problem that will not appear until the badly referenced tapes are used.

So to wrap this up: what am I supposed to do in my case, other than updating .NET? Wait for errors to appear or do something proactively?

Dima P.
Product Manager
Posts: 10710
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » May 29, 2019 3:18 pm

Lucio,

I'd recommend you to raise a case and update .Net components based on the comments from our support team (they will make sure that you wont face any side affects or related issues during the upgrade procedure). Cheers!

Compmor
Novice
Posts: 4
Liked: never
Joined: May 15, 2019 7:04 pm
Full Name: Compmor
Location: Portugal
Contact:

After the 9 5 u4a update I cant control my tape unit

Post by Compmor » May 31, 2019 7:24 am

We also have had no specific problems over 28 years owning here so wondering about the question?

fkozlik
Lurker
Posts: 2
Liked: never
Joined: Jun 18, 2019 12:02 pm
Full Name: Franz Kozlik
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by fkozlik » Jun 18, 2019 12:22 pm

Hi,

I installed a few weeks ago B&R 9.5 U4a and we have exact the same situation.
One of the differences, we have a HP StoreEver MSL2024 with LTO5 drives and barcode reader ... (these info just to show that this happens with automatic too)

However: every single Tape we used had wrong slot numbers as "original slot" and we found out with the fresh tapes we get away with erasing them (short is enough) and the the slot was working.

and now to some "funny" stuff:
After some tape rotation, we put in some tapes we used before and we got the same problem again: wrong slot numbers at the eject procedure.

After looking into the db from Veeam I found in dbo.Tape.tape_mediums that there is the wrong slot set at "original_slot" - even the right one is at "location_address" right next to it.
I can try hundreds of things but "original_slot" won't change to the number it was scanned after switching the tapes.
When this happens, than it is possible to load the tape, but not to eject it afterwards!

Now what I found out (just to help others with the same problem - the library is not 5 walking minutes away ...) load the tapes, erase them and then use HP Tape Tools with the "eject utility". After switching back to online the tape library need to scan itself and then the tape is in the right slot! Also in the veeam db!
Sadly, only the tape which was "emergency ejected"

So, my conclusion to this: Maybe it would be easy for veeam, if there would be a "rescan slots" button anywhere or a powershell command for scanning all slots with the tapes and writing the found slots into the "original:slot"
It may a little stupid program which can do that after swapping tapes and there wouldn't be any big problem for anyone...

For sure, anyone else could do it too, but it would be nice, if the code-writers of Veeam would do this for their created database.

After a long search on the internet I can tell you: I'm not alone with this problem and always telling people "mage a call" won't help on the long way!

:)

fkozlik
Lurker
Posts: 2
Liked: never
Joined: Jun 18, 2019 12:02 pm
Full Name: Franz Kozlik
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by fkozlik » Jun 18, 2019 2:05 pm

...meant "make a call"

why I've written "mage" I don't know :)

owenrock84
Lurker
Posts: 1
Liked: 1 time
Joined: Jun 20, 2019 4:09 am
Full Name: Owen Choong
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by owenrock84 » Jun 20, 2019 4:19 am 1 person likes this post

I try re-add tape server, it solved my tape drive issue and backup is working now.

Didi7
Expert
Posts: 343
Liked: 36 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Didi7 » Jun 28, 2019 3:13 pm

Hi guys, with all those hassles regarding tape maintenance and management, it would be interesting to know, if all those bugs have been removed with VBR 9.5 U4b, which will soon be released.

Wherever I use single tape units or tape libraries in Veeam B&R environments, I avoided to install VBR 9.5 U4a so far!

Regards,
Didi7
Using the most recent Veeam B&R in 19 different environments now and counting!

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Gostev » Jun 28, 2019 3:19 pm

Hello, the bug discussed in this topic sits in .NET, so it cannot be fixed from Veeam side in Update 4b. Thanks!

Didi7
Expert
Posts: 343
Liked: 36 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Didi7 » Jun 28, 2019 3:22 pm

Are those bugs still present, if Net Framework 4.8 is used?
Using the most recent Veeam B&R in 19 different environments now and counting!

Dima P.
Product Manager
Posts: 10710
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » Jul 01, 2019 3:20 pm

Didi7,

Veeam B&R and other products rely on v4.6, so even if you have latest .Net version installed, v4.6 is going to be installed during product installation.

Didi7
Expert
Posts: 343
Liked: 36 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Didi7 » Jul 01, 2019 3:27 pm

I see! So the bug is still present, because NET Framework 4.6 hasn't been fixed so far, right?

Regards,
Didi7
Using the most recent Veeam B&R in 19 different environments now and counting!

Dima P.
Product Manager
Posts: 10710
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » Jul 01, 2019 6:01 pm

This bug is related to .Net v4.6 but can be cured by installing the latest Windows updates.
Gostev wrote:...we were stuck thinking we're dealing with stack corruption, as only that could explain invalid function output. So we had to engage our most hardcore researchers – folks who are really good with disassembler and analyzing the machine code. And after many hours, they've nailed the issue: it appeared to be a bug in JIT (Just In Time) compilation logic of .NET Framework 4.6! A small change in the source code sent the JIT compiler on the path that contains the bug, resulting in invalid optimizations applied to the modified function and the invalid machine code created...

Didi7
Expert
Posts: 343
Liked: 36 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Didi7 » Jul 02, 2019 8:27 am

All those incidents people are reporting here don't make it very trustworthy to install 9.5 U4a (or higher versions) in environments with tape usage.

Best regards,
Didi7
Using the most recent Veeam B&R in 19 different environments now and counting!

Dima P.
Product Manager
Posts: 10710
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » Jul 03, 2019 11:41 am

Didi7,

Unfortunately the original issue reported in this thread was caught on the unpatched Windows machine and .Net framework but not by Veeam code itself. Overall every Update contains fixes for most critical reported issues to our and provides even better quality than initial release.

Didi7
Expert
Posts: 343
Liked: 36 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Didi7 » Jul 04, 2019 7:22 am

Hi Dima P.,

thanks for letting me know. There is no hurry to update to VBR 9.5 U4a. In environments, where we do not use tape storage with Veeam software, we already sucessfully use VBR 9.5 U4a.

We will wait for VBR 9.5 U4b and will test one environment with tape library usage. We always keep our VBR-servers up-to-date, so if this updated environment (VBR 9.5 U4b) keeps stable, I will rollout this version into all environments.

Regards,
Didi7
Using the most recent Veeam B&R in 19 different environments now and counting!

Dima P.
Product Manager
Posts: 10710
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » Jul 04, 2019 11:02 am

Didi7,

Sounds like a perfect plan: Update 4b is literary around the corner and you'll save some time if you go strait to U4b bypassing U4a. Since our updates are cumulative and latest version always contain fixes from previous versions you are getting all the needed fixes with one installation instead of two. Cheers!

dive7
Novice
Posts: 3
Liked: never
Joined: Mar 21, 2018 8:41 pm
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by dive7 » Aug 02, 2019 6:14 pm

Is this still an issue in 9.5u4b? Updated last night and had some issues, trying now to eject a tape and prepare the library for weekly backups. Every time I eject the tape the library shows offline in Veeam and the tape is not ejected. I am going to update my existing case # 03693853 with this question as well.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests