Discussions related to exporting backups to tape and backing up directly to tape.
Didi7
Veteran
Posts: 490
Liked: 59 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Veeam B&R v9.5 U4b (Confirmations) ...

Post by Didi7 » 5 people like this post

Hello everybody,

I would like to inform you, that we had no issues with tape media and the tape library upgrading from Veeam B&R v9.5 U4 to U4b (2 weeks ago) so far. In this environment we have ...

* HPE DL180 G6 Server (QuadCore-CPU and 8GB RAM) with Windows Server 2008R2 SP1 with all Windows Updates
* HPE MSL2024 LTO-6 tape library

In the upcoming weeks we will continue to upgrade the remaining Veeam B&R 9.5 U4 servers to U4b (dissimilar HW-configurations).

All Veeam servers without tape infrastructure components are already running with Veeam B&R 9.5 U4b successfully.

Will post further results as time goes by.

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Dima P. »

Hello Didi7,

Thank you for sharing! Side note - I'd strongly recommended to install the following patch after deploying Update 4b - KB 2983 just to be safe (based on users reports standalone devices below LTO6 are affected, but I'd install this hotfix anyway). Cheers!
CPCS
Influencer
Posts: 17
Liked: 4 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by CPCS » 1 person likes this post

I just stumbled over this thread, thanks to the Newsletter!

I am struggling with "this" issues for nearly two weeks now and on Friday I finally ordered spare HBA and cabling due to Windows throwing LSI_SAS Controller Errors, HP Support kind of confirmed LTO-drive Problems and cross-checking of ports and cabling also didn’t changed anything. Due to the error didn’t Pop up right away after the upgrade and also was not constant, i could bight me in the ass not opening a Veeam ticket also....

Especially because i already spotted that Veeam was only doing Software encryption anymore, but i could really tell if it was Hardware before, because there was also the HPE HW Encryption kit for a few k€ when i set this up.

After installing the patch Hardware encryption works again and backup is running at this moment

I would like to add that we are using a library and not a standalone drive!

HP DL380p gen8 with HBA H221
HP MSL4048 SAS 2x LTO-5 (3000)
timlongbotham
Influencer
Posts: 12
Liked: 1 time
Joined: May 09, 2011 7:45 am
Full Name: Tim Longbotham
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by timlongbotham »

Lucky you Didi7! My pain continues...
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Dima P. »

Hello Tim,

Any chance you have a case ID to shared? I'd love to help you to get it resolved. Cheers!
kmbelt
Influencer
Posts: 14
Liked: never
Joined: Sep 08, 2014 4:08 pm
Full Name: Kevin Belt
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by kmbelt »

CPCS wrote: Aug 12, 2019 7:21 am I just stumbled over this thread, thanks to the Newsletter!

I am struggling with "this" issues for nearly two weeks now and on Friday I finally ordered spare HBA and cabling due to Windows throwing LSI_SAS Controller Errors, HP Support kind of confirmed LTO-drive Problems and cross-checking of ports and cabling also didn’t changed anything. Due to the error didn’t Pop up right away after the upgrade and also was not constant, i could bight me in the ass not opening a Veeam ticket also....

Especially because i already spotted that Veeam was only doing Software encryption anymore, but i could really tell if it was Hardware before, because there was also the HPE HW Encryption kit for a few k€ when i set this up.

After installing the patch Hardware encryption works again and backup is running at this moment

I would like to add that we are using a library and not a standalone drive!

HP DL380p gen8 with HBA H221
HP MSL4048 SAS 2x LTO-5 (3000)
I'm having some issues with mine also. I have DL servers at least Gen8 and a gen 9 or two hosting hp libraries.

MSL4048 with 2 x LTO6 - Can see library but drives are offline and cannot be changed
MSL2024 with 2 x LTO5 - I/O error
1/8 Autoloader with LTO4 - I/O error

All 3 quit working. Opened a ticket, but support takes a couple days to get back to me. I have done the upgrade twice now because i cannot go a week without writing to my tape. I have tried a couple different fixes they sent, but none rectified the issue for me. I'll hold out for now i guess.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Regnor »

We've also had I/O errors with different tape devices at a customer after upgrading and received a hotfix which solved the problem/errors.
I'm not sure if you're affected by the same problem, but please ask support if there's a hotfix available.
Hauke
Influencer
Posts: 22
Liked: 4 times
Joined: Apr 16, 2015 11:25 am
Full Name: Hauke Ihnen
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Hauke »

We had the same issues, tape backups not working after 4b installation.

HPE MSL4048 tape library with 2*LTO6 drives. Latest firmware's.

Looks like KB2983 fixed the issues, currently testing tape inventory. Looking good. Will test a backup job later.

[01.08.2019 12:35:40] < 2572> mt | WARN|The semaphore timeout period has expired.
[01.08.2019 12:35:40] < 2572> mt | >> |DeviceIoControl fails or is pending.
[01.08.2019 12:35:40] < 2572> mt | >> |--tr:Error code: 0x00000079
[01.08.2019 12:35:40] < 2572> mt | >> |--tr:Failed to disable hardware encryption.
[01.08.2019 12:35:40] < 2572> mt | ERR |Checksum error was detected in CommonBlockHeader, startOffset: 0, endOffset: 52.
[01.08.2019 12:35:40] < 2572> mt | >> |--tr:TapeReader failed to read media header.
[01.08.2019 12:35:40] < 2572> mt | >> |An exception was thrown from thread [2572].
Didi7
Veteran
Posts: 490
Liked: 59 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Didi7 » 1 person likes this post

Guys, I am currently in the process to upgrade 5 VBR-servers from VBR 9.5 U4 to 9.5 U4b. These are the specs of the servers ...

1) HPE DL380 Gen10 server (16-Cores-CPU and 32GB RAM) with Windows Server 2016 fully patched with MSL2024 SAS tape library, 1x LTO7-drive
2) HPE DL380 Gen9 server (32-Cores-CPU and 128GB RAM) with Windows Server 2012R2 fully patched with MSL2024 SAS tape library, 1x LTO7-drive
3) HPE DL380 G6 server (16-Cores-CPU and 48GB RAM) with Windows Server 2012R2 fully patched with MSL4048 FC tape library, 2x LTO6-drive
4) HPE DL380p G8 server (4-Cores-CPU and 64GB RAM) with Windows Server 2012R2 fully patched with MSL2024 SAS tape library, 1x LTO6-drive
5) HPE DL380 Gen9 server (48-Cores-CPU and 64GB RAM) with Windows Server 2012R2 fully patched with MSL2024 SAS tape library, 1x LTO5-drive

Pretty nice mix, if you ask me.

Will post the results next week, as soon as I have time. Last Friday of the month (tomorrow) is Active full backup time ;)

Best regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Didi7
Veteran
Posts: 490
Liked: 59 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Didi7 » 1 person likes this post

Hi everybody,

monthly fulls on all 5 upper mentioned VBR-servers went flawlessly, tape-infrastructure worked as it should!

I can successfully confirm, that the upgrade from VBR 9.5 U4 to VBR 9.5 U4b worked like a charm, all tape media is correctly located in their defined media pools!

With this last upgrade of those 5 VBR-servers, we now have 13 of 18 VBR-servers on VBR 9.5 U4b, 6 of them working flawlessly with tape libraries, 5 of them working flawlessly without any tape infrastructure.

Next step is the upgrade of the last 5 VBR-servers running VBR 9.5 U4 to VBR 9.5 U4b. Those VBR-servers have just a single tape drive (HPE LTO7) with forward forever incremental B2D-jobs and synthetic fulls with B2T-jobs.

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Dima P. »

Hello Didi7,

Thank you for updating this thread and glad to hear that all works without any issues! Cheers!
ccouchman
Novice
Posts: 3
Liked: 1 time
Joined: Sep 03, 2019 4:27 am
Full Name: Russ Couch
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by ccouchman »

Hi Folks

I upgraded our production server (Windows Server 2012R2 + MSL4048 LTO6 x 2) and both drives were disabled afterwards, tried KB 2983 but this didn't work. Logged a case and once I finally got to L2/L3 support they put in a non-public fix which worked instantly. Now on Monday after the weekend backups have completed and some tapes swapped out I have a new issue which I believe is related, the inventory of the library/drives won't update (say library state is inconsistent and repeats this until I stop the job) and no jobs will run due to various mismatch of where tapes are. Has anybody struck this? Our DR server seems fine (Windows Server 2016 + MSL4048) and is currently pulling some flat file backups to tape.

Thanks
Russ..
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by wishr »

Hi Russ,

Sad to hear you got into such a situation. I'd suggest you to continue the conversations with our support engineers to get it fully resolved. Could you please let us know your case ID?

Thanks
ccouchman
Novice
Posts: 3
Liked: 1 time
Joined: Sep 03, 2019 4:27 am
Full Name: Russ Couch
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by ccouchman »

Hi

Case # 03741776. We did some more troubleshooting tonight after struggling to get past level 1 all day and being told that the case can't be a sev1 as its for tape....we have a few jobs that go straight to tape on a daily basis that are critical to us, not happy! We tried a new database to clear out any corruptions, re-installed all tape/library drivers plus a load of other steps. Seems we need older code to enable the drive but this isn't compatible with the 4b code, some talk about developing some custom patch for this situation...but nothing back in writing.

Thanks
Russ
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Dima P. »

Hello Russ,

I've asked our T3 support team to review the case details. Stay tuned.
ccouchman
Novice
Posts: 3
Liked: 1 time
Joined: Sep 03, 2019 4:27 am
Full Name: Russ Couch
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by ccouchman » 1 person likes this post

Hi

We have a custom patch applied and everything looks back to normal. However, I did have to ring this morning to recieve this rather than receiving a case update when it was avaialble. I would have happily applied the update last night so tape jobs could run. Also, seems that their is no real urgency on tape related cases as it is seen as non-critical.

Russ
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Dima P. »

Hello Russ,

This is the first time we faced such problem and investigation took longer than expected, sorry for that. I am glad to hear that your tape jobs are up and running. Please let us know if any additional help is required. Cheers!
Didi7
Veteran
Posts: 490
Liked: 59 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Didi7 »

Ok, yesterday and today I got the following error ...

Code: Select all

9/19/2019 10:10:08 PM :: TapeDrive alert: The tape drive has a problem with the application client interface:.
  Check the cables and cable connections. Restart the operation.  
9/19/2019 10:30:17 PM :: Tape error: '1117' (The request could not be performed because of an I/O device error.) 
Tape error: '1117' (The request could not be performed because of an I/O device error.)   
9/19/2019 10:30:38 PM :: Error: Tape error: '1117' (The request could not be performed because
 of an I/O device error.) 

... with one B2T-job in one of those upper 5 mentioned environments.

The tape-library HPE MSL2024 shows no errors in its web-interface, the library is ready and the drive is idle, everything is green.

I will apply Fix_179499_7210362d26.zip now and will observe how that will work out!

Will keep you informed.

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
Didi7
Veteran
Posts: 490
Liked: 59 times
Joined: Oct 17, 2014 8:09 am
Location: Hypervisor
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Didi7 »

Oh, I noticed that the last tape media was not ejected and still in the drive. After ejecting the tape media, everything is ready and green now.

Patch KB2983 was applied.

Regards,
Didi7
Using the most recent Veeam B&R in many different environments now and counting!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by wishr »

Hi Didi,

Thank you for the information.

This is a known issue in B&R 9.5U4b. The fix you mentioned should fully resolve it.
JTT
Service Provider
Posts: 99
Liked: 2 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by JTT »

kb2983 does not help!
LTO4 - with HP 1/8 G2 Autoloader library - Tape error: '1117' (The request could not be performed because of an I/O device error.)

Case # 03795029
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Dima P. »

Hello JTT,

Please keep working with our support team as debug log analysis is required to identify why the mentioned fix is not working for you. We'll keep an eye on your case. Cheers!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Regnor »

JTT wrote: Oct 04, 2019 7:25 am kb2983 does not help!
LTO4 - with HP 1/8 G2 Autoloader library - Tape error: '1117' (The request could not be performed because of an I/O device error.)

Case # 03795029
Ask for the Hotifx 179499 which should resolve this issue.
JTT
Service Provider
Posts: 99
Liked: 2 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by JTT »

Thanks, i´ll do that. Currently the support has sent me of a couple of different fixes, i am not sure is one of it, that You mentioned or not.

But still, with VBR 4a it worked, now in 4b, tape backup is broken and it is almost third week, where no backup is taken!!
Honestly, I'm not happy at all.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by wishr »

Hi JTT,

Could you please let us know if it is fixed after applying the hotfixes provided?

Thanks in advance!
JTT
Service Provider
Posts: 99
Liked: 2 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by JTT »

Still not fixed!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by wishr »

Please keep working with our engineers to get it fully sorted. We do keep an eye on the case.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by Regnor »

Did you receive a hotfix with the ID "179499"?
Perhaps you have a completely different problem, but I've used this one on 4 tape servers with the same error message like you've receiving.
Also, if you have received more than one hotfix I hope that one doesn't revert the other...
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by wishr »

Hi JTT,

As far as I see our support engineers have managed to resolve the issue by manually applying the hotfix. We'll appreciate if you could keep us and the community posted about the job behavior going forward.

Thank you
JTT
Service Provider
Posts: 99
Liked: 2 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Veeam B&R v9.5 U4b (Confirmations) ...

Post by JTT » 1 person likes this post

Yes, finally the issue is resolved and tape backups are working!!
If anybody reads this and has issues with "LTO4 - HP 1/8 G2 Autoloader library" or "error: '1117' (The request could not be performed because of an I/O device error.)" with the tape library going offline (Job has been terminated Error: No tape libraries are online), when backups are started, then contact Veeam support and refer to case # 03795029.
Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests