Veeam B&R v9.5 U4b (Confirmations) ...

[Didi7]

Hello everybody,

I would like to inform you, that we had no issues with tape media and the tape library upgrading from Veeam B&R v9.5 U4 to U4b (2 weeks ago) so far. In this environment we have ...

* HPE DL180 G6 Server (QuadCore-CPU and 8GB RAM) with Windows Server 2008R2 SP1 with all Windows Updates
* HPE MSL2024 LTO-6 tape library

In the upcoming weeks we will continue to upgrade the remaining Veeam B&R 9.5 U4 servers to U4b (dissimilar HW-configurations).

All Veeam servers without tape infrastructure components are already running with Veeam B&R 9.5 U4b successfully.

Will post further results as time goes by.

Regards,
Didi7

[Dima P.]

Hello Didi7,

Thank you for sharing! Side note - I'd strongly recommended to install the following patch after deploying Update 4b - KB 2983 just to be safe (based on users reports standalone devices below LTO6 are affected, but I'd install this hotfix anyway). Cheers!

[CPCS]

I just stumbled over this thread, thanks to the Newsletter!

I am struggling with "this" issues for nearly two weeks now and on Friday I finally ordered spare HBA and cabling due to Windows throwing LSI_SAS Controller Errors, HP Support kind of confirmed LTO-drive Problems and cross-checking of ports and cabling also didn’t changed anything. Due to the error didn’t Pop up right away after the upgrade and also was not constant, i could bight me in the ass not opening a Veeam ticket also....

Especially because i already spotted that Veeam was only doing Software encryption anymore, but i could really tell if it was Hardware before, because there was also the HPE HW Encryption kit for a few k€ when i set this up.

After installing the patch Hardware encryption works again and backup is running at this moment

I would like to add that we are using a library and not a standalone drive!

HP DL380p gen8 with HBA H221
HP MSL4048 SAS 2x LTO-5 (3000)

[timlongbotham]

Lucky you Didi7! My pain continues...

[Dima P.]

Hello Tim,

Any chance you have a case ID to shared? I'd love to help you to get it resolved. Cheers!

[kmbelt]

I just stumbled over this thread, thanks to the Newsletter!

I am struggling with "this" issues for nearly two weeks now and on Friday I finally ordered spare HBA and cabling due to Windows throwing LSI_SAS Controller Errors, HP Support kind of confirmed LTO-drive Problems and cross-checking of ports and cabling also didn’t changed anything. Due to the error didn’t Pop up right away after the upgrade and also was not constant, i could bight me in the ass not opening a Veeam ticket also....

Especially because i already spotted that Veeam was only doing Software encryption anymore, but i could really tell if it was Hardware before, because there was also the HPE HW Encryption kit for a few k€ when i set this up.

After installing the patch Hardware encryption works again and backup is running at this moment

I would like to add that we are using a library and not a standalone drive!

HP DL380p gen8 with HBA H221
HP MSL4048 SAS 2x LTO-5 (3000)

I'm having some issues with mine also. I have DL servers at least Gen8 and a gen 9 or two hosting hp libraries.

MSL4048 with 2 x LTO6 - Can see library but drives are offline and cannot be changed
MSL2024 with 2 x LTO5 - I/O error
1/8 Autoloader with LTO4 - I/O error

All 3 quit working. Opened a ticket, but support takes a couple days to get back to me. I have done the upgrade twice now because i cannot go a week without writing to my tape. I have tried a couple different fixes they sent, but none rectified the issue for me. I'll hold out for now i guess.

[Regnor]

We've also had I/O errors with different tape devices at a customer after upgrading and received a hotfix which solved the problem/errors.
I'm not sure if you're affected by the same problem, but please ask support if there's a hotfix available.

[Hauke]

We had the same issues, tape backups not working after 4b installation.

HPE MSL4048 tape library with 2*LTO6 drives. Latest firmware's.

Looks like KB2983 fixed the issues, currently testing tape inventory. Looking good. Will test a backup job later.

[01.08.2019 12:35:40] < 2572> mt | WARN|The semaphore timeout period has expired.
[01.08.2019 12:35:40] < 2572> mt | >> |DeviceIoControl fails or is pending.
[01.08.2019 12:35:40] < 2572> mt | >> |--tr:Error code: 0x00000079
[01.08.2019 12:35:40] < 2572> mt | >> |--tr:Failed to disable hardware encryption.

[01.08.2019 12:35:40] < 2572> mt | ERR |Checksum error was detected in CommonBlockHeader, startOffset: 0, endOffset: 52.
[01.08.2019 12:35:40] < 2572> mt | >> |--tr:TapeReader failed to read media header.
[01.08.2019 12:35:40] < 2572> mt | >> |An exception was thrown from thread [2572].

[Didi7]

Guys, I am currently in the process to upgrade 5 VBR-servers from VBR 9.5 U4 to 9.5 U4b. These are the specs of the servers ...

1) HPE DL380 Gen10 server (16-Cores-CPU and 32GB RAM) with Windows Server 2016 fully patched with MSL2024 SAS tape library, 1x LTO7-drive
2) HPE DL380 Gen9 server (32-Cores-CPU and 128GB RAM) with Windows Server 2012R2 fully patched with MSL2024 SAS tape library, 1x LTO7-drive
3) HPE DL380 G6 server (16-Cores-CPU and 48GB RAM) with Windows Server 2012R2 fully patched with MSL4048 FC tape library, 2x LTO6-drive
4) HPE DL380p G8 server (4-Cores-CPU and 64GB RAM) with Windows Server 2012R2 fully patched with MSL2024 SAS tape library, 1x LTO6-drive
5) HPE DL380 Gen9 server (48-Cores-CPU and 64GB RAM) with Windows Server 2012R2 fully patched with MSL2024 SAS tape library, 1x LTO5-drive

Pretty nice mix, if you ask me.

Will post the results next week, as soon as I have time. Last Friday of the month (tomorrow) is Active full backup time

Best regards,
Didi7

[Didi7]

Hi everybody,

monthly fulls on all 5 upper mentioned VBR-servers went flawlessly, tape-infrastructure worked as it should!

I can successfully confirm, that the upgrade from VBR 9.5 U4 to VBR 9.5 U4b worked like a charm, all tape media is correctly located in their defined media pools!

With this last upgrade of those 5 VBR-servers, we now have 13 of 18 VBR-servers on VBR 9.5 U4b, 6 of them working flawlessly with tape libraries, 5 of them working flawlessly without any tape infrastructure.

Next step is the upgrade of the last 5 VBR-servers running VBR 9.5 U4 to VBR 9.5 U4b. Those VBR-servers have just a single tape drive (HPE LTO7) with forward forever incremental B2D-jobs and synthetic fulls with B2T-jobs.

Regards,
Didi7

[Dima P.]

Hello Didi7,

Thank you for updating this thread and glad to hear that all works without any issues! Cheers!

[ccouchman]

Hi Folks

I upgraded our production server (Windows Server 2012R2 + MSL4048 LTO6 x 2) and both drives were disabled afterwards, tried KB 2983 but this didn't work. Logged a case and once I finally got to L2/L3 support they put in a non-public fix which worked instantly. Now on Monday after the weekend backups have completed and some tapes swapped out I have a new issue which I believe is related, the inventory of the library/drives won't update (say library state is inconsistent and repeats this until I stop the job) and no jobs will run due to various mismatch of where tapes are. Has anybody struck this? Our DR server seems fine (Windows Server 2016 + MSL4048) and is currently pulling some flat file backups to tape.

Thanks
Russ..

[wishr]

Hi Russ,

Sad to hear you got into such a situation. I'd suggest you to continue the conversations with our support engineers to get it fully resolved. Could you please let us know your case ID?

Thanks

[ccouchman]

Hi

Case # 03741776. We did some more troubleshooting tonight after struggling to get past level 1 all day and being told that the case can't be a sev1 as its for tape....we have a few jobs that go straight to tape on a daily basis that are critical to us, not happy! We tried a new database to clear out any corruptions, re-installed all tape/library drivers plus a load of other steps. Seems we need older code to enable the drive but this isn't compatible with the 4b code, some talk about developing some custom patch for this situation...but nothing back in writing.

Thanks
Russ

[Dima P.]

Hello Russ,

I've asked our T3 support team to review the case details. Stay tuned.

[ccouchman]

Hi

We have a custom patch applied and everything looks back to normal. However, I did have to ring this morning to recieve this rather than receiving a case update when it was avaialble. I would have happily applied the update last night so tape jobs could run. Also, seems that their is no real urgency on tape related cases as it is seen as non-critical.

Russ

[Dima P.]

Hello Russ,

This is the first time we faced such problem and investigation took longer than expected, sorry for that. I am glad to hear that your tape jobs are up and running. Please let us know if any additional help is required. Cheers!

[Didi7]

Ok, yesterday and today I got the following error ...

Code: Select all

9/19/2019 10:10:08 PM :: TapeDrive alert: The tape drive has a problem with the application client interface:.
  Check the cables and cable connections. Restart the operation.  
9/19/2019 10:30:17 PM :: Tape error: '1117' (The request could not be performed because of an I/O device error.) 
Tape error: '1117' (The request could not be performed because of an I/O device error.)   
9/19/2019 10:30:38 PM :: Error: Tape error: '1117' (The request could not be performed because
 of an I/O device error.) 

... with one B2T-job in one of those upper 5 mentioned environments.

The tape-library HPE MSL2024 shows no errors in its web-interface, the library is ready and the drive is idle, everything is green.

I will apply Fix_179499_7210362d26.zip now and will observe how that will work out!

Will keep you informed.

Regards,
Didi7

[Didi7]

Oh, I noticed that the last tape media was not ejected and still in the drive. After ejecting the tape media, everything is ready and green now.

Patch KB2983 was applied.

Regards,
Didi7

[wishr]

Hi Didi,

Thank you for the information.

This is a known issue in B&R 9.5U4b. The fix you mentioned should fully resolve it.

[JTT]

kb2983 does not help!
LTO4 - with HP 1/8 G2 Autoloader library - Tape error: '1117' (The request could not be performed because of an I/O device error.)

Case # 03795029

[Dima P.]

Hello JTT,

Please keep working with our support team as debug log analysis is required to identify why the mentioned fix is not working for you. We'll keep an eye on your case. Cheers!

[Regnor]

kb2983 does not help!
LTO4 - with HP 1/8 G2 Autoloader library - Tape error: '1117' (The request could not be performed because of an I/O device error.)

Case # 03795029

Ask for the Hotifx 179499 which should resolve this issue.

[JTT]

Thanks, i´ll do that. Currently the support has sent me of a couple of different fixes, i am not sure is one of it, that You mentioned or not.

But still, with VBR 4a it worked, now in 4b, tape backup is broken and it is almost third week, where no backup is taken!!
Honestly, I'm not happy at all.

[wishr]

Hi JTT,

Could you please let us know if it is fixed after applying the hotfixes provided?

Thanks in advance!

[JTT]

Still not fixed!

[wishr]

Please keep working with our engineers to get it fully sorted. We do keep an eye on the case.

[Regnor]

Did you receive a hotfix with the ID "179499"?
Perhaps you have a completely different problem, but I've used this one on 4 tape servers with the same error message like you've receiving.
Also, if you have received more than one hotfix I hope that one doesn't revert the other...

[wishr]

Hi JTT,

As far as I see our support engineers have managed to resolve the issue by manually applying the hotfix. We'll appreciate if you could keep us and the community posted about the job behavior going forward.

Thank you

[JTT]

Yes, finally the issue is resolved and tape backups are working!!
If anybody reads this and has issues with "LTO4 - HP 1/8 G2 Autoloader library" or "error: '1117' (The request could not be performed because of an I/O device error.)" with the tape library going offline (Job has been terminated Error: No tape libraries are online), when backups are started, then contact Veeam support and refer to case # 03795029.