Tape restore failed with error - Failed to UnloadTapeFromDriveByScsi

[sasilik]

Hello

I have restored files from tape successfully with Veeam but this time I started to restored one ~13TB VM directly from tapes to VSPhere ESX host and after second tape job failed and error message was:

Code: Select all

Error    Unloading tape AIY456L6 from Drive 3 (Server: backup, Library: FUJITSU ETERNUS LT S2 5.11, Drive ID: Tape2147483639) to Slot 6 Error: Failed to call RPC function 'UnloadMediumByScsi': [MT]SCSI PASS THROUGH DIRECT ended with Scsi Error Fixed format, current;  Sense key: Illegal Request. Additional sense: Medium removal prevented by data transfer element.
Error    Restore job failed Error: [backup] [TapeService] Failed to UnloadTapeFromDriveByScsi 

After that I tried to eject tape manually but I got same error message every time. If I tried to move it from tape device management console UI then I got error message - drive media removal prevented by host.
I guess this can be fixed with serve reboot but I can do that only on weekend. And I am not sure that it guarantees that next try with restore is successful

I opened the case, #03917859, but thought to ask if anyone here has seen something similar or has any experience with it because first thing which was recommended to me was to apply hardware encryption patch https://www.veeam.com/kb2981 . I don't have that installed and I can install it at weekend but I really doubt that it makes any difference because we don't use encryption, don't have encryption enabled on drives. And the backup from which I tried to restore was done last week with same version of Veeam.

[wishr]

Hi Markko,

It's a cumulative hotfix. Usually, it's best to discuss such issues directly with our support engineers since they have much more information from the log files, etc. Could you please give their suggestion a try and let us know if it works for you.

Thanks

[Egor Yakovlev]

Hi Markko.
Since that issue could be caused by various reasons, Support suggested to install a patch to remove a known issue possibility.
I see just a few similar cases logged and most of them end up with failed hardware diagnosis. Since you cannot apply patch till weekend, try installing FUJITSU Library And Tape tool (FLATt) and see if it reports any issues on hardware part.
Thanks!

[sasilik]

I am installing the patch in weekend and try also out FLATt. From its User Guide it seems that I must stop all backup applications and services. I guess I write something back around next week.

[sasilik]

Installing the suggested patch did not change anything. I updated the case and I am going to put results also here.
I installed cumulative fix https://www.veeam.com/kb2981 on Sunday. Just in case I also rebooted both servers, VEEAM and backup server with tape agent. I tried out some commands out with FLATt(Fujitsu Library And Tape tool) like load, eject, move from slot to drive and back, some commands to get information and tape read-write test and all seemed to be OK. No errors or anything. After that I started restore job again with same tape set and as there were no other tape jobs running the restore job used another drive, not the same as before. And job again failed after data from second tape was read and VEAAM tried to eject tape.
Side note, I could eject the tape after ~24 hour waiting.
I then tried with another set of tapes but the result was same only it took this time a little longer. With earlier attempts time was ~11..12 hours but last one failed after 20 hours. And tape was again stuck in the drive and all attempts to eject it failed with same error message. I did restart the "Tape access service" on the server which is connected to tape library and after that I started to get different error messages after couple seconds when I tried to eject tape:
Unloading tape AHH785L6 from Drive 1 (Server: backup, Library: FUJITSU ETERNUS LT S2 5.11, Drive ID: Tape2147483641) to Slot 6 Error: Failed to call RPC function 'UnloadMediumByScsi': The system cannot find the file specified.
I rebooted the tape agent server and after that I could eject tape.
I have restored files and smaller VM's from tape on some occassion but it seems that I can't restore this large VM from tape.

[Egor Yakovlev]

Thanks for update!
Let's work with Support to find root cause.

[sasilik]

So far the problem hasn't been resolved. I have updated media changer drivers and TapeDeviceNotReadyTimeoutSec registry key was set to larger value, used tape server as backup proxy but the result is same when I restore large VM with "Restore directly from tape" option. Restore failed also with VM which has some 2TB disk. In same time there were no problems when restoring ~100GB VM. Also restore directly from backup repository worked, restore through a staging backup repository works and also restoring entire backup works. So it seems that there is some problem restoring large VM's directly from tape. Case is still open.

[Dima P.]

Hello Markko,

Thanks for sharing the update. I'll ask our QA folks to review the case details. Cheers!

[sasilik]

Seems that ticket is gonna be closed. Because apparently the problem is not related to VEEAM itself.
Answer - the cause of this issue is that the tape driver in Windows appears to stop responding or crash temporarily during part of the GPO update. And it causes all tape operations to fail.
So, the solution that I can suggest is to make sure that GPO is not updated during tape operations.

[Egor Yakovlev]

Thanks for the update. Glad it worked out!
/Cheers!

[sasilik]

I forgot to mention that restore job actually crashed after I started the restore job and executed "gpupdate /force" on server. And when I disabled background GP updates, restarted server, logged in and left the user logged in on the server and started restore job then this long restore job (80 hours) actually succeeded. Before that it only lasted 12..24 hours before ending with error message.

[teletek]

Veam Support - Case # 05261612 - Veeam R&B 11 Tape Backup/Restore Problemi

I am having the same problem. What should I do ? Do you have any suggestions?

[sasilik]

My problem was GP update in background and when I disabled them, https://admx.help/?Category=Windows_10_ ... oundPolicy , the problem was solved. But as mentioned here post438668.html#p438668 there can be other reason for this kind of problem.

[Dima P.]

Hello,

Thank you for sharing the update with the community Markko!

Teletek, I'd recommend to keep working with our support team to determine the root cause, unfortunately this error can be generated due to multiple infrastructure specific issues, so detailed investigation is required. Please let us know how the investigation goes!