Discussions related to exporting backups to tape and backing up directly to tape.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: HP LTO6 tape issues

Post by lightsout » 1 person likes this post

Can I suggest looking at this thread, for my fun with a LTO-6 library.

http://forums.veeam.com/tape-f29/spectr ... 18924.html

Specifically this bit:
• At HKLM\Software\Wow6432Node\Veeam\Backup and Replication, create DWORD value SkipTapeAlerts = 1
• Extend the polling interval at HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication\TapeDeviceWatcherTimeoutSec
• DWORD – polling interval in seconds (set to decimal 300 = five minutes)
Since then, it has been working fine. Like you though, my jobs would fail randomly whilst running.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: HP LTO6 tape issues

Post by 20100 »

Hi

I just tried Lightsout changes and unfortunately it still failed.
Then I tried Velimira changes and it failed as well.
When it does, I still can see the following error in Windows Event Viewer: The device, \Device\Tape0, is not ready for access yet.

I should have a new tape controller tomorrow for me to try. Unfortunately it was not possible to add it directly on the second bank (needs a second CPU apparently), so will have to remove a 10Gb NIC to make room for it

I will let you know how it goes.

Once again, thanks for all the contributors. It is great to see such a caring community.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: HP LTO6 tape issues

Post by lightsout »

Only other point I'd add is I found it to better with the built-in Windows drivers, then the IBM LTO drivers. Along with the registry changes I made!
EdZ
Novice
Posts: 3
Liked: 1 time
Joined: Apr 17, 2014 9:41 pm
Full Name: EdZ
Contact:

Re: HP LTO6 tape issues

Post by EdZ » 1 person likes this post

To update my post from yesterday.

Removing the HP WBEM version 10 has resolved my issue.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: HP LTO6 tape issues

Post by 20100 »

Well, last night, as I removed our 10Gb Nic card to leave space for an extra tape controller, I rebooted the server and miracle, this morning I found that the tape copy worked for the first time!

I performed another tape copy from the other job that failed last week-end and it worked too!

I do not know what the fix was, either the HP WBEM either the registry change, but it looks like a reboot was necessary to activate it properly.

I am so grateful with this community to have solve this issue with its help. Thanks all.

If I may hijack this thread a little more: for those that use LTO6, what sort of processing rate do you get?
I was expecting more compared with our LTO5 using HP Dataprotector. However I noticed that HP DP uses more agents to read the disks, so more data are written to tape at the same time. Here that might might the bottleneck as there is only 1 file to write and the tape must be waiting for more data.
Saying that Veeam thinks the bottleneck is the target (tape drive) at 99%, so not too sure.

Thanks
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

Hi,

I've got an HP MSL2024 Tape Library connected via 8gb Fibre and I get 'Processing Rate' anywhere from between 150 MB/s to 80 MB/s..

Not too sure why its so inconsistent !! Seems like in the middle of the job, the 'throughput' remains almost 0 for many minutes before intermittently kicking in again !

Image

Image
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: HP LTO6 tape issues

Post by tsightler »

20100 wrote:If I may hijack this thread a little more: for those that use LTO6, what sort of processing rate do you get?
I was expecting more compared with our LTO5 using HP Dataprotector. However I noticed that HP DP uses more agents to read the disks, so more data are written to tape at the same time. Here that might might the bottleneck as there is only 1 file to write and the tape must be waiting for more data.
Saying that Veeam thinks the bottleneck is the target (tape drive) at 99%, so not too sure.
Since Veeam data is already compressed and there is just a single big file it should be pretty easy for a single agent to stream at the full tape speed, which for LTO-6 is around 160MB/s. I would expect real world speeds in the 130-160MB/s range assuming everything else is optimal. Some people are used to seeing speeds of 300MB/s in other software, but that's writing data that is not already compressed. Since an LTO-6 tape device can do 2.5:1 compression in hardware it is capable of receiving uncompressed data at around 400MB/s (160MB/s x 2.5 compression = 400MB/s), but Veeam backups are generally already compressed at similar levels and thus you'll be much closer to the actual native speed instead.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

but Veeam backups are generally already compressed at similar levels and thus you'll be much closer to the actual native speed instead.
then why is it so intermittent in my case ? Why does the backup get paused intermittently while transferring data to tape ?
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

I'm doing a backup as i type this and again.. the throughput has dropped to 0kb for the last half hour !!

Can someone please shed some light on this ? What logs do i check to see whats going on ?

Image
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: HP LTO6 tape issues

Post by tsightler »

jazzoberoi wrote:I'm doing a backup as i type this and again.. the throughput has dropped to 0kb for the last half hour !!

Can someone please shed some light on this ? What logs do i check to see whats going on ?
I don't know that this is the type of problem that can be resolved via the forum as it's pretty difficult to understand what's going on. Are there other backups running during this time as well, or just the backup to tape? I certainly wouldn't expect that behavior if there's nothing else going on.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: HP LTO6 tape issues

Post by 20100 »

Hi Tom

Thanks for reply. Yes this is the type of speed I am getting, so it looks we are OK.
Obviously our previous backup solution does backup non compressed data which explain the speed difference.
It has also been confirmed with the amount of data on tape, which you already explained to me in a different thread.

Thanks again
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

tsightler wrote: I don't know that this is the type of problem that can be resolved via the forum as it's pretty difficult to understand what's going on. Are there other backups running during this time as well, or just the backup to tape? I certainly wouldn't expect that behavior if there's nothing else going on.
No other backup is running. Checked the Task Manager and everything is normal, CPU is below 20% and plenty of RAM available..
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

Hi, opened up a case # 00829368. Lets see what comes out of it !
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: HP LTO6 tape issues

Post by tsightler »

jazzoberoi wrote:Hi, opened up a case # 00829368. Lets see what comes out of it !
I will follow the case as well as I'm definitely interested in what is causing this, it's certainly not something I would expect. In the vendor testing that I've done we've generally seen quite consistent performance for large files. The only time I ever see anything like what your posting is when backing up directories that have lots of smaller files but that doesn't seem to be the case in your screenshot. It will be very interesting to see what support has to say.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

Hi Tsightler,

Could you please get someone to provide an update to the case ? I've already uploaded the logs as requested.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

Ok.. got a reply from support team..
Hi, Jazz
Thanks for testing and provided the logs.

The first job it takes about 3 hours, the second job take about two and half hours , the difference between these two jobs are half hours.

The bottleneck is showing below

First job: Load: Source 8% > Proxy 13% > Network 11% > Target 99%
Second job: Load: Source 13% > Proxy 21% > Network 14% > Target 98%
Both bottlenecks are target, which is your process rate will depend on the target, 1% writting speed difference is acceptable.
You can also see where the proxy and network are on different load each time, where you have same backup size.
Seriously, !?

I know the difference between 2.5 and 3hrs is 'half hour'.. and i also know that the difference between 98% and 99% is 1%. I also know from the processing stats that its showing the Tape to be the bottleneck.

I did not lodge a support case and upload all the logs to only be told the obvious !?

What i want to know is why is there extended periods of absolute no activity ? Why is the data processing rate so low when everything is connected to Fibre and that Tape Drive is capable of handling speeds upto 160MB/s with close to 20 brand new LTO6 tapes. And why am i sometimes able to complete the job in 1.5 hours and sometimes it takes more than 3 hrs !?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: HP LTO6 tape issues

Post by Dima P. »

Hi Jazz,
Tape Drive is capable of handling speeds upto 160MB/s with close to 20 brand new LTO6 tapes. And why am i sometimes able to complete the job in 1.5 hours and sometimes it takes more than 3 hrs !?
We are aware of this problem and doing our best to get the performance improved. Specifically, we want to make the throughput more stable, and eliminate this “spiky’ issue in the next major releases. By the way, thank you for the logs - it helps a lot in our improvement investigations.

Regarding the support case handling, use ‘talk to the manager’ option in the customer portal anytime, whenever you feel it's needed.
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi » 1 person likes this post

Thanks d.popov and Tsightler..

Now i feel like my case is in the right hands :)

Its also good to know that its a 'known issue' and is under investigation. Otherwise the worst part of lodging a support case in most cases is proving that there is indeed an issue, atleast now half the battle is won :)

Please let me know if you need me to upload any more logs to help you with your investigations... any indication on progress or eta will be highly appreciated !

Regarding the ‘talk to the manager’ option, i wasn't aware of such a function, but i wont need to use it anymore :)
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi »

Hi Guys,

Just received some new communication from support regarding my case.. Just wanted to know whether or not I'm still dealing with the same support person and should do as he says or wait for instructions from either Tom or d.popov ..
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: HP LTO6 tape issues

Post by Dima P. »

Hi Jazz,
Yes, please, keep working with the support – I’ve forwarded you case to QA and DEV teams: as I’ve posted before this seems like an issue we need to fix in the product. I’ll follow up on Monday once I get any response. Thanks!
jazzoberoi
Enthusiast
Posts: 96
Liked: 23 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: HP LTO6 tape issues

Post by jazzoberoi » 1 person likes this post

Hi D.popov..

Finally some good news :)

After following the KB1887 - http://www.veeam.com/kb1887 and creating the DWORD "TapeDeviceWatcherTimeoutSec" with a value "300" , my problem was fixed.

Will this issue be fixed in Patch 2 ?

Image
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: HP LTO6 tape issues

Post by Dima P. »

Awesome - glad to hear, thanks Jazz!

Based on your case, I think we need to review the default timeout we are using now. However, its not going to be changed in the upcoming VBR update – more likely in the major release, just to avoid any collateral damage such correction may bring.
Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests