VTL - Unable to start tape backup session: StartSessionTape

[stevenrodenburg1]

Hello,

I've build a Lab with the QuadStor 2.2.15 VTL Software on a RedHat 6 box. I emulate a HP ESL9000 with four Ultrium 960 LTO3 drives and 12 vCartridges. I connect to the B&R Master VM via iSCSI. It's a Win2008r2 VM.
I loaded the HP drivers that were suggested on this website: http://www.v-front.de/2013/11/testing-v ... -with.html
I configured QuadStor to not have import/export slots. I have 12 "real" slots and all have a virtual tape in them.

I'm using Veeam B&R 7 version: 7.0.0.839

I configured iSCSI. The ESL and it's drives show up in the Windows Device Manager and i loaded those drivers. All looks fine.
In Veeam, I can erase, load, eject and play with the vTapes to my heart's content. When i load a tape in a drive, it is reflected accordingly in the QuadStor Web GUI etc. etc. It all works fabulously.

Then I created a single (test) tape job with 3 backup-jobs in it. Each backup-job has 1 VM in it so Veeam shows 3 VM's as pending when the job starts. When i start the job, the first VM is getting backed-up and the other two are "pending". The first VM is finished succesfully, the tape is unloaded from the drive and all is fine. Then it's the second VM's turn, which always goes wrong. The next two backups (of the two VM's that were pending) fail with "Unable to start tape backup session: StartSessionTape failed".

In short: the first VM inside a Job always works. The second, third etc. etc. VM's inside a tape-job always fail with the error mentioned above.

Looking in the Logs, the moment the above error shows up in the GUI, the following error is logged:

Code: Select all

[13.03.2014 22:20:12] <  5952>      ERR |Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).

[13.03.2014 22:20:12] <  5952>      >>  |) 
[13.03.2014 22:20:12] <  5952>      >>  |--tr:Cannot start new tape backup session 'Backup set 13.03.2014 22:20:02 created' for tape 'Tape0'.
[13.03.2014 22:20:12] <  5952>      >>  |--tr:Client failed to process the command. Command: [startSessionTape].
[13.03.2014 22:20:12] <  5952> cli| Starting to send error string [Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).

[13.03.2014 22:20:12] <  5952> cli| ) 
[13.03.2014 22:20:12] <  5952> cli| --tr:Cannot start new tape backup session 'Backup set 13.03.2014 22:20:02 created' for tape 'Tape0'.
[13.03.2014 22:20:12] <  5952> cli| --tr:Client failed to process the command. Command: [startSessionTape].] to manager.
[13.03.2014 22:20:12] <  5952> cli| Sending error string [Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).

[13.03.2014 22:20:12] <  5952> cli| ) 
[13.03.2014 22:20:12] <  5952> cli| --tr:Cannot start new tape backup session 'Backup set 13.03.2014 22:20:02 created' for tape 'Tape0'.
[13.03.2014 22:20:12] <  5952> cli| --tr:Client failed to process the command. Command: [startSessionTape].] completed.

I read about some libraries needing "exlusive" or "non exclusive" drivers. As i have perfect control over the all drives, loading and unloading tapes etc. etc. and as the first VM in a job always works, i don't suspect my drivers are wrong. There are no other entities locking drives as far as i can see.


I've been fighting this issue for 2 days now. I must be overlooking or misunderstanding something but what?

[veremin]

Hi,Steven, are there any additional options specified in the settings of given device? Compression, encryption, etc.? I'm wondering, since we saw some cases before where VTL devices with per-configured specific settings reacted inappropriately to VB&R activity.

Thanks.

[stevenrodenburg1]

Hi,

Well there are the drive properties like compression etc. that are visible in the Windows Device Manager. They are set to their default values and as the fields are all greyed out, they cannot be changed. In the Veeam Tapejob, i did not enable compression for example.

Thing is, the first VM always works, regardless of how large it is. The ones that come after always fail so maybe something goes wrong with controlling the drive after the first job finishes, rendering it unusable for subsequent jobs ?

[veremin]

They are set to their default values and as the fields are all greyed out, they cannot be changed.

I was talking more about VTL level. During device creation, for instance, did specify any additional settings, like encryption, hardware compression, etc.?

The ones that come after always fail so maybe something goes wrong with controlling the drive after the first job finishes, rendering it unusable for subsequent jobs ?

This was my first assumption, as well. But the logic behind drive control/media change is pretty simple, so, I doubt this is the major reason.

Anyway, if you can't find the root cause yourself, you can always ask our support team for assistance.

Thanks.

[Dima P.]

Hello Steven,
I suggest you installing the latest Quadstor VTL build 3.0.64, as this one was set up in our lab for testing purposes (we use it one SUSE), and it works great.

[stevenrodenburg1]

Hi,

Quadstor VTL build 3.0.64 does not exist. The latest version of VTL is build 2.2.15 which i have. You are referring to their Storage Virtualization product which is 3.0.64.

[quadstor]

You might want to send us an email at vtlsupport at quadstor dot com

The key here is the error [13.03.2014 22:20:12] < 5952> ERR |Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check). which you are seeing.

It seems that after your first backup, and when the next backup is suppose to start there is an error in positioning to the postiion where the next write to tape will happen.

While we don't use Veeam for testing, we could check if there is anything the VTL needs to address/fix

[stevenrodenburg1]

Hello QuadStor,

Thank you for responding. I've send an Email with a link to this Post (no need to tell the whole story again )


Kind regards,
Steven Rodenburg

[Dima P.]

Steven,
Thank you for pointing this out! We will test the VTL build you are referring as well. Meanwhile I hope Quadstor support could help you!
Cheers.

[thomas.grimm]

Hello Steven,

do you already have a solution for the problem. We have exactly the same error after installing Patch 3 for Veeam BR.
We use Quadstor VTL 2.2.15, emulating a HP StorageWorks ESL9000.

Many thanks for a short Information

[quadstor]

Hi Thomas,

Please use the new version of the VTL (2.2.16). We did have an issue in the VTL code when spacing to the end of data. Possibly patch 3 exposed this issue. 2.2.16 seems to fix this issue and confirmed to us by Steven. Many thanks to Steven for reporting this !

[stevenrodenburg1]

Hi Thomas,

Yes, i was just about to post the solution. I reported "success" back to quadstor support this morning.

Solution: it is an issue in the 2.2.15 release. quadstor released version 2.2.16 which fixes the problem. I now have that VTL with four LTO-3 drives pumping data like there is no tommorow.

The quadstor support folks have been great.


I have one issue remaining but that is not a quadstor problem. Every now and then, An entire ToTape backup-job ends with a bogus "Failed status" even though all the data has been backed-up just fine (every VM lists as "succesfull" and there is no error to be seen anywhere). Wierd..
When it happens, i get a message "Unable to backup: Error: SSH Channel error: open failed, answer:" directly after the last of the "New storage <path to .vbi or vbk> was added to backup list" lines, and before the "Processing <VM Name> full pass" lines in the Job status GUI.


Edit 11:45 - The quadstor folks were faster

[stevenrodenburg1]

Tip: I had problems while upgrading the RPM packages from 2.2.15 to 2.2.16. Upgrading from 2.2.15 to 2.2.16 via "rpm –U" did not work.
It worked fine for the core package, but the installation of the itl package bombed out everytime with some module build error.

However, as the VTL config files are not touched during an un-install (they live in a folder outside the scope of the RPM packages), i used that principle to completely de-install 2.2.15 and install 2.2.16 "fresh".
The new version picked up the configuration file, my configuration was loaded without problems and everything worked fine since then.

I got a reply from quadstor on this and it confirms my findings:

rpm -U is untested and will have problems. The recommended procedure is rpm -e and then followed by rpm -i. Reboot of the system after the uninstall is not required unless a kernel upgrade is performed. Rebooting the system after the install is not required unless FC is used.