HP Lefthand VSS Provider Import Snapshot Error

[ArturSchneider]

Case [ID#00667058]

Hello,

i have a big Problem with my Hyper V 2012 R2 and Lefthand P4500 Environment .

My Backup Server is a 2012 R2 physical Server, Hyper-V Role is activated, HP Application Aware Snapshot (Hardware VSS Provider) installed (on Hyper V Hosts too) and he is a Offhost Backup Proxy.
The Hyper V Cluster contains two Hyper V Hosts with Windows Server 2012 R2. I also connected the CSV Luns over iSCSI to the Veeam Server (Offhost Proxy) but the Backup Jobs finish with:
"Unable to create snapshot LeftHand Networks VSS Provider on Hyper-V Offhost Backup Proxy (mode: Crash consistent). Details: Failed to import snapshot: Import snapshot task has failed. Vss error: '0x80042320' "

On the Storage CMC Console i see that one of the Hyper V Server creates a Snapshot but it disappears when the Job Finishs with error.

"
Hope someone can help me.

Best Regards

Artur

EDITED: job log has been removed by a moderator as per forum rules.

[JaYaKaAzZ]

No One with same Issue ???

[veremin]

Hi Artur, it seems that your case is still open and under investigation. So, please keep working with support team on addressing the said problem. Thanks.

[Vitaliy S.]

Can you create a hardware VSS snapshot on the Hyper-V host itself and not on the off-host proxy server?

[JaYaKaAzZ]

Hello Vitaly,

yes already tested it on the Hype-Host itself and it worked fine.

[Vitaliy S.]

Artur, As far as I remember you have to configure access list to the Hyper-V server (acting as an off-host proxy) in HP storage management console. Basically, both production host and off-host proxy should have identical hardware VSS configuration, in this case there should not be any problems with snapshot creation.

[JaYaKaAzZ]

Vitaly, what you mean with Access list to the Hyper-V Server ?

[Vitaliy S.]

In the HP management console did you give read-only access to the LUNs to your off-host proxy server? Did you configure off-host proxy in the same way as production host?

[JaYaKaAzZ]

Yes i did, actually it looks like its a Problem with the HP Lefthand Hardware Provider ... Veeam Supporter will have some testings tomorrow with his colleagues from TK2... than we decide if its suggestive to take HP colleagues in this call...

i will continue ...

[Vitaliy S.]

I would recommend checking VSS providers status via command line in that server. Additionally, while you're waiting take a look at Windows Event log, might be helpful.

[JaYaKaAzZ]

vss Provider Status Looks alright :/ ... ty will check the Event log

[JaYaKaAzZ]

So here is a Update to my Call..

Supporter made same Tests as before with same result... manually making HP VSS Provider Snapshots from Hyper-V works fine... with Veeam B&R its just not working ...

Call will be escelated to TK 2 ... hope this Guy will find the Issue...

Just found an old Article where someone had the same Issue with HP SAN ... but without Results ...

http://forums.veeam.com/microsoft-hyper ... 14879.html

Dont know if this is an Veeam or an HP Issue but i have to say with Backup Exec the HP VSS Provider worked fine ... Maybe there is another mechanism but at this Call State (after 1 month now) iam very disappointed with the Veeam Support... because there is no Information where the Problem is located ...

[Vitaliy S.]

manually making HP VSS Provider Snapshots from Hyper-V works fine... with Veeam B&R its just not working ...

Manual snapshot creating should work just fine, it seems like there is an issue with snapshot transfer, let's see what our T2 engineer will find out after reviewing the debug information.

Details: Failed to import snapshot: Import snapshot task has failed. Vss error: '0x80042320'

[JaYaKaAzZ]

So here is the final Update to my Call...

At first i want to thank Denis Bozhkov and Nicolay Sergeev the T2 engineers who found the solution for our Problem! Really great Job !

So here is the Problem:
In the CMC of the HP SAN the HyperV Servers where all in a Cluster Group which was assigned to the LUN´s. We made a new single entry for the Proxy Server and assigned the LUN´s too and thats the major issue. U have to add the Proxy Server to the Cluster Group and than the Direct SAN Backup will work fine with the Lefthand Hardware Provider. I know sounds very strange but its the solution !

Hope this gonna help someone with same Issue

Best Regards
Artur Schneider

[frankive]

If you add the off-host proxy to the same LUN, will you be able to set only read-access to the LUNs for the off-host proxy?

[JaYaKaAzZ]

Didnt test that Frank... but its a good question

[lowlander]

So here is the final Update to my Call...

U have to add the Proxy Server to the Cluster Group and than the Direct SAN Backup will work fine with the Lefthand Hardware Provider. I know sounds very strange but its the solution !

Thanks for this solution. It indeed works. We added the off host proxy to the cluster group and the backup is working again.

However, all CSV volumes are presented read/write to the off host proxy.

We followed the following advises:
- san policy=OfflineAll
- automount disable
- automount scrub

This is working perfectly and settings remain after a reboot. However when we start a backup that is using the off host proxy, the automount setting is being enabled again. When rebooting the off host proxy this is quite undiserable regarding the CSV volumes being in read write mode.

I opened a support case 00935459

Anyone experienced similar behaviour ? Offhost proxy is Windows 2008 R2

[lowlander]

I monitored the backup processes and apparently the process VeeamHvIntegrationSvc.exe initiates registry set command on the key NoAutoMount=0. As a workaround we wrote a post backup script to set this value to 1 after a backup. Also created a computer policy startup / shutdown to disable automount.

We wait now on support regarding the official statement regarding this issue. Above is a workaround for now.

[Vitaliy S.]

I have discussed this behavior with our R&D team and automount is, indeed, set to enabled mode on the Hyper-V off-host proxy server. This is required by backup job VSS process, however if san policy is configured as OfflineAll, then CSV should not be mounted with a drive letter. Let's see what our support engineer can find out after looking through the logs.

[Vitaliy S.]

@lowlander, do you see the same behavior on any other off-host proxy server?

[lowlander]

Hi,

we only have one off-host proxy. We looked into the SAN policy in combination with the iSCSI volumes. When enabling SAN policy "offline all", the iSCSI volumes are booted in an online state. When enabling SAN policy "offline shared" iSCSI volumes are listed in disk manager as offline (as expected). I will try to simulate this behaviour in our lab, when time is available

* SAN policy we configure in diskpart

[Vitaliy S.]

Hmm... that's weird, cause offline all should also keep new volumes from initialization. I bet you will not see this behavior on another Windows box.

[lowlander]

Quick update. When Veeam is backing up a Hyper-V VM the snapshot volume is mounted with a drive letter. When backup finishes, the disk is removed from diskmgmt.msc. So basically even when SAN policy is set to "offline shared" mode the disk is configured as online disk (with a drive letter). Disk is listed as RAW volume.

[Vitaliy S.]

Under normal conditions drive letter assignment should not be happening, so I wonder whether these volumes have been initialized previously or not, cause SAN policy should definitely help here.

[lowlander]

Hi,

I captured this in a mp4 and updated it with case 00949962