Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

[jorgene]

Hi,
Just wondering if anyone else have had or seen this behaviour.
I have latest Veeam 8.0.0.917 installed.
2-node W2012 R2 Hyper-V Cluster and 2 HP StoreVirtual 4330 SAN nodes for shared storage (Running latest SAN IQ 11.5.00.0673.0 fully patched). Also latest DSM for MPIO 11.5.0.345 and VSS Provider 11.5.0.417 installed on Hyper-V hosts.

Now after Veeam Backups (sometimes, not always) Snapshots on the P4330 units created during veeam backup (thin temp volumes) still remains on the unit and don't get deleted after backup finish.
If I check the Hyper-V host in ISCSI I have a disconnected ISCSI session to the temp space LUN. After further investigation of logfiles I see that I get "Faulting application name: LhnVssProvider.exe, version: 11.5.0.417" "Faulting module name: MSVCR90.dll, version: 9.0.30729.8387" errors. So it seems like the Application Aware VSS Provider crashes, that's probably why I get temp volumes left on the SAN node. This seems to happen when it's time to delete the snapshot, since all backup Job works fine (until temp space volumes fills up the SAN node). I need to manually remove snapshots in CMC.

It seems like I have to put a case with HP about this since I cannot see that Veeam are causing this (if not putting some strange commands to HP VSS Provider causing the crash).

[foggy]

Yes, in case of VSS provider failure it is more appropriate to contact provider vendor. You can try to reproduce the behavior without Veeam B&R in presence, using diskshadow command, just to make sure it is not involved.

[jorgene]

I've opened a case with HP now and will be sending them a bunch of logfiles to analyze. Are there any way to make Veeam "Skip" using Lefthand VSS Provider and use Microsoft Shadow Copy instead?
Except uninstalling HP Lefthand VSS Provider software on the Cluster node?

[foggy]

You an select the provider to be used in the volume-specific settings of the host.

[jorgene]

Great, thx.

[jorgene]

HP case escalated to lvl 3 since they cannot see whats causing the crashes. After switching to Microsoft CSV ShadowCopy problem gone. But it puts more load on Hyper-V hosts during backup.

[foggy]

Thanks for the update. Keep us posted on how the investigation goes.

[TheDonMiguel]

Would be interesting to hear more about this. I'm facing exactly the same issue with my LeftHand Storage

[jorgene]

Hi, TheDonMiguel please report this issue to HP also. As they are aware i'm the only one having this issue, reported. Now they force me to wait until the patch 12 is released for SAN IQ. Should be released in a few weeks, i didn't get an ETA from them yet. If i still have this issue after that patch is applied they will dig deeper.

[jorgene]

Hi all, SAN IQ ver 12.0 released just a few days ago.

I have installed and upgraded SAN IQ, VSS Provider and MPIO. But still same issue, 2 backups went fine, on the 3rd the VSS provider crashes again on Hyper-V host leaving snapshot on the LUN.

Code: Select all

Faulting application name: LhnVssProvider.exe, version: 12.0.0.507, time stamp: 0x54baa84b
Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea1bbd
Exception code: 0xc0000005
Fault offset: 0x000000000003e1e8
Faulting process id: 0x790
Faulting application start time: 0x01d0529ba5a70be1
Faulting application path: C:\Program Files\HP\StoreVirtual\Application Aware Snapshot Manager\LhnVssProvider.exe
Faulting module path: C:\windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\MSVCR90.dll
Report Id: 6375bfef-bef7-11e4-80d2-40a8f0361da0
Faulting package full name: 
Faulting package-relative application ID: 

I will see what HP says, but they suspects Veeam is causing it they told me.
But we will se what support says after my information.

[jorgene]

Veeam case : 00819645

[b.lagace]

keeps us posted.
Interesting stuff.

[jorgene]

Veeam escalated case, still stuck on L3 @ HP with no solution yet.

[b.lagace]

good luck with that.

First you'll need to patch all your servers, apply 7 microsoft KB's, upgrade your switches, replace your cables, change the firmaware...and then if you still have the problem...they'll start looking.
I got years of experience with HP...it makes a 35 years old man turn grey.

[jorgene]

In this case EVERYTHING was fully patched, since it was all new hardware for the Customer. They say this shouldn't happen, the LhnVssProvider.exe shouldn't crash
I haven't been a pain in the ass for them since i switched to use Microsoft CSV ShadowCopy instead and problem is gone. But i'm rellay not upgrading other P4XXX systems to same SAN IQ and Veeam versions until this issuse is resolved. I have several Customers running older SAN IQ and Veeam Versions and it works fine, never had this problem. I will start pushing them harder now since they have had their honemooon weeks now.
How hard can it be to debug their own application crashes? Like why the LhnVssProvider.exe crashes? They have all logs and details just need to debug with their own symbols.

Note: Veeam support offers to talk directly with HP about this issue, but i still await HP response form the last details i sent them until we proceed with that.

[jorgene]

HP has contacted me and they will provide me with an updated Lefthand VSS Provider file.
I will get back when i have tested that one.

[b.lagace]

Very good and interesting.
Looking forward to hear more from this case.

[jorgene]

Hi, i have tested the new binary LhnVssProvider.exe i got from HP for a few days now. So far it looks promising, i got no crashes or LUN snapshot leftovers after backup Jobs.
I will continue monitoring this binary for a week more and awaits HP response if they will include this fix in next SAN IQ update.

[jorgene]

Update:
Been runing the new binary for weeks now and all works normal, backup runs fine as well as no leftover SAN snapshots.

Last update from HP Before case is closed.
The fix will be included in the next major release of SAN IQ 12.5. I have no ETA from HP about that one yet.
So if the information i got so far from HP is correct the fix will NOT be included in the 12.0 upcoming patches.
So if anyone more have the same problem as i have, you need to open a case with HP to get the modified binary i got to resolve the issue.

[foggy]

Jorgen, thanks for keeping this thread live up to the actual resolution, this is highly appreciated!