Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Hyper-V specific discussions

Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Mon Feb 02, 2015 1:44 pm

Hi,
Just wondering if anyone else have had or seen this behaviour.
I have latest Veeam 8.0.0.917 installed.
2-node W2012 R2 Hyper-V Cluster and 2 HP StoreVirtual 4330 SAN nodes for shared storage (Running latest SAN IQ 11.5.00.0673.0 fully patched). Also latest DSM for MPIO 11.5.0.345 and VSS Provider 11.5.0.417 installed on Hyper-V hosts.

Now after Veeam Backups (sometimes, not always) Snapshots on the P4330 units created during veeam backup (thin temp volumes) still remains on the unit and don't get deleted after backup finish.
If I check the Hyper-V host in ISCSI I have a disconnected ISCSI session to the temp space LUN. After further investigation of logfiles I see that I get "Faulting application name: LhnVssProvider.exe, version: 11.5.0.417" "Faulting module name: MSVCR90.dll, version: 9.0.30729.8387" errors. So it seems like the Application Aware VSS Provider crashes, that's probably why I get temp volumes left on the SAN node. This seems to happen when it's time to delete the snapshot, since all backup Job works fine (until temp space volumes fills up the SAN node). I need to manually remove snapshots in CMC.

It seems like I have to put a case with HP about this since I cannot see that Veeam are causing this (if not putting some strange commands to HP VSS Provider causing the crash).
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby foggy » Mon Feb 02, 2015 2:03 pm

Yes, in case of VSS provider failure it is more appropriate to contact provider vendor. You can try to reproduce the behavior without Veeam B&R in presence, using diskshadow command, just to make sure it is not involved.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Tue Feb 03, 2015 11:00 am

I've opened a case with HP now and will be sending them a bunch of logfiles to analyze. Are there any way to make Veeam "Skip" using Lefthand VSS Provider and use Microsoft Shadow Copy instead?
Except uninstalling HP Lefthand VSS Provider software on the Cluster node?
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby foggy » Tue Feb 03, 2015 11:04 am

You an select the provider to be used in the volume-specific settings of the host.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Tue Feb 03, 2015 12:36 pm

Great, thx.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Fri Feb 06, 2015 7:07 am 1 person likes this post

HP case escalated to lvl 3 since they cannot see whats causing the crashes. After switching to Microsoft CSV ShadowCopy problem gone. But it puts more load on Hyper-V hosts during backup.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby foggy » Fri Feb 06, 2015 11:11 am

Thanks for the update. Keep us posted on how the investigation goes.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby TheDonMiguel » Mon Feb 23, 2015 8:45 am

Would be interesting to hear more about this. I'm facing exactly the same issue with my LeftHand Storage
TheDonMiguel
Lurker
 
Posts: 1
Liked: never
Joined: Fri Nov 14, 2014 3:21 pm
Full Name: Michel

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Tue Feb 24, 2015 2:54 pm

Hi, TheDonMiguel please report this issue to HP also. As they are aware i'm the only one having this issue, reported. Now they force me to wait until the patch 12 is released for SAN IQ. Should be released in a few weeks, i didn't get an ETA from them yet. If i still have this issue after that patch is applied they will dig deeper.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Sat Feb 28, 2015 11:38 am

Hi all, SAN IQ ver 12.0 released just a few days ago.

I have installed and upgraded SAN IQ, VSS Provider and MPIO. But still same issue, 2 backups went fine, on the 3rd the VSS provider crashes again on Hyper-V host leaving snapshot on the LUN.
Code: Select all
Faulting application name: LhnVssProvider.exe, version: 12.0.0.507, time stamp: 0x54baa84b
Faulting module name: MSVCR90.dll, version: 9.0.30729.8387, time stamp: 0x51ea1bbd
Exception code: 0xc0000005
Fault offset: 0x000000000003e1e8
Faulting process id: 0x790
Faulting application start time: 0x01d0529ba5a70be1
Faulting application path: C:\Program Files\HP\StoreVirtual\Application Aware Snapshot Manager\LhnVssProvider.exe
Faulting module path: C:\windows\WinSxS\amd64_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.8387_none_08e793bfa83a89b5\MSVCR90.dll
Report Id: 6375bfef-bef7-11e4-80d2-40a8f0361da0
Faulting package full name:
Faulting package-relative application ID:


I will see what HP says, but they suspects Veeam is causing it they told me.
But we will se what support says after my information.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Mon Mar 02, 2015 9:10 am

Veeam case : 00819645
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby b.lagace » Thu Mar 05, 2015 4:46 pm

keeps us posted.
Interesting stuff.
b.lagace
Enthusiast
 
Posts: 31
Liked: 17 times
Joined: Wed Jul 02, 2014 1:23 pm
Full Name: Bjorn Lagace

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Tue Mar 17, 2015 5:04 pm

Veeam escalated case, still stuck on L3 @ HP with no solution yet.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby b.lagace » Mon Mar 23, 2015 10:16 am

good luck with that.

First you'll need to patch all your servers, apply 7 microsoft KB's, upgrade your switches, replace your cables, change the firmaware...and then if you still have the problem...they'll start looking.
I got years of experience with HP...it makes a 35 years old man turn grey.
b.lagace
Enthusiast
 
Posts: 31
Liked: 17 times
Joined: Wed Jul 02, 2014 1:23 pm
Full Name: Bjorn Lagace

Re: Veeam 8 + SAN IQ 11.5 (Store Virtual/Lefthand)

Veeam Logoby jorgene » Fri Mar 27, 2015 8:43 am

In this case EVERYTHING was fully patched, since it was all new hardware for the Customer. They say this shouldn't happen, the LhnVssProvider.exe shouldn't crash ;)
I haven't been a pain in the ass for them since i switched to use Microsoft CSV ShadowCopy instead and problem is gone. But i'm rellay not upgrading other P4XXX systems to same SAN IQ and Veeam versions until this issuse is resolved. I have several Customers running older SAN IQ and Veeam Versions and it works fine, never had this problem. I will start pushing them harder now since they have had their honemooon weeks now.
How hard can it be to debug their own application crashes? Like why the LhnVssProvider.exe crashes? They have all logs and details just need to debug with their own symbols.

Note: Veeam support offers to talk directly with HP about this issue, but i still await HP response form the last details i sent them until we proceed with that.
jorgene
Enthusiast
 
Posts: 73
Liked: 16 times
Joined: Thu Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 8 guests