VSS related error with Equallogic and Windows Server 2012 R2

[StephanF]

Hi there,

we have a disgusting error with Dell Equallogic Hardware VSS and WIndows Server 2012 R2 for with we are searching help since more than 6 months. We had support cases with Veeam and Dell but nobody seems to find the root cause.
Maybe some here as a comparable setup or any idea what we can do. Seems to be my last hope...

Over the months tried to remove as much complexity as possible out of our test scenario. I try to summarize the whole case as short as possible.

Problem:
I am taking a backup of the VMs on my Hyper-V Host. The Equallogic Hardware VSS provider is used. The snapshot is created and the backup is taken successfully.
After the backup the snapshot is released by Veeam. During the snapshot release several errors are logged in the eventlog of the Hyper-V host, starting with many:
MSiSCSI 116 Removal of device MPIO\DISK&VEN_EQLOGIC&PROD_100E-00&REV_8.1_\1&7F6AC24&0&363031394342373134343430343637454332373442353044313432303641 on iSCSI session ffffe00139a9d020-4000013700000008 was vetoed by STORAGE\Volume\{0605d165-264b-11e6-80c0-5cf9ddf26177}
And some more errors afterwards.

According to Dell this happens because the snapshot is removed and the iSCSI session are released while something is still accessing the volume. This leads to the error above, followed by some unsuccessful reconnection attempts etc.
Veeam itself does not log an error. After some minutes the backup job finishes fine.

What makes it really interesting is:
- This only happens on Windows Server 2012 R2. The completely same setup on the same hardware with Windows Server 2012 as the Hyper-V Hosts OS does not produce any errors.
- This only happens when Veeam is creating and releasing the snapshots. Using Microsofts Vssadmin tool for creating and releasing snapshots does not produce any errors.
- Of course, this only happens with the Equallogic Hardware VSS provider.
- The issue does not depend on standalone or clustered Hyper-V Hosts, application aware processing on or off, VMs powered on or off, etc.

I compared the behaviour and logs between Windows Server 2012 and 2012 R2. The only difference that happens is when releasing the VSS snapshot. In Windows 2012 the additional disk in Windows Disk Management (= mounted snapshot) disappears, the iSCSI sessions are closed an the snapshot is deleted. All fine. In 2012 R2 the iSCSI sessions are closed an the snapshot is deleted but the disk stays. Then the errors start. Unsuccessful reconnection attemps etc.

Nobody could find the root cause of this until now. This could also be a disgusting timing issue that behaves a little bit different in Windows Server 2012 R2 or something.

I can not believe we are the only one in the world with this setup and hardware combination. Maybe somebody has seen such errors before or has the same hardware and can tell me if it is working or not.

Environment:
1x Hyper-V Host
- Dell PowerEdge R720
- OS: Windows Server 2012 R2 english
- clean install, only latest patches and drivers
- Dell Equallogic Host Integration Tools 4.8.0

2x Equallogic SAN (PS4110)
- latest Firmware


Veeam case# 01812617


Any input is highly appreciated.

Thanks
Stephan

[PTide]

Hi,

What's the name of the vetoing service? Should be listed right after "Vetoing device"

Thanks

[StephanF]

Hi,

no vetoing service. That's the whole error message that I posted.
I tried tu use the handle tool and process explorer from Microsoft to see what is still open. But no luck here.

Dell Support said it could be that the volume (mounted snapshot) itself is not completely closed/dismounted/whatever before the VSS snaphot is removed.

[Vitaliy S.]

Did you try to run different version of Dell Equallogic Host Integration Tools? Just guessing that there could be something wrong with the VSS provider version.

[brett.hopkins]

I am also having this issue. Case # 01834223
Environment:
7x Hyper-V Host In Failover Cluster
- Multiple Physical host types
- OS: Windows Server 2012 R2
- Dell Equallogic Host Integration Tools 4.8.0

4x Equallogic SAN (PS4000, PS4100, PS6100 X2)
- Firmware 7.1.8

[moderniste]

Same issue here...

3 Hyper-V 2012 r2 (french language) node R630
Dell Equallogic Host Integration Tools 4.8.0
Equallogic PS4210XS Firmware 7.1.12

HIT 4.9 are realeased, anyone tested it?

[Vitaliy S.]

Cannot find any solution in both cases mentioned in this thread. One of them was closed due to no response from the customer.