Hyper-v host evictions during backup window

[collinp]

Does anyone have experience with Hyper-v hosts evicting from the cluster during the Veeam backup window? It started happening more frequently as our environment has grown. Details are listed below.


Veeam Support - Case # 00192516
Microsoft Premier Case Number # 113032610317395
Veeam Version: 6.5.0.128
Microsoft Software VSS

Issue: Hyper-v nodes are evicting nightly during Veeam Backup window causing outages to the virtual machines.

Networking team shows network resources staying online at time of outage. Pinger utility shows network interfaces staying online. Evictions only happen during backup window.

System Log Alerts before cluster eviction:
3:07AM 4201 IPhlpsvc no longer active
3:08AM 1127 Cluster network failed
3:27AM 7036 Cluster service entered stopped state
3:27AM 7024 Cluster service terminated semaphore timeout expired
3:27AM 7031 Cluster service terminated
3:27 4201 iphlpsvc not active
3:27AM 1038 Ownership of cluster disk lost by node
3:29 28 VMSMP


Hotfixes currently installed on Hyper-v hosts:
KB2460971
KB2494016
KB2494162
KB2511962
KB2520235
KB2522766
KB2524478
KB2528357
KB2531907
KB2534356
KB2547551
KB2550569
KB2550886
KB2568088
KB2586470
KB2587660
KB2637197
KB2639032
KB2662288
KB2665347
KB2668084
KB2679255
KB2705759
KB2708549
KB2734056

Cluster settings:
cluster.exe /prop SameSubnetDelay=2000:DWORD
cluster.exe /prop CrossSubnetDelay=4000:DWORD
cluster.exe /prop CrossSubnetThreshold=10:DWORD
cluster.exe /prop SameSubnetThreshold=10:DWORD

[Vitaliy S.]

Collin,

Can you please tell me what Hyper-V version are you running? 2012 or 2008?

Thanks!

[collinp]

Hyper-v 2008 R2 SP1.

I also have a MS Premier case open.

[collinp]

Good news! After installing the following hotfixes, we have had no evictions for the past week.

Does anyone know if it is normal behavior when running 2 simultaneous Hyper-v backup jobs, and when doing onhost backups, for the second job to wait until the first job completes with the message "resource not ready" before backing up?

http://support.microsoft.com/?id=2795696
http://support.microsoft.com/?id=2687646
http://support.microsoft.com/?id=2639032
http://support.microsoft.com/?id=2754704
Cisco Network Adapter drivers
Netapp Host Utilities
Netapp MPIO DSM
Netapp Snapdrive

[foggy]

Does anyone know if it is normal behavior when running 2 simultaneous Hyper-v backup jobs, and when doing onhost backups, for the second job to wait until the first job completes with the message "resource not ready" before backing up?

Colin, this is expected when software snapshot provider is used. In this case, CSV volume needs to be switched to redirected I/O mode until the backup process completes, which makes other cluster nodes incapable of taking snapshots (until the CSV returns to the direct access mode).