instant restore to hyperV (vmware v2v) causing random bluescreens

[DaStivi]

Hello i've some weird issue restoring (vmware vms) to hyperv with instant-restore causing bluescreens on hyperV host!

environment:
2x HyperV Server 2025 (now) with Storage Spaces Direct Volumes (HPE DL380, with 4x25g RDMA direct connections for S2D traffic and 2x10g SET Switches for Mgmt and VM etc...)

issue happend on a test before on server 2022 too! so it shouldn't be dedicated to server 2025!

i did multiple restore tests, and for some reason sometimes, most of the times, the hyperV host that is selected as restore target bluescreens as soon as i start the "migrate to production" phase inside veeam!

mounting the vm on the host is running fine. also starting the vm is working, i'll see the redirection traffic to the veeam server from the veeamagent.exe on the host

i still do some tests.. because it looks like it has something todo also with what is the owner-node of the target cluster shared volume . but its different then i would have thought.. because it looks like, if the target host and the target folder is on the same node bsod is happening...
if the target-volumes owner node is the 2nd node thats NOT the target for the restore operation, i see some ping-losses, but the move is starting... after some time though the target node has also bsod'ed!

i have also tested different target volumes because on one of them i've bitlocker enabled.. but i also created a brand new one without any options enabled and it happens there too!

another finding i had yesterday, in all the trys above i disabled the option for preallocation in the instant-restore wizard... as soon as i let the option for preallocation enabled, the host instantly bsods when the vm is mounted on the host.. so its not even getting registered or able to start then...

as far as i can tell right now, there has to be something wrong with the redirection io traffic that veeamagent.exe is dooing

another test i did, let veeam mount the VM/vhd, no matter if the machine is running or not, i then manually registered the VM inside my cluster (veeam does this later in the migration progress but not on the inital mount allready, whyever?) but after manually registering the VM inside clusmgmt i can use hyper-v/cluster storage move operation, then i "move/copy" the vhd's to my 2nd vmstore... that is working without any issues or bluescreen... but veeam is not aware of what is happening, when i then click "stop publishing" the vm inside the normal hyperv console gets deleted... but the moved/copied disk remains of course and i can recreate a new vm and point to the copied vhd files! (the leftover cluster objects needs to be deleted... all overall its a dirty workaround to get the vms moved now..)

another thing i used now is starwinds V2V converter! that connects to vcenter/esx and can move directly to hyperv too... this is also working without any issues...



its everytime the Multiple-IRP_Complete_Request BSOD like below:
MULTIPLE_IRP_COMPLETE_REQUESTS 0x00000044 ffffb008`2d8a8010 00000000`000027a2 00000000`00000000 00000000`00000000 ntoskrnl.exe ntoskrnl.exe+4b8680 NT Kernel & System Microsoft® Windows® Operating System Microsoft Corporation 10.0.26100.2033 (WinBuild.160101.0800) x64 ntoskrnl.exe+4b8680 C:\WINDOWS\Minidump\010925-27187-01.dmp 64 15 26100 6.754.092 09.01.2025 17:38:00

are there any known issues with instant restore on hyperV (especially with vmware vms maybe?)
is anyone using S2D and testet this?? i'm even not sure if this could also happen if you just to some Hyper-V VM instant restore... i might test this if i get time todo this in the next days....


case for reference: #07565687

[david.domask]

Hi Stephan,

Thank you for the detailed write up and for sharing the case number. Sorry to hear about the challenges here -- the behavior is definitely not expected, but not aware of any known issues regarding instant recovery to HyperV, muchless with host BSODs.

I see the case was opened with an NFR license -- may I ask you to please apply a license to the environment and test the backup once more, then upload fresh logs to the case? You can just export logs using the 3rd radio button to export logs from hosts, select the Veeam server itself to export from, and upload them to the case so it's logged as a case with active support maintenance.

[DaStivi]

i've tested further....
restore to the same hosts, but not into the S2D-CSV, no bluescreen is happening!

i'm pretty sure the issue must have something todo with the S2D & RDMA!

[DaStivi]

recreated the issue on different hardware with same configuration, Server 2025, Hyper-V with S2D (4x25g RDMA NICs direct attached, just in case this could cause the issue!)
this time with fully functional license, just uploaded another set of logs from this environment! please update case back to normal support.

right now its not that important, as i just use it for v2v migration and (sadly) i switch to a different product for this task now... but in a disastercase it would be good if a restore is working

[david.domask]

THank you for the testing. I've reached out to support regarding handling the case as I can see the license was added in the most recent log bundle. Please allow Support time to review the behavior.

[WoenK]

Ran into some BSODs last week while testing out our new 2022 Hyper-V server...first issue was while trying to team the NIcs the traditional way rather than using powershell.
Found out that one of the Intel Nics did not support VQM that well, disabling it in the driver made the BSODs go away.
Maybe your issue is something in that direction.

[DaStivi]

....Found out that one of the Intel Nics did not support VQM that well, disabling it in the driver made the BSODs go away.
Maybe your issue is something in that direction

VMQ should be disabled on Intel nics by default... If you dare to enable this, you also need to set some additional reg keys to fully support it... There is one specifically for 1g adapters... And another for 10g ... Information about this is only rarely and is from old windows server versions...

Broadcom drivers do enable the setting on the nic automatically, bit without the regkeys that may be needed or not...

Overall the Experiences with VMQ are negative! And it causes more trouble then helps!

I've tested it enabled and disabled, and I absolutely didn't saw any difference inside my VM or the Host. But I did had some strange network hickups while it was enabled... Hence I just disabled it all together.

Very good write up about VMQ:
https://charbelnemnom.com/configure-vmq ... -1gb-nics/

I wasn't able to reproduce the benchmark tests from this page...

Another helpfull page was the Cisco UCS configuration page , but again still lot of confusion and errors possible for little improvement
https://www.cisco.com/c/de_de/support/d ... 19-on.html

...first issue was while trying to team the NIcs the traditional way rather than using powershell.

Don't do the normal lbfo teaming through Server manager ! This is deprecated, especially for HyperV!

Use Powershell and use the " switch embedded team" parameter!

If you haven't dedicated host nics, create the Host-Management nic also as "virtual" one with the allowmanagementOS parameter!

Code: Select all

New-VMSwitch -Name "VMSwitch" -NetAdapterName "10G Port 1","10G Port 2" -EnableEmbeddedTeaming:1 -AllowManagmentOS:0  -MinimumBandwidthMode Weight
Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName "VMSwitch"

Set-VMNetworkAdapter -ManagementOS -Name "Management" -MinimumBandwidthWeight 5

[Techie4Life]

We've had similar issues where going from HV to VMware the majority of VMs BSOD/cannot find the disk. Different OS versions and everything. We had to start using Starwinds V2V as well.

Going from VMware to HV wasn't an issue.

[DaStivi]

i just assume you're refering to bsods inside VMs ? from HV to VMWare it could be the simple reason that especially older HV VMs where "IDE" disks... and you need to "convert" them to vmware the same!

the otherway arround from VMWare to HV, Virtual Machine wise it isn't the issue the VMs BSOD, its the Host HyperV Server that Bluescreens


Constant Bluescreens in tests crashed on of my HV Hosts recently... somehow the WMI-Namespace was broken... couldn't restore to the HV-Server anymore.. and also management of HyperV didn't worked anylonger... got following error

Code: Select all

the WMI-Namespace "root\virtualization\v2" is not registered in CIM-Repository

not sure what exactly happend... but nothing has been change on the server... jsut the random bsod reboots... anyway i "fixed" with removing and readding the HyperV Role to to windows server... network was broken afterwards and had to be recreated too (because vm-switch functionality hangs on the HV role..)

[DaStivi]

Update to the issue, with support togehter we might have found the root-cause! it looks like the bluescreen happens when CSV Cache is enabled. after disabling this particular feature i can normally start instant restores, pre-allocation phase is also running and the "move to production" step looks like also is working... though i had another issue while the data migration now, it could just have been a random network issue... after retry its still running now.

[theprojectgroup]

Thanks a lot @DaStivi
Almost the same here. In our case the target node BSODs right after starting the instant recovery job. We don't even reach the phase where we can "migrate to production"

- 3 node server 2025 (CU February 2025)
- Veeam 12.3.0.310

Code: Select all

Bug Check String: MULTIPLE IRP COMPLETE REQUESTS
Driver: FLTMGR.SVS

Works if you disable the cluster shared volume (CSV) cache (works even without taking the volume offline and online):

Code: Select all

Get-ClusterSharedVolume "Volume02" | Set-ClusterParameter  EnableBlockCache 0
WARNING: The properties were stored, but not all changes will take effect until Volume02 is taken offline and then online again.

[DaStivi]

i didn't used the individual CSV Cache settings "per Volume" i just en-/disabled the global in-memory cache setting... easiest through windows admin center:


or via powershell:
(Get-Cluster).BlockCacheSize
BlockCacheSize : 131072
.......

disabling:
(Get-Cluster). BlockCacheSize = 0

but for sure in theory you could change this at volume level to... there btw. its enabled per default.. everytime.. even if the in-memory (global) setting is disabled by default.. this is why i don't opted in to change the setting there.. and dont forgett to enable it back after v2v migrations... in theory its also happening in "real" hyperV instant restore tasks... but my case is at Veeam RnD and they're trying to reproduce the issue and hopefully come up with a fix for this behaviour...


for the global setting it definitly needed that the resource (volume) is moved between the cluster... while troubleshooting with veeam support, and recreating the issue, i enabled it ... then on the evening i had to move VMs in production... and i just crashed the hyper V again.. then i wondered, because i had it disabled allready... but didn't take the volume offline or moved it.. and hence it bluescreened again.. after moving between the nodes it then worked again..


for reference:
https://techcommunity.microsoft.com/blo ... che/371854
https://learn.microsoft.com/en-us/windo ... -csv-cache