Server 2019 Restores to Azure Fail To Boot

[chris.church]

Utilizing VMware and Veeam on premises with an SOBR to Azure. DR is done in Azure with VM's restored from object storage and instantiated as Azure VM's.

While testing our DR to Azure this month we noted that all 14 of our Server 2019 workloads would not power on in Azure after restoring them from object storage. Everything worked correctly back in December during the last DR test. The VM's fail to boot in Azure whether they are restored from on-premises storage or Azure object storage.

Additional testing determined that after an on-premises 2019 VM is updated to 1809 10.0.17763.1852 or greater with the installation of MS patches KB5001342 and KB5001404, the test VM's will no longer boot in Azure after restore. Uninstalling the update does not correct the issue. This essentially renders our DR capability inoperable.

The dump file from the Azure VM indicates the problem is with vmstorfl attempting to allocate a part in memory that it does not have access to.

We have cases open with Veeam and MS but it's been slow going getting to a resolution.

Curious if anyone has encountered this issue, or is able to reproduce it.

Veeam Case ID: #02335964

[HannesK]

Hello,
and welcome to the forums.

As far as I see, this issue has not related to KB5001342 and KB5001404. I just tested it (1809 17763.1879) with before / after and it works fine for me.

As you posted about capacity tier... did you test a restore from on-prem performance tier?

Best regards,
Hannes

[chris.church]

Restores to Azure fail from both the performance tier and the capacity tier.

We've invested considerable time and effort in attempting to isolate the issue and are able to reproduce the failure consistently when KB5001342 and KB5001404 are applied.

Veeam support indicated they were able to reproduce the issue.

Veeam case # is 04739226

[HannesK]

If support could reproduce it, that's good. Then they can continue finding a way to fix it.

In my test, I deployed a VM from a template which I didn't update since some months. I tested restore before and after I installed the latest updates then to be sure only this is the difference. Restore worked fine for me with KB5001342 and KB5001404 installed. So it seems to me that there are more things that need to come together for a failed restore.

[chris.church]

Update on this issue. After working with MS Azure support for several weeks the issue was identified as follows.

-Issue was introduced in the January OS rollup due to changes with the handling of PNP I/Os.
-In Azure during the first power on after restore, a new IDE disk is added while discovering devices, this was exposing a bug in the vmstorfl.sys driver.
-Issue was subsequently resolved in the May OS rollup. Test restores to Azure after application of this rollup were successful.

[HannesK]

thank your for sharing this with the community!