Veeam Bare Metal Recovery on XenServer with PV NIC Drivers

[sc-frank]

Hey all.

I wanted to share a solution to something that has been a nightmare for us for a few years now.

We've had major trouble finding a perfect backup solution for VMs running on Citrix XenServer.
The main problems we've encountered over the years are:

1. The Agent has a GUI that the users can interact with and thus change settings or view the backups stored on the server.
This was a problem for us, because we run a hosted, multi-tenanted solution, so need to centrally manage all backups and ensure that users can't access data on the backup servers.

2. The throughput for backups wasn't fast enough generally.

3. The Bare Metal Recovery ISO, didn't have drivers for XenServer Paravirtualised NICs, so it would use the default Realtek RTL8139 Driver, which only runs at 100Mbps. Too slow for restores of large VMs.

Symantec and ArcServe climbed to the top of the lists through these tests, but sadly both of them fell foul of 1/3 of the items above.
I was excited when Veeam released the Windows & Linux Agents, thinking this would be the answer, however, as we know, the agent was unable to be centrally managed...
...Until glorious 'Update 3' to Veeam B&R!
Now that the agents can be centrally managed, they qualified for further testing.

Happy to say that the centralised management of the agent is great!
However, sad to say, that Veeam also fell foul with a lack of XenServer drivers in the BMR ISO.
Citrix have a great guide to solve this with XenServer 6.5 - https://www.citrix.com/blogs/2015/03/03 ... installed/
However, it doesn't work for XenServer 7 onwards, and we are running 7.1 LTSR.

I was convinced to get this working, so ripped the image apart and after a lot of playing, I'm happy to report that I've got it working!

Here's an outline on what needs to be done. I won't go into details on commands or step by step instructions as with a bit of googling, it's easy enough. Nor will I provide a pre-modified ISO, as I don't know if that would be allowed!

Tools required:
DISM.EXE
OSCDIMG.EXE
Veeam Recovery Media ISO
XenServer Tools ISO
A VM running Windows 10 or Server 2016, with XenServer Tools installed and up to date.
Time, patience and attention to detail!!

By extracting the XenTools setup files from the MSI, you can get your hands on the underlying driver files.
The WinRE allows you to load additional drivers, so these driver files can be loaded once the ISO is booted if you copy them to somewhere accessible.
However, even once loaded, the NIC is still not detected. The reason is that the Realtek driver is already loaded, so there is no hardware for the XenServer driver to attach to.
If you extract the contents of the ISO, and use DISM to mount the WIM file, you can remove the Realtek driver by just deleting rtnic64.sys from Windows\System32\Drivers, and then remove the associated INF files/folders from Windows\System32\DriverStore\FileRepository. While you're there, use DISM or PowerShell to inject all the XenTools drivers (do all of them, not just xennet, otherwise the NIC still won't be detected).
Once done, there are registry entries that need modifying. This is where the scripts in that Citrix guide take care of it, but some of those entries have changed since XS 6.5.
So, mount the SYSTEM registry hive from the image and use regedit to add the following entries, copying from your staging VM.
Note: The source on your staging VM will be CurrentControlSet, but the hive in the image only has ControlSet001, so put them in there.

HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e96a-e325-11ce-bfc1-08002be10318} - Copy the UpperFilters Multi-value string.
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e97d-e325-11ce-bfc1-08002be10318} - Copy the UpperFilters Multi-value string.
HKLM\SYSTEM\CurrentControlSet\Services\XEN - Copy this key and everything underneath it.

Create a new value as follows inside the image.
HKLM\SYSTEM\CurrentControlSet\Services\xenfilt\WindowsPEMode - DWORD - Value = 1

That it. Unmount the registry hive.
Then, save the WIM and generate a new ISO from it. Boot from it and voila, if all goes well, you'll be up and running with XenServer PV network drivers.

I hope this helps anyone else who has trouble with WinPE ISOs running under XenServer.

Frank

[Mike Resseler]

Frank,

Wow. And awesome. I am glad to see that you found a nice solution for your problem, and I am absolutely convinced that this will help others on these forums also.

Great job man,

Cheers
Mike

[selgan]

I can confirm this works. You are correct, with enough googling, one can decipher your steps provided. I'll provide a bit more detail to help the next person. I'll point out that I only cared about a WIM file and not a bootable ISO. My use case was with MDT and booting PXE to WDS. If you need to get a WIM from and ISO or make a WIM into an ISO, research that.

Extract the tools.

Code: Select all

msiexec.exe /a <XSTools MSI path> /qb TARGETDIR=<path to temp XSTools>

Determine WIM Index to mount. It will likely be Index 1.

Code: Select all

dism /get-wiminfo /wimfile:<path to wim>

Mount the WIM.

Code: Select all

dism /mount-wim /wimfile:<path to wim /index:1 /mountdir:<path to temp mountdir>

Take ownership of the following folders and then grant yourself full control. Then delete them.

Code: Select all

<path to temp mountdir>\Windows\system32\DriverStore\FileRepository\netrtl64.inf_amd64_8e9c2368fe308df2
<path to temp mountdir>\Mount\Windows\WinSxS\amd64_netrtl64.inf_31bf3856ad364e35_10.0.16299.15_none_cc07f879fba34860
<path to temp mountdir>\Mount\Windows\WinSxS\amd64_netrtl64.inf.resources_31bf3856ad364e35_10.0.16299.15_en-us_45ec42bc03abec68

Inject XS Tools Drivers into the WIM.

Code: Select all

Dism /Image:<path to temp mountdir> /Add-Driver /Driver:<path to temp XSTools> /Recurse

Open Regedit. Mount the following hive and give it a name of WinPE.

Code: Select all

<path to temp mountdir>\Windows\System32\config\SYSTEM

Assuming you have mounted the path correctly, use this registry file to make the appropriate changes.

Code: Select all

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\WinPE\ControlSet001\Control\Class\{4d36e96a-e325-11ce-bfc1-08002be10318}]
"UpperFilters"=hex(7):58,00,45,00,4e,00,46,00,49,00,4c,00,54,00,00,00,00,00

[HKEY_LOCAL_MACHINE\WinPE\ControlSet001\Control\Class\{4d36e97d-e325-11ce-bfc1-08002be10318}]
"UpperFilters"=hex(7):58,00,45,00,4e,00,46,00,49,00,4c,00,54,00,00,00,00,00

[HKEY_LOCAL_MACHINE\WinPE\ControlSet001\Services\XEN]

[HKEY_LOCAL_MACHINE\WinPE\ControlSet001\Services\XEN\Unplug]
"DISKS"=dword:00000001
"NICS"=dword:00000001

[HKEY_LOCAL_MACHINE\WinPE\ControlSet001\Services\xenfilt]
"WindowsPEMode"=dword:00000001

Unmount the registry hive.

Close any open file/folder handles to your temporary wim mount folder.

Unmount the WIM

Code: Select all

Dism /Unmount-Image /MountDir:<path to temp mountdir> /Commit

Convert your WIM to ISO or boot from your WIM. Upon booting, press F8 in the WinPE environment and type ipconfig/all. You should see XenServer PV Network Device. Enjoy gigabit speed in WinPE on XenServer.

Reach out to me @selgan on Twitter for any questions. Thanks Frank for the initial write up!

[Gostev]

Thank you so much for your time, guys. This is exactly what makes this community so great!!

[archonnn]

Wow, I mean, this is a fantastic workaround. We have the exact same problem, thanks man!

[BackupIsBest]

I have exactly same problem. Lost 2 weeks with Veeam Support for nothing and finally get this link.
So why Veeam won't released Veeam Recovery Media with this PV XEN drivers yet ???
Veeam at least should provide Exactly helpcenter with every manual command to create VeeamRecoveryMedia itself.
Now Im wasting time for googling for Xen Tools ISO ... ehh no comments.

[BackupIsBest]

In addition we're using Oracle VM Hypervizor based on XEN but drivers are Oracle-PV-Drivers.
NIC is: Oracle VM Virtual Network Adapter
Have no idea how to inject Oracle-PV-Drivers which are bundled in Setup-3.4.4-signed.exe