Keyboard not working with recovery media in HyperV

[Linkfin]

Hi
I am having an issue with recovering a backup to a HyperV Virtual machine. When booting a gen 2 Hyperv Machine to the recovery media created for our backup, I am unable to enter a password to unlock a bitlocker encrypted drive (containing the backup) as the keyboard is not working. I have mouse control but no keyboard. Has anyone else had this issue or have a solution for it?

Thanks

[Mike Resseler]

Mark,

That is strange, since I assume you are in the virtual connect window of Hyper-V and that normally should work. There are cases with a physical BMR where there is a USB keyboard (those advanced ones) that we need to add an additional driver to use the keyboard, but in this case it should not be.

Can you create a support case so this can be investigated?

Thanks
Mike

[Linkfin]

Hi Mike
I have just realised I may be in the wrong forum here as this issue is with endpoint backup free. Am I still able to raise a support case for this? I have booted the recovery media to a Gen 1 hyperV Vm and it works fine but the recovery I am trying to perform is for a UEFI boot machine so that is of no use. it also works fine if I boot it to an esxi vm, it is only Gen 2 HyperV vms that don't have keyboard input.

Regards

[Mike Resseler]

Mark,

Yes, our free version also comes with support, however, we do not give an SLA on it. So we do this on a best effort basis, but we always will come back to you to assist.

Thanks
Mike

[Linkfin]

Hi Mike
The issue I have here is that it appears I need to do this from the installation on the server, however, the server in question does not have internet access. Is there any other way I can open a case for this?

Thanks

[Mike Resseler]

Mark,

Do you have (somewhere) another version of VEB running that has access to the internet? Do it from there and state in the notes explicitly that the case is for another installation. Our engineers will probably ask you to upload specific files from that server after that.

Mike

[Dima P.]

Hi,

I have mouse control but no keyboard.

How both devices are connected to the Hype V host? Thanks.

[tekno]

Hello that seems the same thing i had with w2k12 r2 and gen 2.. the integration tools were missing in our internal image, so i could not type. Thats why im still using gen 1 VM's @ HyperV. Is it something missing as well for endpoint ?

[lreese]

Maybe a dumb question, but if your mouse is working have you tried using a different keyboard?

[Dima P.]

This could be related to drivers - once recovery media is booted from an ISO go to Tools > Load Drivers and check the all the required USB drivers are installed.

[Gostev]

check the all the required USB drivers are installed.

Dima, so which specific drivers are required for the keyboard to work on Gen2 Hyper-V VM?

[AlexHeylin]

This is still a problem in v4.0.1.2169 created from Windows Server 2012. This happens regardless of if media is created directly on original machine, or created via a VBR "Create recovery media".

One thing that might be relevant - the original server was running in legacy BIOS mode, not UEFI so even if restored into the Gen 2 VM it wouldn't have booted anyway (unless recovery media pokes it to make it work? - because the OS can boot from UEFI). Possibly when creating recovery media for a BIOS mode machine it doesn't load the drivers required for keyboard in Gen 2 VM?

https://pasteboard.co/JQaDcle.png
https://pasteboard.co/JQaDo55.png

[Dima P.]

Hello Alex,

Thanks for the feedback, we are looking into this. Mind me asking why you are using BMR to Hyper-V instead of Instant Recovery? Last should be much easier and faster way to restore agent backup into Hyper-V environment. Cheers!

[AlexHeylin]

Thanks Dima P. - we tried BMR as an alternative to restoring the backup to VHD because that was quite slow and only processed about 33MB/s even though every step in the chain (hyper-v server - VBR server - SP CCG - SP repo) were all running at minimal load (20% max) and networking shouldn't have been an issue (though we later realised it might be, due to VCC forcing "local" connections to go through NAT reflection on the external IP even if a local route exists between the LANs and the VCC was added to the VBR by "local" IP). BMR only managed 16MB/s. We wouldn't think to use Instant Recovery as we had a downtime window, and the whole VM needed restoring and lots of work to get it cleaned up and ready for production.

Would Instant Recovery have avoided the VHDX getting created in 32,000 fragments?
We had to defrag the VHDX file on disk before attaching to the VM or the VM was very slow. It'd be great if Veeam had the option to avoid this, which should probably be enabled by default. (Just lazy preallocate the whole VHDX at the start of the restore).

I would assume instant recovery is actually slower overall than a straight restore, plus we'd have likely had to shut it down to defrag the VHDX file anyway. Were our assumptions wrong?

Thanks

[Dima P.]

Regarding the keyboard/mouse issue - looks like it completely related to MS, to cut the log story short the recovery media was created from the 2012 machine without keyboard drivers injected wont have such drivers in it. The workaround would be to recreate the recovery media, from another machine where drivers were injected.

we tried BMR as an alternative to restoring the backup to VHD

Was that export agent backup to a VHD disk, right?

because that was quite slow and only processed about 33MB/s even though every step in the chain

Would it be possible to run a test over the same network setup but with regular repository instead of cloud connect repo? Another test is to perform same restore from local repo next to the Hyper-V machine? Just to better understand the bottleneck.

I would assume instant recovery is actually slower overall than a straight restore, plus we'd have likely had to shut it down to defrag the VHDX file anyway. Were our assumptions wrong?

My assumption is that since we start the machine from the backup, store the changes in the datastore and then transfer altogether to the target that should be faster. I'll check that and get back to you.

[Dima P.]

to defrag the VHDX file on disk before attaching to the VM or the VM was very slow

Checked: the disk is pre-allocated before recovery, so it should not be fragmented. QA team proposed to investigate this behavior with support folks to understand the root cause. Cheers!

[AlexHeylin]

Hi Dima,

Regarding the keyboard/mouse issue - looks like it completely related to MS

Well that would certainly explain it! At least we know and have a workaround. Thanks

Was that export agent backup to a VHD disk, right?

Yes.

Would it be possible to run a test over the same network setup but with regular repository instead of cloud connect repo?

I can't restore that VM from a local repo to the same Hyper-V server (local repo is on client site, our copy is in our cloud repo).
I can find another similar sized HV-VM (not VAW) backup to restore a VHD from onto the same hypervisor host - if that'll be a valid test?

Checked: the disk is pre-allocated before recovery, so it should not be fragmented.

Thanks for checking this. I've been having a look at a number of our hypervisors and backup servers, and based on which files are heavily fragmented and which aren't, I think it's safe to say I have anecdotal evidence that:
- Restoring whole VMs = heavily fragmented VHD(X)
- Restoring (exporting) backup to VHD(X) = heavily fragmented VHD(X)
- Backup to VCC repo = fragmented files, even the VIBs (say 3000 fraggles for a 6GB file). The VBKs can be in hundreds of thousands of fragments (1TB file) - presumably due to the rebuild process (these numbers might be influenced by ReFS block sharing).
This got so bad yesterday that one tenant running compact on their 3.5TB of backups hung the job on the VBR, and when job was restarted with compact disabled it ran an unscheduled integrity check and crippled the repo server for many hours. We've had to kick of defrag of the SP repo disks using Ultradefrag 7.1.4 (last free version) - let's hope it's ReFS shared blocks safe!. Of course this whole situation is made worse by MS's scheduled defrag skipping all files over 64MB and just leaving them fragmented. SP repo server disks are about 50% full.
- Backup to local Windows repo = some file fragmentation. 15GB VIB = 300 fraggles. 1.4TB VBK = 14,000 fraggles. (disk 50% full, NTFS with dedupe - these files not deduped yet). This level isn't unacceptable, but there still might be room for improvement.

I get that there are arguments for and against preallocation, especially for SP CC repo - but I think when restoring to hypervisor the case is quite clear cut. Fragmentation is a bad thing, and preallocation should help avoid it. For the restore (export) I mentioned earlier, it was creating a dynamic VHDX (it doesn't offer a choice of fixed?) on a freshly built hypervisor with 40x the amount of space the VHDX consumed so I don't think this level of fragmentation was at all inevitable. Restore was using VBR v10. I can open a case for this if you like, though I think this is something you can reproduce easily?

Thanks

Alex