EFI boot issue

[carlos.hernandez]

Hello,
Sorry if this post is a duplicate, searched and couldn't find something similar. Has anyone seen the issues outlined here? http://communities.vmware.com/message/2170214

[Vitaliy S.]

Hello Carlos,

EFI boot support was added as a part of version 6.1 patch 1 update, since then we haven't heard any negative feedback on this.

Thank you!

[carlos.hernandez]

Hi,
I just started getting this issue today with my 2012 domain controller. Deleting the nvram file seems to be the only fix. I'm running Veeam 6.5 with latest patch.. do you think this issue is unrelated to Veeam then?

Thank you

[Vitaliy S.]

If you can reproduce this issue without Veeam, then this should be investigated by VMware support team. Anyway, I would appreciate if you could report about your findings to our technical support team first.

[lejim]

Hi,

Interesting, I also get the same issue with me domain controller only.
Like I said in the vmware communities post, I ran a bunch of test and was unable to reproduce the bug, using only vmware snapshots from vcenter and lot of backup and replication job from veeam on a fresh 2012 vm and on a 2012 promoted to dc.
On my first DC impacted I removed the corrupted nvram file, it was a week ago, as for now the issue doesn't appear again.
On friday my second DC was impacted. Same path for resolution, still working.
I opened a case this week end at veeam, they just awnser that it should be vmware related and I should open a case at vmware. Vmware want me to open a case at veeam
Oh I provided a corrupted nvram to veeam in my case.

Ok. At this moment everything is working but issue could came back.

But one strange thing is that it seems only DC get their nvram corrupted. So maybe we should open a case at microsoft ^^

The bad thing is that restore job based on veeam lab won't work anymore since the vm can't boot.

[VMwareDarius]

Hi Vitaliy,

I'm one of the developers of VMware's virtual EFI firmware, and I'm investigating this issue from our side. We not yet succeeded in replicating this issue here at VMware, although I'm yet to gain access to a Veeam setup, and we have observed what so far seems to be a strong correlation between reported Veeam usage and this particular failure mode.

The NVRAM files we're seeing from affected customers have had the EFI Boot Options (the Boot#### variables) replicated massively until there is no more EFI NVRAM space left. The final failure occurs when there is insufficient NVRAM to write the bloated BootOrder variable (~400 entries), which leaves the VM without a BootOrder and renders the VM unbootable until the .nvram file is removed.

Possibly relevant information: Our virtual EFI firmware sometimes uses one of the "reserved" bits in the Attributes field of the EFI_LOAD_OPTION descriptor (to solve a problem that we could find no other way of solving). We're seeing that any Boot Option that our firmware automatically creates with that reserved bit set is subsequently replicated many times, with the copies having that reserved bit cleared. Boot Options that do not have the reserved bit set do not get replicated.

I emphasize that we haven't yet determined how the entry is replicated, so it still might not be anything to do with Veeam, or it might be that Veeam is using a Windows API or tool which indirectly triggers this behavior. I'd be interested to learn more about how Veeam interacts with the EFI Variable Store and/or the Windows Boot Configuration Database so that we can try to get to the bottom of this.

Thanks!
--
Darius

[Gostev]

Hi Darius, we are using ImportStore and ExportStore functions to interact with BCD. Thanks!
http://msdn.microsoft.com/en-us/library ... s.85).aspx
http://msdn.microsoft.com/en-us/library ... s.85).aspx

[dellock6]

I would only add to the discussion that is soooo nice to see two companies working together to solve customers' issues instead of the usual "pass the buck" behaviour.

Really appreciated, even if I'm not one of those beeing affected by the problem.

Thanks guys!

Luca.

[Gostev]

We have confirmed the issue without Veeam present using just BCD import/export functions, but also in our testing we have found an easy way to prevent the issue from happening (cleaning BCD before importing), so we are going to address this from our side in Patch 3 which we were planning to release in the coming weeks anyway.

[regi]

Thanks for the patch I have installed and done 10+ application aware backups on my 6 X Win 2012 VM AD's & all seems good but I like to test in few weeks' time. Hope your patch fixed this issue which is only happening in Win 2012 AD VM's.

[regi]

Happy to say all my Win 2012 AD VMs are working fine, looks like patch 3 worked. Like to monitor another 2 weeks. Thanks Veeam & VMWare – Keep up the good work