If domain controller is down, can I do full vm restore with domain account?

[folerx]

Hi
Windows AD
I use domain account for app aware backup domain controller(s). If this DC go down, can I restore it? Who will authenticate?
Last Windows server update crash many DCs and I want to prepare.
Update 1/17/21: Microsoft has released OOB updates to fix the Windows Server bugs.
The latest Windows Server updates are causing severe issues for administrators, with domain controllers having spontaneous reboots, Hyper-V not starting, and inaccessible ReFS volumes until the updates are rolled back
Yesterday, Microsoft released the Windows Server 2012 R2 KB5009624 update, the Windows Server 2019 KB5009557 update, and the Windows Server 2022 KB5009555 update as part of the January 2022 Patch Tuesday.
After installing these updates, administrators have been battling multiple issues that are only resolved after removing the updates.

[Gostev]

Hi. No as no connection as it won't be possible for Veeam to establish any connection that uses a domain account. This is just one of many reason not to place backup server into your production domain. With the main and by far the biggest one being security (backup server isolation in case of cyber attack leading to a hacker taking over your production domain). Thanks!

[folerx]

Backup server is.not in domain, even is not in same network. I use domain account inside veeam for app aware procesing

[Gostev]

From credentials perspective, for full VM restore only a vCenter connection is needed. If that one does not use a domain account, then no concerns here. However, remember that domain controller may also implement DNS, meaning DNS resolution may not be possible if DC is down, while DNS is also required to establish network connections.

Bottom line: if you really want to know if you are able to restore in certain scenario, then schedule a downtime, re-create that specific scenario and do a test restore. Otherwise, you will be finding out everything you did not think of when you need to perform an actual production restore. This is exactly why recoverability testing is the crucial part of any backup strategy.

[remosito]

However, remember that domain controller may also implement DNS, meaning DNS resolution may not be possible if DC is down, while DNS is also required to establish network connections.

Ran into that one once. Since then all my backup infrastructure nodes (esxi hosts, repositories) are in the etc/host file...

[folerx]

Yes, DC and DNS is on same VM. How to deal with this if both DC is down and need to be restored? So no DC and DNS.

[folerx]

instant vm restore try 1

Code: Select all

1/26/2022 4:28:07 PM Error    Failed to publish DC-2 Error: Failed to remove original VM from disk
                              
                              Failed to remove disks that are not present in backup from the original VM
                              
                              Cannot authenticate user.
                              Soap fault. Cannot complete login due to an incorrect user name or password.Detail: '', endpoint: ''
                              SOAP connection is not available. Connection ID: [vcenter-2.domain.local].
                              Failed to create NFC download stream. NFC path: [nfc://conn:vcenter-2.domain.local,nfchost:host-10,stg:datastore-57409@DC-2_1f2749a6-436f-4be8-8c6c-308ec4ddcfcc/DC-2.v
__________________

instant vm restore try 2

Code: Select all

1/26/2022 4:34:36 PM Error    Failed to publish DC-2 Error: Failed to connect to host esxi1.domain.local
                              
                              Failed to login to "vcenter-2.domain.local", port 443, user "domain\domainaccount", proxy srv: port:0
                              
                              Cannot complete login due to an incorrect user name or password.

___________________

full vm restore try

Code: Select all

1/26/2022 4:35:06 PM Error    Restore job failed Error: NFC storage connection is unavailable. Storage: [stg:datastore-31303,nfchost:host-88785,conn:vcenter-1.domain.local]. Storage display name: [SATA-EN02-01].
                              Failed to create NFC download stream. NFC path: [nfc://conn:vcenter-1.domain.local,nfchost:host-88785,stg:datastore-31303@DC-1/DC-1.vmx].
                              --tr:Unable to open source file [nfc://conn:vcenter-1.domain.local,nfchost:host-88785,stg:datastore-31303@DC-1/DC-1.vmx].
                              --tr:Failed to copy local file. Source: [nfc://conn:vcenter-1.domain.local,nfc

[veremin]

Daniel,

Kindly, avoid posting debug log snippets here - this is against the forum rules: neither team behind community is able to analyze them, nor other community members benefit from reading them.

So please discuss the experienced issue with the support team and post the support case number.

This will be the most productive way not only to get the problem fixed, but also to allow us and other forum participants to have something concrete to refer to, should the similar issue appear.

Thank you for the understanding!

P.S.: We will preserve the log snippets for several days, so you can provide them to the support engineer. After that, the logs will be removed.

[folerx]

Ran into that one once. Since then all my backup infrastructure nodes (esxi hosts, repositories) are in the etc/host file...

C:\Windows\System32\drivers\etc\hosts
I need to write data about my infrastructure into this file on veeam backup server so I can restore without dns?
For example from this topic and restoring dc/dns I need to write into backup server hosts file name and ip for this dc/dns? anything else, like ip for v-center server etc?

@veremin
Sorry for posting error code.

[veremin]

Absolutely no worries, it's just a reminder of the most effective procedure to solve the problem

[Gostev]

Yes, DC and DNS is on same VM. How to deal with this if both DC is down and need to be restored? So no DC and DNS.

Usually most environments have a secondary DNS server as per best practices, otherwise you have a single point of failure.
Anyway, when DNS is completely down, there's always an option to re-add everything to Veeam using IP addresses.

[rennerstefan]

C:\Windows\System32\drivers\etc\hosts
I need to write data about my infrastructure into this file on veeam backup server so I can restore without dns?
For example from this topic and restoring dc/dns I need to write into backup server hosts file name and ip for this dc/dns? anything else, like ip for v-center server etc?

Well, like on most of the operating systems on the market, you can use the local "hosts" file to do exactly what you ask about here.
Using names that get translated to IPs. The hosts file is always used before DNS server is contacted so be sure you know where you added what and that you have to change it in case your environment changes.
to your question. It will not only be about adding it to your VBR servers hosts file but also to other Veeam components involved (in case you don't have a "all-in-one" backup server).
DNS or better say name-resolution is key for communication between components (I guess that you did not use IPs over FQDN for the installation).
So best will be to get prepared for such DR scenarios and even test is out (in a non production test lab), so you can handle it if it happens.
I would not recommend to use "hosts" files over the usual DNS as in my experience someone will forget that it is configured like this years later during an update or environment change and then the troubleshooting begins.

Thanks

[Gostev]

My only concern with the hosts file on the backup server is that it only helps network connections which Veeam establishes itself. While for example vCenter to ESXi connections required for restore may still be failing due to vCenter server not being able to resolve ESXi host name. However, you should always be able to add ESXi host via IP address directly to Veeam, and perform the emergency restore this way (VM data will be sent from Veeam directly to ESXi host in this case).

There's also one cool restore scenario to keep in mind: instant restore into a Hyper-V VM of the built-in Hyper-V on the backup server itself. We support this for any image-level backup (in other words, you can restore a VMware VM backup or a physical server backup). But I would go "direct to ESXi" emergency restore first before resorting to that, just because it only takes a few seconds to register ESXi host by IP address with Veeam.

[folerx]

Conclusion is to add some independent DNS server on network, add servers to veeam via ip address instead fqdn and better to ignore editing hosts file?

[Gostev]

I would go with redundant DNS. While adding servers to Veeam via IP addresses is totally fine in environments with static IP addresses, DNS is a pre-requisite for Kerberos authentication, Veeam CDP functionality etc. Besides, DNS is probably as important for most other software you're using, so it makes sense to ensure it is highly available.

And as I've said, in case of a complete disaster when everything fails including your redundant DNS, you can still very quickly restore a DNS server by temporarily registering required ESXi host by IP address in Veeam and restoring directly to it. Even if this same ESXi will be already registered with Veeam (over vCenter connection or directly) using a DNS name, you can still register it again by IP address. I'm highlighting this because normally Veeam will not let you register the same host twice.

[dloseke]

From credentials perspective, for full VM restore only a vCenter connection is needed. If that one does not use a domain account, then no concerns here. However, remember that domain controller may also implement DNS, meaning DNS resolution may not be possible if DC is down, while DNS is also required to establish network connections.

Bottom line: if you really want to know if you are able to restore in certain scenario, then schedule a downtime, re-create that specific scenario and do a test restore. Otherwise, you will be finding out everything you did not think of when you need to perform an actual production restore. This is exactly why recoverability testing is the crucial part of any backup strategy.

With the recent update affecting Windows Server 2012 R2 DC's boot looping, I performed restores of the DC's. The biggest issue was DNS to get to the SMB repository. Eventually I got one DC to stay up long enough to get DNS queries to find the share. Had that not worked, I was ready to edit the hosts file to point Veeam to the share. Fortunately, that share used NAS based authentication. Had it been using LDAP/AD integration, that would have been another issue. Don't use AD creds if you can avoid it. In this case, I was restoring entire VM's, but had it been something like a file level restore or something that used application aware processing, that would have been another story.

[dloseke]

Usually most environments have a secondary DNS server as per best practices, otherwise you have a single point of failure.
Anyway, when DNS is completely down, there's always an option to re-add everything to Veeam using IP addresses.

And if that second point of failure fails..... In my case I got DNS running, but the hosts file was my next stop to avoid having to re-add everything to Veeam by IP.