Domain Controllers fail Surebackup script tests

[Moebius]

In this infrastructure there are several geographically distributed datacenters, with all Windows servers in the same AD domain.
The main datacenter hosts one DC with all FSMO roles (Win 2008), plus two more DCs. At all other locations there is only one DC each (Win 2012R2).

Each location has a local Veeam server, and at each location a Surebackup job with all the local VMs runs daily. Each SB job has the "local" DC as the one VM that constitutes the Application Group, and the backup job(s) as linked jobs.

The problem is as follows. While at the main location (the one where the DC is Win 2008 and has the FSMO roles) the SB job runs successfully, at the other locations the SB job runs for several days with no problems, then suddenly the DC starts failing, and this being the application group, the entire job fails.
In the surebackup job, the DC starts correctly and replies to the ping test, but it fails all three script tests (DC, DNS and global catalog). If I get into the console of the DC and try to log in, I get the message "There are currently no logon servers available to service the logon request". There is no way to log into the server and no way (that I know of) to check the event logs.

I found this workaround. After rebooting the DC, and allowing another backup to be taken, the surebackup resumes working successfully for some weeks (2-4 weeks, not sure) then everything starts again.
This does not happen at the main location.

Does anyone have any hints on where to start investigating?

[Moebius]

Bump.
Nobody? Really? That's hard to believe as there is nothing unusual with my setup. Does anybody see anything wrong with it?

[PTide]

Hi,

Do you mind if I ask you several questions?

at the other locations the SB job runs for several days with no problems, then suddenly the DC starts failing, and this being the application group, the entire job fails.

Did you mean that your surebackup jobs at non-main locations run for several days and then fail? Please elaborate.

After rebooting the DC, and allowing another backup to be taken, the surebackup resumes working successfully for some weeks (2-4 weeks, not sure) then everything starts again.

So you reboot the DC in the SB job, take another backup of your real DC and after that your SB works fine for a couple of weeks and then fails again? If so then have you checked the amount of space available on your vPower NFS server?

Thank you.

[Moebius]

Did you mean that your surebackup jobs at non-main locations run for several days and then fail? Please elaborate.

Exactly. Sorry if this wasn't clear enough.

So you reboot the DC in the SB job, take another backup of your real DC and after that your SB works fine for a couple of weeks and then fails again? If so then have you checked the amount of space available on your vPower NFS server?

No. To make the SB job work again I have to reboot the real DC and allow for another backup being taken. Then the following runs of the SB job start working again.

A key point here, in my opinion, is that when the SB job starts failing I try to log into the instant-recovered DC (created by the SB job) but I'm unable to do so because it says "There are currently no logon servers available to service the logon request". This looks like the reason the role scripts fail. This does not happen on the "real" DC, where I can always log in with no problems.
Again, rebooting the "real" DC and waiting for another backup to be taken fixes the issue (for some time).

[PTide]

I have to reboot the real DC and allow for another backup being taken

That's strange. Since all SB jobs are performed in an isolated environment any actions with your real infrastructure should give no impact on SB job behaviour. Please open a support ticket so they can take a closer look at your configuration and post your case ID here.

Thank you.

[PTide]

One more question - do you have AAP enabled in your DC backup job? If so then please disable it and run another backup and try to log in into your test environment DC again.

Thank you.

[remko.de.koning]

Not sure if this is related but I had an issue once with my surebackups where after the restore in the lab environment the NIC was associated with the "public" network instead of the domain network.
This caused my scripts to fail as well because of the different firewall rules.
This happened on Win2008 machines.

[grimson]

Support case:
[ID# 01899756] SureBackup fails on scripts (error 10060)

A key point here, in my opinion, is that when the SB job starts failing I try to log into the instant-recovered DC (created by the SB job) but I'm unable to do so because it says "There are currently no logon servers available to service the logon request". This looks like the reason the role scripts fail. This does not happen on the "real" DC, where I can always log in with no problems.
Again, rebooting the "real" DC and waiting for another backup to be taken fixes the issue (for some time).

I did encounter the same findings.
- SureBackup 1 DC 2012 all FSMO roles + 1 Exchange 2010 server in one job.
This was running OK every Sunday but failed this last Sunday with script errors '10060'.

When trying again I could not logon on the DC in the SureBackup environment, keep getting 'there are no logon servers available to service the logon reguest' until failing the job.

These servers do not reboot often and due to this topic I tried the reboot thingy with success.
(also this reboot was part of a maintenance window, updating Windows, ESX etc. but the reboot was there like the topic starter).

So if the conclusion is to reboot in time for a successfull SureBackup rises the following question:
- If you do not have SureBackup configured and do not reboot a DC in time, will the recovery of this DC more painfully?
- Why is this happening?

[foggy]

Unfortunately, there's no case ID in this thread to check for the possible explanation of the issue, so please continue working with your engineer to identify the reasons of this behavior. Thanks.

[agrob]

Good Day

I have exactely the same problem. Surebackup job with DC 2012 R2 was running fine. Suddenly it stopped working. Error is 10060.
If i try to login when the DC is started from the Surebackup job i get the "there are no logon servers available..." DC's are getting patched and rebootet once a Month. There was no reboot or patch installation between the last running surebackup and the first failed one.
I have now rebootet the DC (without patch installation or any other changes), made a backup and then started the sure backup job again. it works now. i also can login to the DC started from the sure backup job obviously without the "there are no logon servers available..." message. it seems that other people have the same issue. is there already somone from veeam working on this or should i open a case if it happens again to further investigate the issue?

[foggy]

Hi Andre, looking into Arian's case and I'm not sure the there was any resolution to this issue except the mentioned workaround. So please contact support for further investigation. Thanks.

[agrob]

Thanks foggy. I'll open a case once the problem does happen again.

[agrob]

so the problem did reocure again and i opened a ticket for it. id is 01965562. i'll post here if there is any fix for the problem

[agrob]

Support was very quick. To fix this issue, create the following reg key on the affected domain controller:
HKLM\Software\Veeam\Veeam Backup and Replication
create new DWORD and name it "UseGranularBcdRestore" and set value to 0

now create a new backup and the dc in the surebackup job will run fine again.
i still waiting for an info what exactely this key does. does it affect something else?

[grimson]

Thanks for the update!
This issue seems to happen more frequent than before, unfortunately I could not troubleshoot it myself (remote client).

Googling got to this topic also;
veeam-backup-replication-f2/surebackup- ... 37064.html
Logic was changed in 9u2 so this explains the issue.

[veremin]

Thanks for the update!
This issue seems to happen more frequent than before, unfortunately I could not troubleshoot it myself (remote client).

What were the results of support investigation? Did you also implement the regkey suggested above? Thanks.

[agrob]

since i have implemented the key surebackup works well.
Support told me that setting this key would fix a timing Problem. without this fix, dc boots into ad restore mode which causes this Problem.

[foggy]

As mentioned in the thread referred to above, Veeam B&R 9.5 will change the logic back, so there're will be no need to use it with the new version.

[blithespirit]

What version of Veeam Backup & Replication are you using? if V9 then Update 2 has a fix for the netlogon issue.

[grimson]

What were the results of support investigation? Did you also implement the regkey suggested above? Thanks.

No, sorry, 'my case' was before this reg key / fix was mentioned;
At the support stage from Veeam we got to the conclusion that it was stuck in the recovery mode. Therefore had to obtain logs from this stage and had to login the server with the AD recoverymode credentials.
Unfortunately time got to me that further troubleshooting was nog possible (client is not accessible remotely so must do all actions on premises) any further and that a reboot just before the last veeam backup used with surebackup was sufficient at the moment.

My next visit will introduce the fix or Veeam 9.5 will already be available at that moment .

[grimson]

What version of Veeam Backup & Replication are you using? if V9 then Update 2 has a fix for the netlogon issue.

I read that 9u2 introduced a change in logic therefore caused this issue. With the reg fix this changed logic was reversed.
So update 2 is not the fix but the issue here my guess?
9.5 should reverse the logic without the fix must be the conclusion?

[foggy]

Arian, your conclusion is correct.