Comprehensive data protection for all workloads
Post Reply
Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Domain Controllers fail Surebackup script tests

Post by Moebius » Jan 11, 2016 3:14 pm

In this infrastructure there are several geographically distributed datacenters, with all Windows servers in the same AD domain.
The main datacenter hosts one DC with all FSMO roles (Win 2008), plus two more DCs. At all other locations there is only one DC each (Win 2012R2).

Each location has a local Veeam server, and at each location a Surebackup job with all the local VMs runs daily. Each SB job has the "local" DC as the one VM that constitutes the Application Group, and the backup job(s) as linked jobs.

The problem is as follows. While at the main location (the one where the DC is Win 2008 and has the FSMO roles) the SB job runs successfully, at the other locations the SB job runs for several days with no problems, then suddenly the DC starts failing, and this being the application group, the entire job fails.
In the surebackup job, the DC starts correctly and replies to the ping test, but it fails all three script tests (DC, DNS and global catalog). If I get into the console of the DC and try to log in, I get the message "There are currently no logon servers available to service the logon request". There is no way to log into the server and no way (that I know of) to check the event logs.

I found this workaround. After rebooting the DC, and allowing another backup to be taken, the surebackup resumes working successfully for some weeks (2-4 weeks, not sure) then everything starts again.
This does not happen at the main location.

Does anyone have any hints on where to start investigating?

Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by Moebius » Feb 17, 2016 9:07 am

Bump.
Nobody? Really? That's hard to believe as there is nothing unusual with my setup. Does anybody see anything wrong with it?

P.Tide
Product Manager
Posts: 5120
Liked: 443 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by P.Tide » Feb 17, 2016 10:14 am

Hi,

Do you mind if I ask you several questions?
at the other locations the SB job runs for several days with no problems, then suddenly the DC starts failing, and this being the application group, the entire job fails.
Did you mean that your surebackup jobs at non-main locations run for several days and then fail? Please elaborate.
After rebooting the DC, and allowing another backup to be taken, the surebackup resumes working successfully for some weeks (2-4 weeks, not sure) then everything starts again.
So you reboot the DC in the SB job, take another backup of your real DC and after that your SB works fine for a couple of weeks and then fails again? If so then have you checked the amount of space available on your vPower NFS server?

Thank you.

Moebius
Veeam ProPartner
Posts: 158
Liked: 20 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by Moebius » Feb 17, 2016 11:25 am

PTide wrote:Did you mean that your surebackup jobs at non-main locations run for several days and then fail? Please elaborate.
Exactly. Sorry if this wasn't clear enough.
PTide wrote:So you reboot the DC in the SB job, take another backup of your real DC and after that your SB works fine for a couple of weeks and then fails again? If so then have you checked the amount of space available on your vPower NFS server?
No. To make the SB job work again I have to reboot the real DC and allow for another backup being taken. Then the following runs of the SB job start working again.

A key point here, in my opinion, is that when the SB job starts failing I try to log into the instant-recovered DC (created by the SB job) but I'm unable to do so because it says "There are currently no logon servers available to service the logon request". This looks like the reason the role scripts fail. This does not happen on the "real" DC, where I can always log in with no problems.
Again, rebooting the "real" DC and waiting for another backup to be taken fixes the issue (for some time).

P.Tide
Product Manager
Posts: 5120
Liked: 443 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by P.Tide » Feb 17, 2016 11:45 am

I have to reboot the real DC and allow for another backup being taken
That's strange. Since all SB jobs are performed in an isolated environment any actions with your real infrastructure should give no impact on SB job behaviour. Please open a support ticket so they can take a closer look at your configuration and post your case ID here.

Thank you.

P.Tide
Product Manager
Posts: 5120
Liked: 443 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by P.Tide » Feb 17, 2016 5:39 pm

One more question - do you have AAP enabled in your DC backup job? If so then please disable it and run another backup and try to log in into your test environment DC again.

Thank you.

remko.de.koning
Enthusiast
Posts: 90
Liked: 16 times
Joined: May 21, 2014 12:15 pm
Full Name: Remko de Koning
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by remko.de.koning » Feb 22, 2016 3:26 pm

Not sure if this is related but I had an issue once with my surebackups where after the restore in the lab environment the NIC was associated with the "public" network instead of the domain network.
This caused my scripts to fail as well because of the different firewall rules.
This happened on Win2008 machines.

grimson
Novice
Posts: 9
Liked: 1 time
Joined: Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by grimson » Sep 09, 2016 8:27 am

Support case:
[ID# 01899756] SureBackup fails on scripts (error 10060)
A key point here, in my opinion, is that when the SB job starts failing I try to log into the instant-recovered DC (created by the SB job) but I'm unable to do so because it says "There are currently no logon servers available to service the logon request". This looks like the reason the role scripts fail. This does not happen on the "real" DC, where I can always log in with no problems.
Again, rebooting the "real" DC and waiting for another backup to be taken fixes the issue (for some time).
I did encounter the same findings.
- SureBackup 1 DC 2012 all FSMO roles + 1 Exchange 2010 server in one job.
This was running OK every Sunday but failed this last Sunday with script errors '10060'.

When trying again I could not logon on the DC in the SureBackup environment, keep getting 'there are no logon servers available to service the logon reguest' until failing the job.

These servers do not reboot often and due to this topic I tried the reboot thingy :) with success.
(also this reboot was part of a maintenance window, updating Windows, ESX etc. but the reboot was there like the topic starter).

So if the conclusion is to reboot in time for a successfull SureBackup rises the following question:
- If you do not have SureBackup configured and do not reboot a DC in time, will the recovery of this DC more painfully?
- Why is this happening?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by foggy » Sep 09, 2016 10:18 am

Unfortunately, there's no case ID in this thread to check for the possible explanation of the issue, so please continue working with your engineer to identify the reasons of this behavior. Thanks.

agrob
Expert
Posts: 181
Liked: 19 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by agrob » Oct 20, 2016 10:32 am

Good Day

I have exactely the same problem. Surebackup job with DC 2012 R2 was running fine. Suddenly it stopped working. Error is 10060.
If i try to login when the DC is started from the Surebackup job i get the "there are no logon servers available..." DC's are getting patched and rebootet once a Month. There was no reboot or patch installation between the last running surebackup and the first failed one.
I have now rebootet the DC (without patch installation or any other changes), made a backup and then started the sure backup job again. it works now. i also can login to the DC started from the sure backup job obviously without the "there are no logon servers available..." message. it seems that other people have the same issue. is there already somone from veeam working on this or should i open a case if it happens again to further investigate the issue?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by foggy » Oct 20, 2016 11:42 am

Hi Andre, looking into Arian's case and I'm not sure the there was any resolution to this issue except the mentioned workaround. So please contact support for further investigation. Thanks.

agrob
Expert
Posts: 181
Liked: 19 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by agrob » Oct 21, 2016 10:36 am

Thanks foggy. I'll open a case once the problem does happen again.

agrob
Expert
Posts: 181
Liked: 19 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by agrob » Nov 02, 2016 6:59 am

so the problem did reocure again and i opened a ticket for it. id is 01965562. i'll post here if there is any fix for the problem

agrob
Expert
Posts: 181
Liked: 19 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by agrob » Nov 02, 2016 9:48 am 1 person likes this post

Support was very quick. To fix this issue, create the following reg key on the affected domain controller:
HKLM\Software\Veeam\Veeam Backup and Replication
create new DWORD and name it "UseGranularBcdRestore" and set value to 0

now create a new backup and the dc in the surebackup job will run fine again.
i still waiting for an info what exactely this key does. does it affect something else?

grimson
Novice
Posts: 9
Liked: 1 time
Joined: Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by grimson » Nov 07, 2016 8:32 am

Thanks for the update!
This issue seems to happen more frequent than before, unfortunately I could not troubleshoot it myself (remote client).

Googling got to this topic also;
veeam-backup-replication-f2/surebackup- ... 37064.html
Logic was changed in 9u2 so this explains the issue.

veremin
Product Manager
Posts: 16491
Liked: 1371 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by veremin » Nov 07, 2016 8:36 am

grimson wrote:Thanks for the update!
This issue seems to happen more frequent than before, unfortunately I could not troubleshoot it myself (remote client).
What were the results of support investigation? Did you also implement the regkey suggested above? Thanks.

agrob
Expert
Posts: 181
Liked: 19 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by agrob » Nov 07, 2016 8:57 am

since i have implemented the key surebackup works well.
Support told me that setting this key would fix a timing Problem. without this fix, dc boots into ad restore mode which causes this Problem.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by foggy » Nov 07, 2016 1:27 pm

As mentioned in the thread referred to above, Veeam B&R 9.5 will change the logic back, so there're will be no need to use it with the new version.

blithespirit
Veeam Software
Posts: 57
Liked: 9 times
Joined: May 03, 2016 2:06 pm
Full Name: Paul Szelsi
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by blithespirit » Nov 08, 2016 9:33 am

What version of Veeam Backup & Replication are you using? if V9 then Update 2 has a fix for the netlogon issue.

grimson
Novice
Posts: 9
Liked: 1 time
Joined: Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by grimson » Nov 08, 2016 9:40 am

v.Eremin wrote:What were the results of support investigation? Did you also implement the regkey suggested above? Thanks.
No, sorry, 'my case' was before this reg key / fix was mentioned;
At the support stage from Veeam we got to the conclusion that it was stuck in the recovery mode. Therefore had to obtain logs from this stage and had to login the server with the AD recoverymode credentials.
Unfortunately time got to me that further troubleshooting was nog possible (client is not accessible remotely so must do all actions on premises) any further and that a reboot just before the last veeam backup used with surebackup was sufficient at the moment.

My next visit will introduce the fix or Veeam 9.5 will already be available at that moment :).

grimson
Novice
Posts: 9
Liked: 1 time
Joined: Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by grimson » Nov 08, 2016 9:44 am

blithespirit wrote:What version of Veeam Backup & Replication are you using? if V9 then Update 2 has a fix for the netlogon issue.
I read that 9u2 introduced a change in logic therefore caused this issue. With the reg fix this changed logic was reversed.
So update 2 is not the fix but the issue here my guess?
9.5 should reverse the logic without the fix must be the conclusion?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Domain Controllers fail Surebackup script tests

Post by foggy » Nov 08, 2016 1:33 pm

Arian, your conclusion is correct.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 25 guests