Windows 2012 R2 DC fails to boot in Sure Backup

[cffit]

I have a case open for this: 00644604

I have an issue with a Sure Backup job. My Windows 2012 R2 DC suddenly fails to come up after working fine every day for many months. It fails where it does the DNS, DC and GC tests. The error code is 10061. The support engineer suggested I check our production version of this VM to see if these services are working there. That's not very helpful. He also suggested I delay the VMware Tools services at startup, which I will try tomorrow after tonight's backups. Here's what I've done so far:

1. Booted up the single DC in a SB job with no other vms. It fails the same way.
2. Booted up the single DC in a SB job and watched the console. When it is up I try to login to it, but it says there are no login servers available.
3. Booted up the problem DC with another DC (Windows 2008 R2). Set the other DC to boot first, then when the problem DC booted up I was able to log into it. Here I could see that the problem DC is booted up in Safe Mode and never leaves there. That explains why the DNS test doesn't pass as the DNS service doesn't start in Safe Mode.

I noticed that this started happening after I installed a few month's worth of MS patches on the problem DC. It didn't happen the very night I installed, them, but it did start happening the next day and has been a problem since. With that, I'm not sure if one of the MS patches could be the issue. Any help or suggestions I can get on here would be appreciated.

[foggy]

Looks like the DC stays in DSRM mode longer than usual. You can try to increase application group timeouts and see whether this helps.

Also, the list of installed MS updates could help support to identify the issue.

[cffit]

I'm not getting anywhere with support. Their suggestions are not helpful at all I don't feel the tech has a grasp on what is going on. I've tried extending the time to 10 minutes and that doesn't help. When I watch the console, it boots up into safe mode and then never leaves that state. It just sits there until 10 minutes passes and the lab fails. If I login to the console and manually run the commands that I think is supposed to happen automatically, and it reboots all is fine. But then the next time I run it again it fails unless I manually do this.

[foggy]

As far as I can get from the case notes, engineer's suggestions could help identify the nature of the issue. However, if you feel your case is not handled properly, you can always ask to escalate it to a higher tier.

10 minutes could still be not enough. Setting extremely high timeout (e.g. 1 hour) can let it boot finally. Probably MS could clarify why does it take so long to go through DSRM.

[cffit]

In one response he asks me if I have the AD server checks set on the SB job. That was after I told him initially that it fails at that point, so obviously they are set.

Then he has me run the commands manually that VEEAM should do for me while the VM is in safe mode, which is great, but then as soon as the SB lab is shutdown that is all gone and the problem will happen every time after. He thought this was a resolution, not a test.

We have been a VEEAM customer for several years now and I dread calling into support. I like that they are US based, but I never seem to get techs who are able to help me. The one time I did get good assistance was when I ended up getting escalated, but that took many days to get to. That's when I end up figuring things out on my own or by coming on here and getting the input of you guys and other customers.

I will set the timeout for one hour and see what happens and post back. I can say this isn't an issue with Windows 2008 R2 DCs in our environment.

Thanks for the quick response.

[cffit]

After letting it go one full hour it still times out with the application group failing to start.

[VladV]

Try running the commands listed under Solution from this article manually and see if it reboots in normal mode.

http://www.veeam.com/kb1277

Maybe this will help the techs figure out what is happening.

I had this issue with 2008R2 (case #00459425) and, like in your case, it took a lot of remote sessions and a lot of emails just to make the support team understand the situation (although it should have been clear from the first remote session). After approximately 2 months (if I remember correctly) of emails and support sessions they figured out the cause and sent me a patch before the final bugfix was released in Patch #4 (build 7.0.0.871).

[cffit]

Hi Vlad,

I did try the commands in the article above. I booted up the DC in SureBackup, then while it was waiting the 10 minutes that I had set for the application initialization, I logged into the console and entered these commands and rebooted. Then the server came up like it should. I guess for some reason this isn't happening automatically. However, once I do this it's just for the current SB job and the next time it runs it's broken just like before.

I feel your pain with support. They are friendly and nice to work with, but I feel like it takes way too long to get a resolution and I find it faster to come on here and seek help.

[cffit]

And also, I do have Patch 4 on.

[cffit]

I'm asking for escalation with support and getting them new log files.

My other issue with support is that they tend to send you one email per day and then don't communicate back until the next day. That makes resolution take forever.

[VladV]

Yes, that happens to me every time (except if you have a critical case). Because of this you have to be extremely careful when writing. If not, after 24 hours, you find out that support misunderstood your email and you have to wait another day to see if you were clear enough the second time around.

[cffit]

After a week of working with the first tech that took my case, I asked to be escalated. Once escalated, the tech I then got was able to figure out the issue within 5 minutes. It's a bug that requires a dll replacement within the VEEAM application. The fix file was named Fix_30998, so I assume the number 30998 is a number associated with this bug/issue. I asked how the first tech wasn't able to help me with this after spending a week on it, and the second level tech mentioned that this issue is listed in their internal wiki page under their bug fixes, and he would remind that initial tech to check that wiki for things like this.

Hopefully that helps others who might run into the same issue. I'm not sure if this is the same fix that you had Vlad, but if so, it must not be included in patch 4.

[foggy]

After a week of working with the first tech that took my case, I asked to be escalated. Once escalated, the tech I then got was able to figure out the issue within 5 minutes.

That's why I've actually encouraged you to escalate the case a week ago.

Hopefully that helps others who might run into the same issue. I'm not sure if this is the same fix that you had Vlad, but if so, it must not be included in patch 4.

I've just checked and this particular fix was not included in patch 4, but will be addressed in one of the following releases/updates. Thanks for getting back with the update!

[cffit]

This broke again after upgrading to v8. I have a case open again on it. Just wanted people to know who might be reading this. I thought the issue was fixed in v8, but it's not looking that way.

[foggy]

No, not in v8. However, a private fix will be made available for v8 as well.

[cffit]

There was no hotfix for v8, so they are working on developing one for me. I can't believe that I'm the only person seeing this issue, but apparently so.

[hyvokar]

I had the same problem. Starting the problem DC in safe mode and running
bcdedit /set safeboot dsrepair
bcdedit /deletevalue safeboot
shutdown -t 01 -r
and after taking new backup it worked a while. Never actually got the DLL fix and again my sure backup hasnt been working for a couple of weeks since I haven't had time to fix it again. Glad to hear fix for v8 is coming.

[veremin]

If I were you, I would open a separate ticket on that in order to get a private fix as soon as possible. Thanks.

[cffit]

I will post back when support has a hotfix for me, and then you can reference my case number at the beginning of this thread and ask for the same thing. Hopefully that will save you the time of going through all the troubleshooting that they had me do. They expected to have it ready last week or this week.

[cffit]

Just to vent a little bit... I finally got this working with the fix VEEAM support had to make. Then as soon as I put patch 1 on for v8 it all broke AGAIN and my DCs no longer work in SureBackup. Called support AGAIN and now they are looking at a fix for it AGAIN. This makes the third time I've gone through this process from patches and upgrades breaking this. I would think if they have been aware and making fixes for this issue for quite some time, they could add this fix into the new versions and new patches for the new version. Or at a minimum, realize ahead of time that they will need to have a fix ready with each patch or version upgrade. I'm used to VEEAM being more attentive than this and it's been frustrating how much time I've had to spend on it through the 3 incidents.

I'm currently waiting on the fix for v8 with patch 1 on.

[foggy]

For some reason, this hotfix was not included into Patch 1, however is already included in the next patch, so this was the last time you had to request it. Sorry for inconvenience.

[cffit]

And for the fourth time in a row after either installing a patch or upgrading, after installing the latest update for VEEAM in V8 it broke my SureBackup jobs yet again. How is this not fixed yet?

I'll have to open a support case and wait for the developers to create another patch. Very frustrating.

[foggy]

Christensen, first of all, sorry for the inconvenience caused by this issue. Actually, the fix I was referring to above, is included with Veeam B&R v8 Update 2, I've just checked this. So most likely you're experiencing another issue showing up similar symptoms. I've already asked QA guys to look into your latest case without waiting for it to be escalated.

[cffit]

The case number is 00911043.

It's a pain to have to deal with this for the fourth time now. It's bad enough I have to worry about patching the patch, but each time I end up spending a considerable amount of time with support gather logs and explaining things over, then going through their suggestions each time that don't work, and then finally I am able to escalate and get to someone that can then work with a developer to get the patch created.

The symptoms now are exactly the same as they were the last three times. The DC doesn't boot into the proper mode and then ends up failing with an application initialization error.

[foggy]

It's a pain to have to deal with this for the fourth time now. It's bad enough I have to worry about patching the patch, but each time I end up spending a considerable amount of time with support gather logs and explaining things over, then going through their suggestions each time that don't work, and then finally I am able to escalate and get to someone that can then work with a developer to get the patch created.

This time you shouldn't wait for escalation that long.

[cffit]

I apologize for sounding so crabby. It's very frustrating though. If you can put in a word for me on this case I would appreciate it.

[foggy]

I totally understand your frustration, no worry. Let's see what R&D guys can suggest after taking a closer look at the logs.

[michaelryancook]

I've been using the commands from KB1277 on my DCs in surebackups as well. I thought my issue stemmed from not using VSS on the backups. Could these commands be run in a script within the Surebackup job perhaps?

[cffit]

Michael Cook, did you just update to Patch/Update 2 for v8 or are you on an earlier version? If you are on an earlier version, call support and there is a patch you can apply that fixes this. There are usually one or two dll files that you replace and then everything works fine. One other thing is that sometimes I need to set the Application Initialization Timeout for the DC in the Application Group to be as much as 600 seconds.

[cffit]

Foggy - My support person is suggesting it could be an anti-virus issue and is asking me to re-create all parts that make up the SB job. This all worked the day before the patch, and now it doesn't, and the symptoms are exactly the same as all the other times a patch broke SB for me. I really don't want to go through all this troubleshooting again as I've done all this in the past more than once.