SYSVOL stops replicating after Veeam 7 backup.

[Fiskepudding]

I recently discovered that our SYSVOL was not replicating. We made a successful migration from FRS to DFSR not long ago.
I did a lot of testing with backups and restore with DFSR, it then seemed “unbreakable”.
More on that can be found here: http://forums.veeam.com/viewtopic.php?f=2&t=7000.
These tests were all done with Veeam 6.5. I did a lot of backups and restores testing exactly SYSVOL replication.

To fix SYSVOL replication I had to do a authoritative sync of SYSVOL. It worked well during the day, but next day it was not working again.
In the end one suspect was Veeam, because there were some DFSR warnings and logs during the Veeam backup window. And there was an pattern of fhis.

So a new authoritative sync needed to be done. Then ran a new backup, and SYSVOL replication was again broken.
To cut it short, Veeam 6.5 backups did NOT break our SYSVOL replication, (neither does windows server backup). But Veeam 7 does.

A bit about the setup, for those interested:
2 DC’s, both Win 2008 R2
1 backup job for each DC, which are scheduled to start at the same time.

I have to add that, this ONLY happens if I run the jobs at the same time, manual or by schedule.
In 6.5 we also had the jobs set up to RUN at the same time. (did not have this problem then)

I have tested this many times over and over, and the only way I can keep my SYSVOL replication, is to NOT run the jobs at the same time.
If I run the jobs at the same time (in parallel), one DC will show an event 4102 in the DFSR logs. Witch basically leaves the server waiting for “Initial replication”:

Code: Select all

“The DFS Replication service initialized the replicated folder at local path C:\Windows\SYSVOL_DFSR\domain and is waiting to perform initial replication. The replicated folder will remain in this state until it has received replicated data, directly or indirectly, from the designated primary member.”

If I run the jobs at different times (in sequence), the above event also shows, but is followed with an event 4604, and replication works.

Code: Select all

  “The DFS Replication service successfully initialized the SYSVOL replicated folder at local path C:\Windows\SYSVOL_DFSR\domain. This member has completed initial synchronization of SYSVOL with partner DC001.domain.com.  To check for the presence of the SYSVOL share, open a command prompt window and then type "net share".”

Is this a known issue, or expected behaviour?
If not, what have changed from 6.5 to 7, which can explain this change in behaviour?

Workaround is easy, just not run the DC backups at the same time. So for me this is no longer a problem.
But I guess there might be others that run DC jobs at the same time, and I guess they should double check that SYSVOL replication is working. Our SYSVOL content does not change that often, so it was purely luck we discovered this so soon.
Off cause this could be an odd thing related to our system, something with the FSR -> DFRS migration, or something …

[foggy]

Espen, thanks for pointing that out, we would like to look into this closer, so please kindly open a case with technical support, provide all the logs and post the case ID here.

[Fiskepudding]

CASE ID: 00444733

[foggy]

Thanks, I will pass this to R&D and get back once I have something to share.

[trafsta]

Looks like we are having the EXACT same problem here. I noticed our DFSR SYSVOL replication was broken, traced it back to Veeam v7 (event log errors for DFSR started Aug 29th, that's when we switched to Veeam v7). We too switched from FRS to DFSR around 3-6 months ago, but I'm not sure that has anything to do with it.

Our ticket # is 00445116. They have linked our ticket to yours Fiskepudding. They are also checking to see if anyone else has reported this problem.

One thing, could you please elaborate on how you fixed your broken DFSR sysvol replication? Did you do an authoritative SYSVOL restore (mentioned here http://support.microsoft.com/kb/290762 ) [Also mentioned here http://social.technet.microsoft.com/For ... -to-resync ] or did you do something different? The reason I ask is you mentioned an "authoritative SYNC", but I'm guessing a sync is the same as a restore... I've performed one of these in the past but I want to make sure thats what you did... if you could give any further information on exactly what you did I would greatly appreciate it.

Thanks!

[trafsta]

Errr, those articles are for FRS... I guess this is what I want: http://support.microsoft.com/kb/2218556 <-- for DFSR

[trafsta]

I did an authoritative restore following the article above and it fixed the SYSVOL replication problem. I have this same Veeam backup job starting in 15 minutes, this time the two DC's are in the FIRST and LAST spots of a 12 server backup, so as per your testing that should stop it from causing issues due to parallel processing of the DCs. I will report back to let you know the results...

[trafsta]

Confirmed what Fiskepudding discovered - stopping the parallel backups of DC VMs prevents DFSR SYSVOL Replication from breaking. It is now working just fine.

I will await a patch to prevent this from happening in the future when DCs are backed up in parallel

[emk23]

I also had SYSVOL stop replicating recently (around 8/29 - which was shortly after V7 upgrade).

I performed an authoritative sync of AD from the PDC and all has been well since then.

I just removed all DC's from Veeam except the PDC. Will wait and see where this thread leads before putting them back in.

To add some info, our domain was DFSR from the get go. Windows 2008 R2 native since day 1. No FRS ever.

[Fiskepudding]

Errr, those articles are for FRS... I guess this is what I want: http://support.microsoft.com/kb/2218556 <-- for DFSR

Yes, used this procedure. I did authoritative synchronization of DFSR. The article fails to mention that you have to stop DFS Replication service on all DC's before starting the process.
As long as i dont run the jobs in parallel, i have no problems with SYSVOL

Good to hear I am not alone in this boat, will probably help Veeam to prioritize this if many users are affected.

[Fiskepudding]

To add some info, our domain was DFSR from the get go. Windows 2008 R2 native since day 1. No FRS ever.

That was very good to hear.
Even though SYSVOL replication worked fine after migration with Veeam 6.5, I have wondered if something went sideways with migration.

[Gostev]

All, I asked R&D to take a look into this first thing on Monday. I'd be surprised if the issue is specific to v7, I think parallel processing just made it surface... probably, you could still run into this even with previous versions, by running multiple jobs in parallel. Anyway, the research will show.

[trafsta]

Thanks Gostev, we appreciate it. And thanks Fiskepudding for finding and posting about this issue about a day before I discovered it. You saved me a lot of troubleshooting and research time, that's for sure!

[Fiskepudding]

Glad I could help trafsta.

Gostev: With 6.5 we ran backup of DC1 and DC2 at the exact same time, in two separate jobs, different proxys and different repositories. (just like now)
So they RAN at the exact same time without creating a SYSVOL issue. In "parallel".

When I run the exact same two jobs after upgrading to V7, (only changed compression back to the new optimal) i get this SYSVOL problem.
I have enabled parallel processing, but i dont se how that should impact, since it’s only these two jobs. And it RAN them at same time with 6.5.
Have double checked the Veeam 6.5 logs, and they did run at the same time.

We also have a Veeam “Free” server that we kept the 6.5 snapshot of, and when I Veeam Zip the DCs at the same time there is no problem.
When I Veeam Zip the same DC’s at the same with V7, SYSVOL stops working. (I must admit with Veeam 7 “Free” it was hard to get them to start at the same time for some reason. Usually one DC was 30% done, when the other started. THIS caused no error with SYSVOL, only when they did start at the same time, like 1-2% dfference. That leads me to believe that the problem occurs very early in the backup process.

Looking forward to hear from Support

[Gostev]

THIS caused no error with SYSVOL, only when they did start at the same time, like 1-2% difference. That leads me to believe that the problem occurs very early in the backup process.

Based on these observations, it is possible that the problem only happens when application-aware guest processing of DC VMs happens at the same time. This can be easily achieved by putting all domain controllers in the same job, and starting this job before any other job (so that there are plenty of backup infrastructure resources available to process multiple VMs from this job in parallel).

[m.novelli]

Based on these observations, it is possible that the problem only happens when application-aware guest processing of DC VMs happens at the same time. This can be easily achieved by putting all domain controllers in the same job, and starting this job before any other job (so that there are plenty of backup infrastructure resources available to process multiple VMs from this job in parallel).

Hi Gostev, putting the first DC at the top of the job and second DC at the bottom should workaround the issue?

Marco

[Fiskepudding]

I dont posess the powers of Gostev, but I guess that would work fine. As long as you dont run to many any in parallel and the list is so short that they could actually run at the same time regardless what order you put them in.

Support have not yet confirmedd this to be a general issue, so you might not even be affected.But it should not hurt setting it up like you suggest, then you are on the safe side

[Gostev]

Hi Marco, R&D was not able to confirm the issue internally yet, so just as Espen said, I cannot make any official recommendation on how to work around possible non-existent issue judging solely on the observations posted in this topic, yes - setting up your job this way should help you to avoid the issue.

I will update as soon as I have anything from R&D. Trust me I treat this potential issue very seriously, and have specifically asked everyone involved in researching this to copy me on any progress updates.

[Fiskepudding]

Update.

Just finished a WebEx with support, they recreated the issue just creating a shadow copy at the same time on both DC's. Meaning it should not be related to Veeam or Veeam 7. However, that does not yet explain why I could run the jobs at the same time with Veeam 6.5 and also Windows server backup (since that also uses VSS).

Might be that Veeam 7 uses VSS in a different way, options, parameters..?... Just a personal thought.

Anyways, the support person told me they were to escalate the case.

[Gostev]

Looks like what is required here is backing up all domain controller VMs at once... and parallel processing makes it so much easier to achieve.

[Fiskepudding]

That might be the case Anton.

Support did another WebEx and extracted some REG settings, so things are defiantly moving forward!

[Fiskepudding]

new update

Just got of a new WebEx with support.
They tried a fix, (changed a few .dll's and modified some registry settings on the DC's BackupRestore hive)
Unfortunately it did not help. They are to investigate further, and get back to us.

[jadams159]

So what's the suggestion here before a patch is released? Leave DC's on v6.5 or disable parallel processing for a complete v7 environment?

[foggy]

just put DC VM backups on different schedule, so that they are not backed up at the same time.

[Gostev]

*ALL* existing domain controllers must be backed up at the same time to trigger the issue.

[itfnb]

In the Patch 1 release notes, it had this issues listed as resolved, yet is seems that I'm still experiencing it. I was really excited about having all of my Windows VMs under one job with the SAN hardware snapshots and parallel processing... I'd really like to avoid having a job with just one DC in it if I can (It's not the end of the world if I can't.) Any ideas? Open a support ticket? I know... I know. My two DCs are on the same LUN (but different vSphere hosts).

Support Case: 00455534

[Gostev]

I heard about one case where the fix did not help (even before the patch was released), but that was isolated and I am not sure what the research ended up with. Thanks for opening a support case.

[itfnb]

All is well. I got the SYSVOL replication fixed and I've excluded my PDC from the main job and created a supplemental job to run after the other one finishes. And it was SUPER easy to add the new Backup Job to my Backup Copy Jobs. #slick. I'll report back after I hear from support.

[Fiskepudding]

I heard about one case where the fix did not help (even before the patch was released), but that was isolated and I am not sure what the research ended up with. Thanks for opening a support case.

That cold have been me . Support told me they had a hotfix and replaced some DLL's.
However that did not help on my system.

I was planning on changing my 2008R2 servers with 2012, since i had migrated from FSR to DFSR recently, and I suspect that plays a part.
Have not found a suitable time yet. Will try and run the jobs at the same time after this done.

But first I will try the patch

EDIT: The patch did not help me

[itfnb]

Support is still digging through my logs.