SYSVOL stops replicating after Veeam 7 backup.

[stevemcmillan]

Hi,

I have had this issue also on a greenfields server 2012 domain with two domain controllers.
The environment only had 2 domain controllers and is not yet in production, so to work around I had just demoted one of the DC's. My intention is to re promote soon.
However even with only 1 DC, I am still getting DFSR warning events in the event log when the backup does its application aware part on the DC.
These are event ID 2109, and 4110 source DFSR. (I have tested this in a separate lab, and can replicate the issue there too)
Although the DFSR database seems to repair on it's own, it does make DCDIAG ugly, as it reports DFSR errors "in the last 24 hours".

I have logged a case for this (Case # 00460943), so far nothing of use.

[Fiskepudding]

I have now swaped my 200R2 DCs to 2012 and the issue is still there.

Running backups in sequence is fine, running them in parallel still stops sysvol replication.

So still using a workaround with running them in sequence.

Steve:
I guess the warnings in the DFSR logs are expected?... replication has to stop at some point druing the backup process i guess.

When i run a single the the backup of just DC1
I get these warnings: 2109,4110 AND 4102 on DC1
About 10 seconds BEFORE the above warnings i get 5014 warning on DC2 (not running backup of this DC)

When running a single backup of just DC2
I get these warnings: 2109,4110 AND 4102 on DC2
About 10 seconds BEFORE the above warnings i get 5014 warning on DC1 (not running backup of this DC)



Sysvol works fine after this, and NO ERROS in DFSR logs, just these WARNINGS mentioned above.


Running the jobs in paralell is another storry, then i GET ERRORs 4612,5002 in additoin to the warnings.


Support would like to look furtner into this issues if the domain server change did not help..
Wil post back if they find anything worth sharing.

[jadams159]

So Patch 1 did or did not resolve this issue?

[foggy]

It was supposed to fix the issue, however there seem to be cases where the fix did not help, so if you still experience the issue after patch installation, I recommend opening a case for investigation. Thanks.

[stevemcmillan]

So Patch 1 did or did not resolve this issue?

Hi, thanks for the question, unfortunately in patch1 has not resolved in my instance. I have a case open as mentioned, then last update I received a few days ago was that the engineers are working on it.

[mlehto]

I have had trouble with this too. I have actually had to repair my replication using ADSIEDIT each time. I will apply patch 1 and let everyone know what I experience.

Thanks

[Fiskepudding]

I have had trouble with this too. I have actually had to repair my replication using ADSIEDIT each time. I will apply patch 1 and let everyone know what I experience.

Thanks

With all the testing I have done on my own accord and with Veeam support, I guess I have used ADSIEDIT about 50 times to restore replication, and that’s no exaggeration.

[Gostev]

Very sorry for this... I keep bugging R&D for updates.

[Fiskepudding]

Don’t be sorry for us Anton . We in this thread know how to work around this.(Don’t run DC backups in parallel if you are affected).

It's worse for the systems that are affected and don’t know it yet, or don’t know why their sysvol is not replicating anymore. The Backup job might not be the first suspect...

I have told Support that they can look at my system any time, if they still cannot reproduce the issue in the lab.
In the end it very possible that it is not a veeam issue at all. So I am happy that Support are still looking into this.

[Fiskepudding]

Support, tested some shadow copy yesterday on my DC's, and did some further tests today. So they are still hard at work..
No solution yet though.

[lobo519]

Whats the status on this issue? I don't see anything listed in the R2 release notes..

[Gostev]

No significant updates. The issue only affects a handful of customers, and we are still trying to figure out what is common between them, and why this does not affect anyone else.

[Fiskepudding]

Like Anton says, they are still investigating it. After my last post support have been doing several new tests and collecting more logs. Our issue was not resolved in R2 (was not expected or promised fixed either)
As I understand, it is still not clear whether it is a Veeam issue or general domain issue.

[GL@MO]

I am experiencing the same issue but I am not backing up both of my DCs at the same time or with the same job. After a restart of both DCs they seem to replicate again.
The issue appeared when I was setting up some GPOs, doing a gpupdate /force on some clients and servers which failed because a "gpt.ini" could not be found (DFSR stopped).

[Gostev]

I am experiencing the same issue but I am not backing up both of my DCs at the same time

Then it is not the same issue. Please open a support case, and feel free to create a separate topic, not to derail this one. Thanks!

[euro]

Hello guys,
come across this post searching to fix our DC Synchronisation problems ..
but i will open a new ticket (could not find one from GL@MO), because our DC backups are not running at same time

[GL@MO]

Hi euro,

Posted a ticket yesterday
Case # 00487012

already got a first reply and was asked if I have an active "Application Aware Image Processing" in the backups.

[MikeFranz]

I have a customer with case 00492801 who is also having the same issue; I'm adding the case number here for reference.

[euro]

Hello,

my problems are gone since i run an authorative Replication of my DCs.
Seems that we initially had both DCs in one BackupJob and they were not replicating any more since then.
Running each day a backup from different DC, no more problems anymore

thanks for this thread!

[Fiskepudding]

Yeah Stefan, doing an authorative restore of the DC's and then setting the jobs to run in sequence will be a solution (workaround).
Glad the thread could be of some use for you

But it should not happen in the first place, so I hope Veeam can figure out why it is sctually happening.
It has been a long time since I heard from them regarding this case, hope they have not “dropped the ball” just yet.
Merry Christmas !

[dlieshout@client.nl]

Half good news. Today Veeam Support put 4 beta-Veeam-DLL's at one of our customers' site. It seems the SYSVOL replication (DFSR) now keeps working after a backup.
Now Veeam will discuss some further if this will be the definite solution. I guess this will be put in the next upcoming patch-release.

In the meantime, just uncheck parallel processing in the global options and fix the DFSR (http://support.microsoft.com/kb/2218556, authoritive sync like "D4 ...").

[Fiskepudding]

Sweet. Thanx for the update. Hope they nailed it

[Fiskepudding]

I got a mail from support today too, and installed the new DLL's. I can confirm that they fixed our problem too.
Thanks Veeam!

[dlieshout@client.nl]

Good news. I also stated in another item. Veeam placed 4 beta-Veeam-DLL's on one of our customers' site. An now the SYSVOL-replication is fine after the backups.

Until you have the new DLL (probably in a next patch-set), you can uncheck the Parallel Processing in de global options.

[Gostev]

Please note that this fix is not final, but rather private/pilot version to confirm that we found exactly what is causing this. Essentially, the current "fix" disables some parts domain controller processing during backup. The excluded steps is needed to addresses possible DFS-R SYSVOL replication issues in certain scenarios with full VM restore (as reported by some v6 customers).

I actually discussed the final fix with the devs just a couple of hours ago, and proposed some ideas - let's see what they will come up with.

Naturally, the best way to avoid this issue for now is simply not to backup all your domain controller VMs at the same time. You can do that by placing one of the domain controllers in the separate job, and chain it with your main job - or start it before any other jobs, when no backups are running for sure.

[Fiskepudding]

That was important info Anton. I did not recieve such info in the mail.
Just a procedure how to swap the DLLs.
I tested and replyed my results. (And kept the new DLLs)

What you are actually saying is that we should put back the old DLLs and kepp running the DC backups in sequence until further notice.

[Gostev]

The removed steps will not cause severe issues for sure (as the product worked this way all the way up until v7), but may complicate the recovery in a sense that you may have to fix SYSVOL replication after domain controller recovery in certain cases. I don't have all the details about this other issue handy unfortunately, but apparently we've had enough support cases to justify addressing it in v7.

[trafsta]

Support contacted me today and provided me with the updated DLL's. I will be testing it hopefully on Friday night (that way I can fix SYSVOL replication over the weekend in the event that it breaks).

I will try to report back here next week with the results.

[heiicat]

Same Issue - Case #00498919

"Solution" was to put one of the two DCs in a seperate backup job... not really a great solution but so far has been working. We use the HP Storage Plugin to do SAN snapshots which if i recall, when implemented was when these issues started. Seemed to be because it created VMware snapshots of the domain controllers at the same time and thus broke the replication.

[waynewall]

Thanks guys for all the info! I was racking my brain to figure out what was going on until now. Does anyone have steps to fix this issue in server 2012? All the sites I checked on were server 2008.