DFSR & VBR & VSS Crashing All Over

[y0da]

I am having a strange issue that I am working around, but it is causing the fileserver migration to take longer than it should.

We are migrating from an old 2008 r2 fileserver to a 2016 fileserver.

To do the migration, I am taking shares that currently arent in DFS and putting them into dfs, letting dfsr do the replication, then breaking down the dfs and pointing users to new share.

The issue thats happening is as follows
1. On a normal day I never have any issues with backing up both old a new servers during this migration
2. I go ahead and create a share on the new server and start dfsr migration. The issue tends to occur when its a big share 300GB+. Small shares dont have the issue, because they finish before the next backup interval
3. All runs fine until its time for the every 4 hour veeam backup (no vmware tools quiecnese - just native application aware vss). The backup starts to run, and fails with a vss timeout. Then I check dfs and it is no longer replicating. I check the vss writers, but can't get the command to run. Seems to lock up.
4. I then have to reboot the server, which then starts dfsr again and it moves along hopefully finishing before the next interval.
5. Once dfsr finishes that first sync, it doesnt crash again during backups

Any ideas?

Support Case ID 03249002

[Mike Resseler]

Hi y0da,
First: welcome to the forums
Second: From my memory. Windows server 2008R2 DFS replication (to whatever version) uses VSS to do the replication. At that point in time, it wasn't the best implementation and when two systems connect to VSS (which you are doing with Veeam application aware processing and DFS) the writer goes "nuts" and in a failed state. Only a reboot does solve this (it used to be worse, you needed to re-register all writers their DLLs and stuff... was a serious pain).

Now, since this is a use-case which doesn't happen very often. Is the failure of the backup on the 2008R2? (I assume so). What will probably need to be done (temporarily) is to lower the backup interval. If you have the opportunity to start the replication in the morning, wait with the backups and monitor until the initial replication is done, then you probably can perform the migration as such.

But I would also continue to work with support because there might be somewhere a known hotfix or something like that to solve it

[mengl]

Hello y0da,

also my first post here

I had a similar issue in the past.
In my situation we had a replica server in a remote location which should do an initial DFS replication of ~500GB over a 4mbit WAN line.
As far as I had observed when Veeam takes a VSS snapshot it suspends DFS replication and continue after backup is completet. The pause should normally begin after completing the current file transfer finnished. Due to the slow line that transferdidn't complete during the VSS timeout. This ended with an VSS error for Veeam and a stuck DFS service. The DFS service had to be restartet before began replication again.

My workaround was to disable Guest processing for that VM until the initial replication finished.
After that had completet I ajusted the DFS replication times to avoid replication durring the beginning of backup.
Since that no more problems (with Guest processing).

[skrause]

Have you tried seeding the files with robocopy before setting up the DFS replication?

https://docs.microsoft.com/en-us/window ... h-robocopy

Also, you are probably also running into the issue where your staging folder is still set to the default size which for large volumes (or ones with a lot of large files) causes DFSR to be really slow.

[y0da]

Hello,

Just to close this out. I basically did what Mike said and stopped backups during the day and let replication go a little. Worked fine and kept the issue at bay.

With respect to skrause's point, I increased my staging folders quota size to 20GB and didnt have an issue there. Also the link between this endpoints is 1gbps.

All in all, it worked out by timing the initial seed to run outside of the backup window. Lost some retention points, but what can I do.

Thanks all.

[Mike Resseler]

Thanks for letting us know how you solved it. And also good to know that this causes issues (again). Did you started backups during the day again? I hope that now the incremental replication won't fail during those backups.

[Trelor]

Hello,

Just to close this out. I basically did what Mike said and stopped backups during the day and let replication go a little. Worked fine and kept the issue at bay.

With respect to skrause's point, I increased my staging folders quota size to 20GB and didnt have an issue there. Also the link between this endpoints is 1gbps.

All in all, it worked out by timing the initial seed to run outside of the backup window. Lost some retention points, but what can I do.

Thanks all.

FWIW you should check the staging folder recommendations based off the larges 10 files; below is a modified version of MS's ps script

Code: Select all

Function Select-FolderDialog
{
    param([string]$Description="Select Folder",[string]$RootFolder="MyComputer")

	[System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms") |
     Out-Null     

	$objForm = New-Object System.Windows.Forms.FolderBrowserDialog
	$objForm.Rootfolder = $RootFolder
	$objForm.Description = $Description
	$Show = $objForm.ShowDialog()
	If ($Show -eq "OK") {
		Return $objForm.SelectedPath
	} Else {
		Write-Error "Operation canceled by user."
	}
}
$folder = Select-FolderDialog
$big32 = Get-ChildItem "$folder" -recurse | Sort-Object length -descending | select-object -first 32 | measure-object -property length -sum
write-host "For the folder: $folder, the DFS staging area should be: $([Math]::Round($($big32.sum /1gb), 2))GB."

[Mike Resseler]

Trelor,

FWIW... It will be worth a lot for people having the same issue so thanks for the script. The community will love it!