snapshot of DFSR fileserver not supported by MS!?

[mark.harre]

Hi,

We plan to backup and replicate our DFSR fileserver using veeam, however I just read that Microsoft does not support snapshots of any multi-master database (including DFSR) and restoring from a snapshot may cause DFSR replication to fail. See http://support.microsoft.com/kb/2517913
My question is, does anyone here use veeam to backup or replicate a DFSR server? And if yes, how do you avoid the restore problems reported in the KB article.

thx for your help
regards,
Mark

[Gostev]

Hi Mark, we integrate with VSS to properly quiesce all Microsoft applications before taking a snapshot, so the above KB article does not apply to Veeam B&R. Thanks.

[mark.harre]

Thx for the reply Gostev,
What makes me unsure is that KB article says that you may have replication problems AFTER you restore from a snapshot because DFSR does not support USN rollback (still investigating what this actually means). Sure the data can be backed up correctly using VSS but that doesn't mean that you can simply restore it from a different point in time if it is not supported by the application. I guess i need to do some testing before i implement this in production.
regards,
Mark

[Gostev]

Hi Mark, the thing is that proper VSS backup prepares the application for correct restore as well. For example, USN rollback is also an issue when restoring Active Directory Domain Controller from regular, "non-enlightened" snapshot. However, restoring Veeam backup will not have this issue, thanks to leveraging our VSS integration before taking the VM snapshot.

So, unless DFS does not take care of USN rollback in its VSS writer (which is unlikely), then there will be no problems. Plus, if there were some problems, we would sure have heard about them in 4 years from other customers (DFS is pretty widely used anyway).

Having said that, testing before implementing anything at all is always a good idea!

Thanks.

[mark.harre]

For those interested.
I just tested backup and restore of a DFS/R server and it did indeed work perfectly.

procedure:
create 2 (Win 2008 R2) servers with with a DFS-R replicated share.
backup server1 with Veeam
change/add/delete some files on the replicated share (check all in sync)
delete and restore server1
check both server1 and server2 are in sync (incl what was changed after the backup).

thx for the help Gostev

[mark.harre]

Sorry I spoke too soon.

Although DFSR initially synchronised following the restore, it failed to replicate any subsequent files from server1 to server2. (Strangely it replicated files from server2 to server1 which fooled me into thinking it was working). I then found errors in the event log indicating that the DFSR database was corrupt and replication for the volume was stopped. These are the same DFSR event ID 4102 and 2004 errors described in Microsoft KB 2517913

To quote from the article:

Snapshots are not supported by the DFSR database or any other Windows multi-master databases. This lack of snapshot support includes all virtualization vendors and products. DFSR does not implement USN rollback quarantine protection like Active Directory Domain Services.

Under no circumstances should you create or restore snapshots of computers running DFSR in a production environment.

I am trying to find a workaround and I will open a case with Veeam, but at this point i can not use Veeam in a production environment to backup or replicate our file server.

regards,
Mark

[mark.harre]

I opened a case with Veeam and they confirmed that this is a Microsoft limitation. Snapshots (incl Veeam backup & recovery) of a VM using Microsoft DFS replication are NOT supported and may result in a corrupted DFSR database and replication failures after restoration. So unfortunately we can't use Veeam to backup our live file server.

[tsightler]

Doesn't this only limit your ability to perform a full VM restore? Certainly you can still backup and use the file level restore aspects of Veeam, you simply can't perform a full VM restore and expect that it will not break your DFSR replication except in the case of a full disaster.

[mark.harre]

sure the backup worked and i could recover single files or restore the full server. It was only replication that broke. For us this is critical but for others it may not be.

ps, it is possbile to delete are recreate the corrupted DFSR database. Just don't expect any support from Microsoft or Veeam.

[tsightler]

I guess my question is, what software actually provides the ability to perform the restore without breaking the replication? As far as I know, none since it's not supported by the VSS writer. For example, Symantec Backup Exec simply restores the files into the "PreExisting" folder which is not replicated to other DFSR nodes so the admin then has to move it, or, if the DFSR folder has been removed, newer versions of Backup Exec allow restores to be redirected to a regular filesystem, but not to another DFSR node. Once again, the admin has to move it.

[mark.harre]

Good question. I also tested Backup Exec backup and restore to/from tape. The functionality depends on whether you do a authorative or non-authoritive restore. I found if you try to restore an older file non-authoritatively (the default) then the existing (newer) file is not be overwriten and the file will instead be restored to the ConflictedAndDeleted folder (better to restore single files to an alternative location and manually copy to where you want). Whereas an authorative restore overwites the existing file and moves the existing file to the ConflictedAndDeleted folder. In both cases DSFR re-initializes replication (presumably checking all files are in sync). So Backup Exec does work but any conflicts are moved to a separate folder as you said. Even a full system (non-authoratative) restore worked with Backup Exec. The system was restored and the latest files were replicated from the partner.

[massimiliano.rizzi]

Hello,

we are building a new Veeam Backup & Replication 6.5 backup job from scratch.

This job will include a Windows Server 2012 Standard Edition Active Directory member server VM running Microsoft DFS Replication service.

Are there any known issues or caveats on backing up a live VM running Microsoft DFS Replication service using Veeam Application-Aware Image Processing ?

Since some Internet resources I have found seem to be (wrongly in my opinion) claiming that there might be problems with Microsoft DFS Replication service when performing backup operations, I would like to make sure I can treat this VM just like any other VM from a Veeam Backup & Replication perspective.

Any help/information and real-life field experiences will be greatly appreciated.

Thanks and Regards,

Massimiliano

[massimiliano.rizzi]

Hello,

thank you for merging the new thread I started yesterday and for appending it to the thread started by mark.harre.

Based on my understanding after carefully reading through the entire topic, the limitation from the Microsoft side described in the topic should only break DFSR replication functionality should the need arise to perform a full VM restore in the case of a full disaster, whereas the VM can be backed up correctly just like any other VM since Veeam Backup & Replication integration with VSS properly quiesces all Microsoft applications before taking a snapshot.

While searching for additional information about backing up a live VM running Microsoft DFS Replication service I have ran into a very recent (April 5th) and interesting blog post from the Microsoft Storage Team at http://blogs.technet.com/b/filecab/arch ... -dfsr.aspx.

To quote from the article:

The Big Cannoli – do not restore snapshots or whole machine backups
By now, you’re asking yourself “interesting, but… what does this have to do with virtualization?” Multi-master replication relies on this incrementing logical clock and – as any sci-fi buff knows – time travel carries a whole host of problems when going backwards and forwards.
So if you read nothing else in this blog post, read this:
• Virtual Machine Saved States/Snapshots. When virtualizing DFSR, start the virtual machine, run DFSR and if you need to stop the virtual machine, fully shut down the guest OS. Do not use saved states or snapshots.
• Backing Up Virtualized DFSR. When backing up virtualized DFSR, perform a guest side backup using a backup product that is VSS aware. Do not perform or restore from host side backups of virtualized DFSR servers.

Based on my understanding after carefully reading through the entire blog post from the Storage Team, Microsoft recommends that users perform a guest side backup using a backup product that is VSS aware (i.e. Veeam Backup & Replication), whereas special attention should be paid when performing a restoration.

Can somebody from Veeam confirm if my understanding is correct ?

Thank you again for your support.

Regards,

Massimiliano

[foggy]

Can somebody from Veeam confirm if my understanding is correct ?

Yes, looks consistent to me.

[massimiliano.rizzi]

Thanks !

[nunciate]

We have several very large Windows file servers in our environment. A couple over the 7Tb size. They are virtual servers with multiple 2Tb drives. I use Veeam to backup these servers today with no major issues.

I am planning to build out a DFS namespace on our domain and want to do a full migration of the data. I will likely not point DFS to the existing servers but rather build out new back end storage for DFS and then migrate the data into the new system. My reasoning is that I want to reduce the drive sizes on my backend storage servers to something more manageable from a backup/restore perspective.

Has anyone setup DFS while keeping Veeam in mind as the backup solution? Is anyone successfully backing up the backend storage servers for their DFS with Veeam today and are granular restores easy? What about with DFSR turned on? Is it easy to restore data that is being replicated to a secondary site. For instance Home shares that reside on a primary server in production and are DFS replicated to a standby DR site (not actively used).

Here is my basic plan so far.
Enable DFS on our domain.
Create virtual Windows 2012 Servers for backend file storage. The servers will have multiple drives. Virtual disks created for each server in VMware connected to our SAN not RAW LUNs. Each disk will have a max size to make it easier to restore entire disks if the need arises. I am thinking 512Gb or 1Tb max but there will be multiple drives so probably a max of 4-5Tb/server. I will add servers as needed to get to the capacity I need. I am thinking 4 to 5 servers max.
I will backup all the servers using Veeam every day.

Thoughts? Is there a better way to do this?

[fmt1962]

Not having been aware of this issue prior to now, we have been using Veeam B&R for ALL our servers for years... however this week I had a DR situation with a full crash of one of our servers.

Veeam was able to do a full restore of the server quickly and that was great... but IT DID totally shatter DFS replication..... we use DFS replication on a lot of our machines and now I am very worried about the viability of using Veeam for a large number of our server!

I am about to open a case with MS support to see if they can assist in recovery. There is an article http://support.microsoft.com/kb/2517913 that indicates they may be able to help, I hope so as we have a lot of DFS replicas on the file server concerned.

I am surprised this issue has not been highlighted more!

[Gav@GH]

Any updates on this limitation with recent version updates?

(original thread now a couple of years old and MS article posted 2011).

[Vitaliy S.]

To the best of my knowledge there were no updates to this behavior. After restoring the VM you should re-configure DFS replication.

[rogerdu]

I'm assuming that this Issue has not changed? Using VB&R for file level restore is OK...

How would one go about replicating these types of servers to a DR site then? Replication with Veeam still uses snapshots at the Hypervisor level (Hyper-V or VMware)...