Best Practice for MS Server 2012 DeDup Repo

Availability for the Always-On Enterprise

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Wed Nov 14, 2012 12:41 am

Some of them have been, some havent.

Size: 8.33, Size on Disk: 7.6
Size: 6.6, Size on Disk: 4.2

Not too bad...

I've re-done my scheduled tasks and I'm going to try to get a full pass on both. I found that I could increase memory usage up to 80% from the scheduled tasks, and I've disabled the time restriction... so we'll see what happens I guess.
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby tsightler » Wed Nov 14, 2012 1:15 am 3 people like this post

It's VERY critical to understand that Windows 2012 dedupe is not really designed for high data ingest rates. In general, the Microsoft recommendation is 100GB/hr, so that means, assuming you use the default 8 hour dedupe process, you can only process about 800GB a day. You can tweak the default job schedule to run more, but even at 24 hours that's only 2.4TB/day. How big are your full backups?

Windows dedupe can scale past 100GB/hr by using multiple datastores and running dedupe jobs concurrently (each dedupe job will only use a single core) but of course that would be a separate dedupe pool.

In other words, comments that it "works great" don't really take into account the impacts of scaling beyond smallish repositories (say 10TB or less), and with your repositories (30TB and 60TB) I'd have to assume that you are ingesting a significant amount of data. I doubt that you have completed a full pass at this point based on your savings rate.

I'm working on a whitepaper with some guidelines but it probably won't be ready for a few more weeks as it takes time to test various setups, but for data of your size, it would likely involve splitting up the volumes into smaller chunks (perhaps 16TB each) and running a dedupe job on each. Note that this might not save significant space compared to Veeam compression with reverse incremental since you have to always have enough free space to store at least one pass of uncompressed full backups. The primary use case is for long term archiva; (months) in which case the Windows dedupe can be a huge win.
tsightler
Veeam Software
 
Posts: 4872
Liked: 1819 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Wed Nov 14, 2012 3:02 am

I figured out how to make it run at a higher priority, and I'm now seeing _some_ progress...

Running with 80% memory (39GB) and high priority has started to yield some reduction in data... I'll see how it goes.
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Windows 2012 Dedup Issues...

Veeam Logoby mongie » Wed Nov 14, 2012 4:42 am

Looking in Resource Monitor shows that fsdmhost.exe is reading from a VBK at ~ 14MB/s on both servers.

It appears to be writing at around 30MB/s at the same time.

I guess that gives a processing rate of ~ 50G/hr. Not BRILLIANT, but not horrible either.
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Gostev » Wed Nov 14, 2012 2:03 pm

mongie wrote:I have two repository servers at the moment. One ~30TB and one ~60TB.
I find that both background processing jobs, and scheduled jobs take FOREVER.

Quote from Veeam forums digest 3 weeks ago:

There seem to be a lot of anxiety among our customers and partners regarding using Windows Server 2012 deduplication in conjuction with Veeam B&R. Many of you are already testing it, and sharing some pretty impressive results. Now, while I do think the technology is great and all, and have been promoting it myself heavily - I want to make sure you do not start implementing it for production use without fully understanding its scalability aspects. For example, from what I gather, you cannot just rip and replace single-pool deduplicating storage appliance with WS2012 deduplicating file server. Because, while Microsoft deduplication is designed to scale towards a datacenter environment, you can only scale it in a very specific way. Here is a great resource to start with > Plan to Deploy Data Deduplication.
Gostev
Veeam Software
 
Posts: 21603
Liked: 2405 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby TheJourney » Thu Nov 15, 2012 11:11 pm

Testing win 2012 dedup. This is many different Veeam jobs.
Code: Select all
Volume                             : H:
VolumeId                           : \\?\Volume{f7ceba25-cc4d-48be-adfe-38855f2681c1}\
Capacity                           : 8.95 TB
FreeSpace                          : 4.81 TB
UsedSpace                          : 4.14 TB
UnoptimizedSize                    : 22.43 TB
SavedSpace                         : 18.29 TB
SavingsRate                        : 81 %
OptimizedFilesCount                : 275
OptimizedFilesSize                 : 22.44 TB
OptimizedFilesSavingsRate          : 81 %
InPolicyFilesCount                 : 275
InPolicyFilesSize                  : 22.44 TB
LastOptimizationTime               : 11/15/2012 3:45:34 PM
LastOptimizationResult             : 0x00000000
LastOptimizationResultMessage      : The operation completed successfully.
LastGarbageCollectionTime          : 11/10/2012 4:56:39 PM
LastGarbageCollectionResult        : 0x00000000
LastGarbageCollectionResultMessage : The operation completed successfully.
LastScrubbingTime                  : 11/10/2012 5:10:26 PM
LastScrubbingResult                : 0x00000000
LastScrubbingResultMessage         : The operation completed successfully.
TheJourney
Influencer
 
Posts: 13
Liked: 2 times
Joined: Thu Dec 10, 2009 8:44 pm
Full Name: Sam Journagan

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby ryan1212 » Tue Feb 26, 2013 5:52 pm

Has anyone measured the difference between "Dedup friendly" and "no compression" in regards to de-duplication with server 2012?
ryan1212
Influencer
 
Posts: 16
Liked: 5 times
Joined: Tue Feb 26, 2013 3:48 pm
Full Name: Ryan

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby tsightler » Tue Feb 26, 2013 7:04 pm

I would expect it to be very similar to other dedupe appliances. Typically "dedupe friendly" compression provides only a 10-20% reduction in the initial size of the VBK and VIB files, while costing roughly that same amount dedupe savings, perhaps slightly more. Saving 10-20% may not sound like much, however, for customers backing up 10's or 100's of TB, this can be a significant savings in network bandwidth and it also generally makes for faster restores, and sometimes even slightly faster instant recovery since 10-20% less data must be read from the backup repository, so it can be a reasonable compromise. Effectively you are trading some hard disk space overall (because of less dedupe) for some up front network and disk bandwidth savings. If you're happy with your current performance and want to maximize dedupe, I would leave compression disabled.
tsightler
Veeam Software
 
Posts: 4872
Liked: 1819 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby baatch » Fri May 24, 2013 2:34 pm

Have read through most if this thread but can't seem wrap my head around the Server 2012 dedup function.

It says 100GB per Hour and that equals to 2,4 TB per day.

So if my full backup is 3,5 TB, should I not use the Server 2012 dedup function then?
baatch
Enthusiast
 
Posts: 30
Liked: 4 times
Joined: Thu May 16, 2013 12:52 am
Full Name: Alexander

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Vitaliy S. » Fri May 24, 2013 3:54 pm

Why not? Since you're using forward incremental backup mode then subsequent incremental job runs will not touch this file, so it shouldn't be an issue to offload 3.5 TB backup file to a dedupe volume. It will just take a bit more time.
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby baatch » Fri May 24, 2013 4:00 pm

Ok so the dedup can resume on that big file even after it has stopped processing it because I set the dedup duration for only 24 hours?
baatch
Enthusiast
 
Posts: 30
Liked: 4 times
Joined: Thu May 16, 2013 12:52 am
Full Name: Alexander

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Vitaliy S. » Fri May 24, 2013 4:09 pm

Hmm...good question. I believe it needs to be verified, but why not specify more than 24 hours?
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby baatch » Fri May 24, 2013 4:15 pm

Going to try have more that 24 hours. Wonder if I can set it to 0 ? Will it always let it run until it completes then?
baatch
Enthusiast
 
Posts: 30
Liked: 4 times
Joined: Thu May 16, 2013 12:52 am
Full Name: Alexander

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Vitaliy S. » Fri May 24, 2013 4:22 pm

Wait a minute...are you referring to the file age parameter that can be set to 0? Just want to make sure we are on the same page right now.

b. File Age: Deduplication has a setting called MinimumFileAgeDays that controls how old a file should be before processing the file. The default setting is 5 days. This setting is configurable by the user and can be set to “0” to process files regardless of how old they are.

Here is the link for more info: http://blogs.technet.com/b/filecab/arch ... -2012.aspx
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby baatch » Fri May 24, 2013 5:15 pm

I was referring to the optimization throughput duration.
baatch
Enthusiast
 
Posts: 30
Liked: 4 times
Joined: Thu May 16, 2013 12:52 am
Full Name: Alexander

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Google Feedfetcher, sg1 and 5 guests