Comprehensive data protection for all workloads
allankjaer
Service Provider
Posts: 12
Liked: never
Joined: Apr 09, 2014 9:11 am
Full Name: Allan Kjaer
Contact:

ReFS 3.0 and Dedup

Post by allankjaer » Jan 26, 2019 11:38 am

In the what's new document for Veeam B&R 9.5 Update 4 is says:

Added experimental support for block cloning on deduplicated files for Windows Server 2019 ReFS. To enable this functionality,
create ReFSDedupeBlockClone (DWORD) value under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
registry key on the backup server.

But what value should the reg (DWORD) have?

Gostev
SVP, Product Management
Posts: 24299
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS 3.0 and Dedup

Post by Gostev » Jan 27, 2019 12:56 am

1

kghague-ccx
Lurker
Posts: 1
Liked: never
Joined: Jan 31, 2019 5:44 pm
Full Name: Karl Hague
Contact:

Re: ReFS 3.0 and Dedup

Post by kghague-ccx » Jan 31, 2019 5:47 pm

And this is set on each Repo server which has a deduped ReFS volume or only the primary backup server?

Gostev
SVP, Product Management
Posts: 24299
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS 3.0 and Dedup

Post by Gostev » Jan 31, 2019 9:25 pm

This is only on the backup server.

jrick
Service Provider
Posts: 22
Liked: 2 times
Joined: Jan 31, 2018 9:31 am
Full Name: Julien Rick
Location: Luxembourg
Contact:

Re: ReFS 3.0 and Dedup

Post by jrick » Feb 06, 2019 8:47 am

I'm currently testing ReFS with Dedup on server 2019, I have added the ReFSDedupeBlockClone DWORD value to my backup server (Win server 2016).
My test job is configured to create a daily synth full but don't use the block clone feature.

An idea why?

doktornotor
Enthusiast
Posts: 35
Liked: 10 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by doktornotor » Feb 06, 2019 3:39 pm 1 person likes this post

Probably because it requires Server 2019? :idea:

HannesK
Veeam Software
Posts: 3425
Liked: 409 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: ReFS 3.0 and Dedup

Post by HannesK » Feb 06, 2019 3:56 pm

correct, that key is only working on Windows 2019

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Feb 06, 2019 7:07 pm 1 person likes this post

On WS2019 I saw it already working on one very short chain. Wasn't particularly fast but there's a lot of IO load going on.

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Feb 10, 2019 1:17 pm

I hit my first compact and it seems that fast clone is not fast on deduplicated ReFS. It's seems to just copy (not clone - free space on volume is reducing about as fast) blocks to new file and target file is not deduplicated (size on disk = size).
Currently I'd say that it's no faster (if not slower) than NTFS deduplication-based compact.

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Feb 11, 2019 8:33 am

I hit a bigger compact and it's just as slow as with NTFS deduplication, about 6 hours to compact ~2,5TB of VBKs.
Is this expected? ReFS volume was created on WS2019, chain was restarted using active full on WS2019 on ReFS (eg all data has always been on this ReFS volume) and job log shows that "[fast clone]" is being utilized.

HannesK
Veeam Software
Posts: 3425
Liked: 409 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: ReFS 3.0 and Dedup

Post by HannesK » Feb 11, 2019 8:36 am

Hello,
if the job log says that "fast clone" is used, then it seems unexpected to me. Could you please open a support case and share the case number here?

Thanks,
Hannes

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Feb 11, 2019 9:55 am

#03408303

j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: ReFS 3.0 and Dedup

Post by j.forsythe » Feb 12, 2019 2:22 pm

Hi.
What is the file size limit for ReFS + DeDup at Server 2019?
Thank you!
:)

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Feb 12, 2019 2:31 pm

Only first 4TB of a file is processed by deduplication but there is no hard file size limit. 64TB volume limit places some practical limits though.

YoMarK
Enthusiast
Posts: 48
Liked: 5 times
Joined: Jul 13, 2009 12:50 pm
Full Name: Mark
Location: The Netherlands
Contact:

Re: ReFS 3.0 and Dedup

Post by YoMarK » Feb 15, 2019 2:16 pm

What is to be expected with ReFS 3.0(with dedup) on Windows 2019 compared to ReFS 2016/2019 without dedup in a case where you use per-VM backups and everything fast-clone/Syntectic full's?
As I expect offline duplication now also is able to dedup the whole volume instead of only the backup chain(fast clone), but does someone know how effective it really is in this scenario? Veeam: some lab test you are willing to share? :)

Gostev
SVP, Product Management
Posts: 24299
Liked: 3331 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS 3.0 and Dedup

Post by Gostev » Feb 16, 2019 2:20 am 1 person likes this post

Hi Mark, this was the last minute development and we did not have any chance to perform effectiveness testing (only reliability testing), which is why the feature is declared as experimental. Also, the idea was to hear the effectiveness results from the community, as lab environment don't have real-world data anyway. Thanks!

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Feb 19, 2019 7:46 am 1 person likes this post

Case resolved by support.
Slowness has be reproduced in lab and it's as slow as with NTFS deduplication.

absentminded
Lurker
Posts: 2
Liked: never
Joined: Mar 13, 2019 12:11 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by absentminded » Mar 13, 2019 12:14 pm

Our fast clone synthetic takes hours. How was your case resolved?

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » Mar 13, 2019 1:38 pm

It's wasn't resolved per se, just replicated in lab and confirmed to work... badly.
Summary is that all block clone benefits are lost if file is deduplicated. ReFS deduplication has no real integration with block clone and metadata operations become full read and write operations (copied from dedup store to new blocks). Also, written data is no longer deduplicated.
TL;DR: fast clone is slow with ReFS Deduplication.

jorgedlcruz
Veeam Software
Posts: 253
Liked: 109 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: ReFS 3.0 and Dedup

Post by jorgedlcruz » Mar 13, 2019 1:50 pm 1 person likes this post

Hi guys,
But that will be expected I will say isn't? To obtain the best possible performance, at a logical level really, you will need to use the usual ReFS for your synthetic, might be weekly or so, and then once you know your synthetic file is created, that is the file you want to apply dedupe.

So, look at your Windows Deduplication schedule and make sure you are not doing dedupe on the files of the open chain which will be used to create the synthetic, as if you do the server will need to dedupe all that files before do the ReFS, which will end in that long time to do the fast-clone.

I see this ReFS 3.0 and Dedupe kind of what Exagrid does, you have a landing zone for your weekly, where you run your synthetic operations, and then after you apply your dedupication to the chains which are closed already. Of course all of this is DIY vs. ExaGrid which gives you all of this and more out of the box.
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

absentminded
Lurker
Posts: 2
Liked: never
Joined: Mar 13, 2019 12:11 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by absentminded » Mar 13, 2019 2:23 pm

I confirm we do dedup files older than one day. Have to try eight days.

frankive
Service Provider
Posts: 884
Liked: 103 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: ReFS 3.0 and Dedup

Post by frankive » May 13, 2019 11:02 am

soo... I should stick with server 2019 and Refs and not enabling deduplication? :)

m.novelli
Veeam ProPartner
Posts: 353
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: ReFS 3.0 and Dedup

Post by m.novelli » May 13, 2019 2:47 pm

Hi guys, slight off topic... does Veeam support restore of deduplicated files stored on a source VM Windows Server 2019 , ReFS 3.0 volume?

Do I need to have Veeam installed also on box with Windows Server 2019 with "Data Deduplication" feature installed?

Thanks,

Marco

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » May 13, 2019 2:55 pm

Mount server must have Windows Server 2019 and data deduplication feature installed.

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS 3.0 and Dedup

Post by DonZoomik » May 13, 2019 2:58 pm

frankive wrote:
May 13, 2019 11:02 am
soo... I should stick with server 2019 and Refs and not enabling deduplication? :)
While I was running it, merge seemed a bit more stable than with NTFS deduplication - no other benefits. With NTFS, I got occasional transient errors during merge (probably due to file locking from deduplication engine).

csydas
Expert
Posts: 193
Liked: 46 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: ReFS 3.0 and Dedup

Post by csydas » May 23, 2019 6:05 pm

Hi Veeam,

Does this key work if the Windows 2019 ReFS Volume is presented as CIFS? Or does Veeam need to have the volume added as a Windows Server?

HannesK
Veeam Software
Posts: 3425
Liked: 409 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: ReFS 3.0 and Dedup

Post by HannesK » May 24, 2019 6:21 am

Hello,
yes it does fastclone via SMB / CIFS on a REFS dedupe filesystem. I just tested it in my lab.

Anyway, I'm not sure whether it makes any sense. SMB is the slowest way for backups and the comments say that ReFS dedupe & fastclone is also slow :-)

Best regards,
Hannes

sege
Novice
Posts: 8
Liked: never
Joined: Mar 12, 2012 4:32 pm
Full Name: Johan Segernäs
Contact:

Re: ReFS 3.0 and Dedup

Post by sege » May 29, 2019 8:00 am

As I understand the reg-key is to be set at the backup server, not the backup repo?
My case is that I have a Win2016 backup server and the repo server is a 2019 with ReFS.

Will that work?

HannesK
Veeam Software
Posts: 3425
Liked: 409 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: ReFS 3.0 and Dedup

Post by HannesK » May 29, 2019 8:12 am

that's what the document says. So if it does not work, please open a support case and post the case number here for reference.

matt.burnette
Service Provider
Posts: 7
Liked: never
Joined: Aug 11, 2017 9:09 pm
Full Name: Matt Burnette
Contact:

Re: ReFS 3.0 and Dedup

Post by matt.burnette » Jun 25, 2019 7:03 pm

Has anyone had any issues so far?
I thought about enabling it, though it looks like the only place that talks about block cloning with data dedup references Veeam.
I also haven't found where this is supported by Microsoft.
I also am unsure of the exact mechanisms that might be affected by having both on at the same time.

Does anyone have more information on how these can or can't be enabled at the same time?

I feel it could be a bad idea to have both at the same time since both are chunking data and replacing it with metadata/pointers.
How is it possible that these do not conflict with one another?
Is there any Veeam guidance on how they are testing it or what the best practices might be to enable this?

Post Reply

Who is online

Users browsing this forum: No registered users and 31 guests