-
- Influencer
- Posts: 13
- Liked: 1 time
- Joined: Jan 23, 2018 8:21 am
- Full Name: OL
- Contact:
Veeam and ReFS deduplication
Hi,
Veeam has a blogpost regarding using Windows builtin deduplication together with Veeam on a storage server: https://www.veeam.com/blog/data-dedupli ... veeam.html
I assume this is for NTFS deduplication. With Windows Server 2016 build 1709, deduplication was added to ReFS: https://docs.microsoft.com/en-us/window ... erver-1709
Is this a supported, and/or recommended, configuration with Veeam? Does the same configuration apply to ReFS, ref the Veeam blogpost?
regards
Olav Langeland
Veeam has a blogpost regarding using Windows builtin deduplication together with Veeam on a storage server: https://www.veeam.com/blog/data-dedupli ... veeam.html
I assume this is for NTFS deduplication. With Windows Server 2016 build 1709, deduplication was added to ReFS: https://docs.microsoft.com/en-us/window ... erver-1709
Is this a supported, and/or recommended, configuration with Veeam? Does the same configuration apply to ReFS, ref the Veeam blogpost?
regards
Olav Langeland
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: Veeam and ReFS deduplication
Would be interested to see how well this might work, in addition to block cloning.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Veeam and ReFS deduplication
Starting Update 3, Veeam B&R disables block cloning on ReFS volumes with built-in deduplication enabled (unless the repository is CIFS, where we cannot check). The two features cannot be used simultaneously.
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: Veeam and ReFS deduplication
so what to choose? Refs and block cloning or refs with data deduplication?
-
- Expert
- Posts: 117
- Liked: 31 times
- Joined: Oct 30, 2012 7:53 pm
- Full Name: Chris Jones
- Contact:
Re: Veeam and ReFS deduplication
The best answer is, it depends. And I know that sounds like a copout, but it really does.
If you perform Active Full Backups then stick with deduplication as that will catch duplicate blocks across all files on the volume. Remember, Veeam dedupe only occurs within a single backup file, such as data within a single VBK and not across multiple backup files for the same job. ReFS won't give much, if any, benefit when performing Active Fulls.
However, if you are performing Synthetic Fulls, then ReFS is a good candidate as it provides much faster creation of the synthetic full VBK file. Be sure to check the Veeam B&R console logs to confirm the job is using Fast Clone to perform the synthetic operation. This indicates that Veeam detected the backup repository is ReFS and is leveraging it for your synthetic full creation, which dramatically reduces the time to create the new VBK.
If you saw this week's Veeam Digest you will see that Microsoft have approved ReFS for use on SAN volumes (iSCSi/FC/FCoE) as long as the devices in the path, including the storage array are in the Windows Server Catalog (https://www.windowsservercatalog.com/). Previously Microsoft would only support ReFS when using on direct-attached storage. To quote Gostev in the email, "ReFS is in fact fully supported on ANY storage hardware that is listed on Microsoft HCL". You can read about ReFS here: https://docs.microsoft.com/en-us/window ... s-overview
I work for NetApp so I'm happy to see that all of our current systems are fully supported by Microsoft with Windows Server 2016 so we can use ReFS for Backup Repositories and then use ONTAP's dedupe to get the best of both worlds ... although if you were using ONTAP you would be best to leverage ONTAP snapshots and SnapMirror for complete protection whilst still retaining Veeam's features like application-aware processing and SureBackups. But it's great to have the options now.
If you perform Active Full Backups then stick with deduplication as that will catch duplicate blocks across all files on the volume. Remember, Veeam dedupe only occurs within a single backup file, such as data within a single VBK and not across multiple backup files for the same job. ReFS won't give much, if any, benefit when performing Active Fulls.
However, if you are performing Synthetic Fulls, then ReFS is a good candidate as it provides much faster creation of the synthetic full VBK file. Be sure to check the Veeam B&R console logs to confirm the job is using Fast Clone to perform the synthetic operation. This indicates that Veeam detected the backup repository is ReFS and is leveraging it for your synthetic full creation, which dramatically reduces the time to create the new VBK.
If you saw this week's Veeam Digest you will see that Microsoft have approved ReFS for use on SAN volumes (iSCSi/FC/FCoE) as long as the devices in the path, including the storage array are in the Windows Server Catalog (https://www.windowsservercatalog.com/). Previously Microsoft would only support ReFS when using on direct-attached storage. To quote Gostev in the email, "ReFS is in fact fully supported on ANY storage hardware that is listed on Microsoft HCL". You can read about ReFS here: https://docs.microsoft.com/en-us/window ... s-overview
I work for NetApp so I'm happy to see that all of our current systems are fully supported by Microsoft with Windows Server 2016 so we can use ReFS for Backup Repositories and then use ONTAP's dedupe to get the best of both worlds ... although if you were using ONTAP you would be best to leverage ONTAP snapshots and SnapMirror for complete protection whilst still retaining Veeam's features like application-aware processing and SureBackups. But it's great to have the options now.
-
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: Veeam and ReFS deduplication
I think I've posted this before, but I'm experimenting with this methodology to combine ReFS Fast Clone with Deduplication:
https://charbelnemnom.com/2016/10/how-t ... p-storage/
Essentially, the Veeam backup repository is a VM running on a Hyper-V host. Inside the VM you run ReFS with Fast Clone, and on the host you run NTFS with deduplication (or eventually ReFS with deduplication). I was holding off on this until the ReFS issues have been sorted out, but with the latest updates looking promising, I'm going to start working on moving my repositories into new 2016 VMs and see how it works.
https://charbelnemnom.com/2016/10/how-t ... p-storage/
Essentially, the Veeam backup repository is a VM running on a Hyper-V host. Inside the VM you run ReFS with Fast Clone, and on the host you run NTFS with deduplication (or eventually ReFS with deduplication). I was holding off on this until the ReFS issues have been sorted out, but with the latest updates looking promising, I'm going to start working on moving my repositories into new 2016 VMs and see how it works.
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: Veeam and ReFS deduplication
I'd be very interested in the IOPs this results in. I've Deduped CSV volumes in the past with much success. However there is a corresponding increase in read IOPsnmdange wrote:Essentially, the Veeam backup repository is a VM running on a Hyper-V host. Inside the VM you run ReFS with Fast Clone, and on the host you run NTFS with deduplication (or eventually ReFS with deduplication)
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: Veeam and ReFS deduplication
any suggestions from Veeam on this? refs with block cloning vs refs with deduplication?
-
- Veteran
- Posts: 942
- Liked: 53 times
- Joined: Nov 05, 2009 12:24 pm
- Location: Sydney, NSW
- Contact:
Re: Veeam and ReFS deduplication
As per the article below from Veeam:
https://helpcenter.veeam.com/docs/backu ... tml?ver=95
Windows Data Deduplication (2012 - 2016) is incompatible with Fast Clone feature.
Hence we can only choose one feature benefit, not both.
Is that still the case today ?
https://helpcenter.veeam.com/docs/backu ... tml?ver=95
Windows Data Deduplication (2012 - 2016) is incompatible with Fast Clone feature.
Hence we can only choose one feature benefit, not both.
Is that still the case today ?
--
/* Veeam software enthusiast user & supporter ! */
/* Veeam software enthusiast user & supporter ! */
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Veeam and ReFS deduplication
Yes, and as stated above:
foggy wrote:Starting Update 3, Veeam B&R disables block cloning on ReFS volumes with built-in deduplication enabled (unless the repository is CIFS, where we cannot check). The two features cannot be used simultaneously.
-
- Veteran
- Posts: 942
- Liked: 53 times
- Joined: Nov 05, 2009 12:24 pm
- Location: Sydney, NSW
- Contact:
Re: Veeam and ReFS deduplication
Great, many thanks for the clarification Alex.
--
/* Veeam software enthusiast user & supporter ! */
/* Veeam software enthusiast user & supporter ! */
-
- Lurker
- Posts: 2
- Liked: 1 time
- Joined: Oct 03, 2018 10:37 am
- Contact:
Re: Veeam and ReFS deduplication
FIY:
Be aware that REFS+DEDUPE is not possible for Windows Server 2016 Version 1607 which is an LTSC release.
For REFS+DEDUPE you need to have Version 1709 which is Semi Annual channel release. In place upgrades from 1607 to 1709 are not possible.
Be aware that REFS+DEDUPE is not possible for Windows Server 2016 Version 1607 which is an LTSC release.
For REFS+DEDUPE you need to have Version 1709 which is Semi Annual channel release. In place upgrades from 1607 to 1709 are not possible.
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Sep 07, 2011 1:10 am
- Full Name: Mike Sullivan
- Contact:
Re: Veeam and ReFS deduplication
Is it still the case with Server 2019 that you can not use fast cloning if deduplication is enabled on a volume? Are all the ReFS integration features disabled if deduplication is enabled?
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Veeam and ReFS deduplication
Hi Mike,
Actually, in update 4, there will be a way to use fast cloning on deduplicated files (obviously WS19 ReFS + Dedupe only). However, at this point in time (because of full regression testing going on) I can't tell you yet how this will work. We want to make sure it is very safe to use it first (which I think you can understand )
Actually, in update 4, there will be a way to use fast cloning on deduplicated files (obviously WS19 ReFS + Dedupe only). However, at this point in time (because of full regression testing going on) I can't tell you yet how this will work. We want to make sure it is very safe to use it first (which I think you can understand )
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Veeam and ReFS deduplication
Actually, full regression testing has finished on Dec 28, when we shipped the RTM build
In Update 4, there will be an experimental option to enable fast cloning on the deduplicated files via a registry value. Depending on how it shows itself, we will consider enabling that by default in the next update.
In Update 4, there will be an experimental option to enable fast cloning on the deduplicated files via a registry value. Depending on how it shows itself, we will consider enabling that by default in the next update.
-
- Expert
- Posts: 127
- Liked: 29 times
- Joined: Oct 10, 2014 2:06 pm
- Contact:
Re: Veeam and ReFS deduplication
Is there any documentation on that registry key?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Veeam and ReFS deduplication
There will be, once Update 4 is generally available. This actually reminds me to put it in What's New!
-
- Influencer
- Posts: 16
- Liked: 3 times
- Joined: Jan 25, 2019 2:35 pm
- Contact:
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Veeam and ReFS deduplication
It's documented in the What's New document, and there's actually the existing topic going with early adopters sharing their experiences. Thanks!
Who is online
Users browsing this forum: Semrush [Bot], ybarrap2003 and 95 guests