Comprehensive data protection for all workloads
Post Reply
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Safe to update to V11? (dedup + compression)

Post by aschepers »

Hi,

when V11 was released there was an issue with Windows 2019 ReFS with dedup and compression enabled. Backups were very slow or didn't even work at all IIRC.

Now I see the following in the top issues tracker:
#2: Backup hangs at 0KB processed (usually when Windows deduplication is enabled on the repository)
Symptoms: Backup sits at 0KB on the hard disk read step. Potentially, any process that interacts with a backup file may hang when trying to open a backup file.
Cause: Issue with the new high-perf backup backup file interaction engine logic that can happen if a backup storage is very slow to respond to a request to open a backup file. Issue ID 299730.
Status: Fixed in P20210319 or later. If you previously created the DisableHtAsyncIo registry key as a temporary workaround, remember to delete it from the backup repository server.
Does this mean I can safely upgrade to V11? Disabling deduplication is not an option for us as we need the space.

Thanks!
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by HannesK »

Hello,
It's recommended do disable windows deduplication on REFS in any case (forum search finds you comments on performance drawbacks of that approach). There is a good reason, that the combination of REFS block cloning and windows deduplication is only available via a registry key and that the feature is marked as "experimental".

To answer the question: from that bug perspective it's safe.

Best regards,
Hannes
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

Hi Hannes,

thank you for your reply. However, I'm not sure what you mean by Deduplication only accessible by registry key?

I can choose Deduplication from Server Manager:
Image

Are we still talking about the same issue here?

Many thanks.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by HannesK »

Hello,
we are talking about the same issue. Everything fine with that.

I just tried to improve the performance of your environment while maintaining disk space efficiency :-) Except you are using "active full" for whatever reason, I recommend re-thinking the design to use bock-cloning instead of Windows deduplication. If you are not aware of any reg key, then you are not using block cloning right now.

https://www.veeam.com/blog/advanced-ref ... suite.html

I also recommend to check out the Sticky FAQ section about deduplication and REFS to get an idea of the space savings with block cloning http://forums.veeam.com/veeam-backup-re ... tml#p95276

Best regards,
Hannes
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

Thank you again Hannes!

Seems like I have some thinking to do. Admittedly, I'm a little overwhelmed at all the options and their implications.

For instance, you make it seem like active full backups are a bad thing :D Why is that? And if I disable deduplication, will I have enough space or do I need to provide temp storage for backups to seed to while I clean the old respositories?

Apologies if this goes way deeper than the forums are for. If that's the case I will ask our VAR to take a look as well.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by Gostev »

Active fulls are not necessarily a bad thing, they just take a long time and put a significant load on your production storage. Most of the larger customers don't use them, simply because they have so much data that active full backup of their entire environment takes a few days to finish.

We did not even have Active Fulls as an option in the product until v4, when we added them for compliance reasons (some customers had this as a strict requirement).
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by HannesK »

nothing do add from my side. active full vs. synthetic full is also covered in a section of the FAQ I posted above (search for "backup mode")

as you sound happy with your todays solution, I would keep it for simplicity. My concern was that you might be running the default setting (synthetic full) on deduplicated REFS. That would be useless then.

Hard to say whether you need temporary storage - maybe :-) https://social.technet.microsoft.com/Fo ... n-a-volume
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

Thanks guys!

I will stop using Active Fulls in any case, as it provides no benefit to us. I'm not sure why I even enabled them in the first place anymore.

Then I will have to decide if I go to V11 first or first disable deduplication (and how). I have a lot of thinking to do :)
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by PKaufmann »

Hi,

I'd like to know how you are going to proceed as I am in the same situation..

When I set up the repository, I followed the Windows Server Deduplication Part of this guide:
https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html


Also if I disable dedup now and enflate all the files, my disk would be full :(
Capacity : 21.83 TB
FreeSpace : 11.07 TB
UsedSpace : 10.76 TB
UnoptimizedSize : 21.06 TB
SavedSpace : 10.3 TB
jbraes
Novice
Posts: 4
Liked: never
Joined: Jun 15, 2015 3:52 pm
Full Name: Jeroen Braes
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by jbraes »

The problem I have with synthetic full backups is that I don't have the same options for scheduling in Veeam V10.
I can only specify a day of the week , not a specific month, week or day. Things I can do when I schedule an Active Full Backup.
Priit
Novice
Posts: 8
Liked: never
Joined: Sep 07, 2020 11:10 pm
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by Priit »

PKaufmann wrote: Apr 05, 2021 3:01 pm Hi,

I'd like to know how you are going to proceed as I am in the same situation..

When I set up the repository, I followed the Windows Server Deduplication Part of this guide:
https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html


Also if I disable dedup now and enflate all the files, my disk would be full :(
Capacity : 21.83 TB
FreeSpace : 11.07 TB
UsedSpace : 10.76 TB
UnoptimizedSize : 21.06 TB
SavedSpace : 10.3 TB
I followed the same article you did when setting our backups up and as it mentioned in the article "In WS2019 dedup is supported for ReFS but will only dedup the first 4TB of a file.", I enabled that also. Been running that for over 2 years now that way. Still running V10 currently and I would not be able to inflate my backups no matter how much I wanted. I have 2 stores with backup data on them and the deduplication rate is very high:
Volume1:
Capacity : 36.38 TB
FreeSpace : 29.81 TB
UsedSpace : 6.58 TB
UnoptimizedSize : 55.78 TB
SavedSpace : 49.2 TB

Volume2:
Capacity : 36.38 TB
FreeSpace : 28.98 TB
UsedSpace : 7.4 TB
UnoptimizedSize : 117.11 TB
SavedSpace : 109.7 TB

Specially on the second volume, the savings are so substantial from the deduplication that there is no options to inflate and keep that drive without deduplication.
Looking at the recommendations here mentioning issues with the ReFS and deduplication, I have run into that issue twice in 2 years where a backup gets that problem. I have set my deduplication delay to 2 days, so the last 2 day backups dont get deduplicated and if the issue rears its head, I move the affected file off the drive and copy it back, this leaves that file without deduplication for 2 days and allows Veeam to do its operations with it just fine.
If the issue is fixed, that is great, but I just want to verify that the article: https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html is still valid and running ReFS with deduplication is a supported solution as per "In WS2019 dedup is supported for ReFS but will only dedup the first 4TB of a file."?

Priit
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

I would like to know as well. I disabled active full backups and now my weekly backups are still merging (65 hours and counting) and disk space usage is through the roof, almost filling up one of my 2 repositories. I'm wishing I hadn't made this change now.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by HannesK » 1 person likes this post

Hello,
everyone using active full: 3rd party deduplication is fine (whether Windows deduplication on NTFS or REFS or anything else)

for everyone who is happy with his current setup: no need to change anything right now.

As far as I understand, the article I linked earlier, there is no inflation.

Synthetic full on REFS in combination with block cloning: "experimental support" (see reg key in V9.5U4 what's new document). https://www.veeam.com/kb2976 - that will probably never change, because the setup has too many downsides. https://www.veeam.com/veeam_backup_9_5_whats_new_wn.pdf has the information on the reg key.

The best practice guide article is about REFS without block cloning.

Best regards,
Hannes
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by soncscy » 1 person likes this post

I disagree with "best practice" regarding Refs + Dedup; I consider it essential to pick one or the other. I've tested this on dozens of client servers with varying specs, from "budget storage box" to "absolute beast of a machine", and ReFS + Dedup just doesn't work. It's useless at best, and a liability at worst. I've seen the processing rate increase by 100+% by simply disabling dedup and the change in on-disk data usage when both are enabled is not even enough to move the needle sometimes.

I'm not sure who is selling the idea of ReFS+Dedup, but it's a sham as best I can tell, and it simply defies physics. Avoid it, abandon the dream of violating the laws of entropy and getting even more reduced space savings, and just stick to one data-reduction technology. I've yet to see an environment that has "good" ReFS + Dedup that did accurate performance testing against a Dedup/ReFS exclusive environment at production scale; everything is just lab simulations; once you get to production level values (> a few TB of user generated data), the situation become untenable, or at the very least, not a valid challenger for straight dedup or ReFS.
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

I disagree concscy. I have a Veeam B&R server with a Xeon 4110 (16 core) processor at 2.1GHz, 64GB of RAM and a MSA1050 disk array attached to it and I am getting (well, was, until I started messing with no active fulls) ~7Gbit per second throughput during a full backup. This B&R server is fed by 2 proxies in different DC's simultaneously. I also had major space savings as you can see from my post of March 29th and this is all production data.

Meanwhile, my weekend backupjobs are still merging. I thought this was supposed to be lightning fast with ReFS block cloning? Am I now hitting some roadblock concscy hinted at, did I do something wrong or are my expectations just plain wrong? I'm getting more and more confused every day ...
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by HannesK »

I consider it essential to pick one or the other.
yes, correct. As mentioned, the best practice guide article is without block cloning.
I'm not sure who is selling the idea of ReFS+Dedup
at least not Veeam :-) What partners / customers to is sometimes hard to influence...
I thought this was supposed to be lightning fast with ReFS block cloning?
that's what I tried to emphasize multiple times already: Veeam deactivates block-cloning per default, if someone configures deduplication on REFS. See my posts and documentation links above.
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

I could have sworn back in 2018, when this environment was spun up, ReFS + dedup was the way to go when Server 2019 was released. I looked through our documentation but could not find any hint on where we got this impression from, unfortunately.

Appreciate the replies though, even as a long time customer you learn new stuff every day! Now to see how to get out of this hole I dug myself into ...
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by PKaufmann »

I could have sworn back in 2018, when this environment was spun up, ReFS + dedup was the way to go when Server 2019 was released
Me, too - that's why I build it like this ;)

Thanks for all the answers by the way ..

To Sum this up:
If I did not activate block cloning manually it will be disabled by default once enabling deduplication in windows.
And with that config, it's safe to update to v11 - correct?

How can I compare Veeam Dedup with Windows Dedup from a space savings perspective?

Merging takes a while in our environment, too, but best practice with windows dedup and refs is using active full anyway.
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

Merging takes a while in our environment, too, but best practice with windows dedup and refs is using active full anyway.
Can you tell me where you found this best practice?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by HannesK »

@aschepers: I don't believe that this "best practice" is written down anywhere directly. It's just a the logic of "if you have no block cloning, then go with active full to avoid merges".

With the "speed per GB" of spinning disks during the last years, active full is usually the only way to use them.

I also linked an example of REFS disk space savings in the sticky FAQ: post95276.html#p95276 / post368308.html#p368308
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

I see, that makes sense now that I think about it. I reverted the change so we are now doing active fulls once again. When the merge is done (still running ..) I will look into moving away from Windows dedup.
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by PKaufmann »

aschepers wrote: Apr 08, 2021 6:36 am Can you tell me where you found this best practice?
Sure, on the veeam best practice page ;)

https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html
In the part "Windows Server Deduplication" you can find this line:
"Use backup jobs configured to perform Active full with Incrementals"
aschepers
Enthusiast
Posts: 28
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by aschepers »

I see, thanks!
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by PKaufmann »

Does not help me as I only keep the last 7 days on this server, so no active fulls possible as it will ruin the retention..
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by Gostev » 1 person likes this post

1. Performance
2. Disk space usage
3. Cost

Choose any two.
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by PKaufmann »

Quick question: I have

- evacuated all backups from the performance extend
- deleted the old repository
- disabled windows dedup on volume
- deleted the Volume in Windows
- created a new bigger volume (same drive letter as before)
- created new repository and added to SOBR

Still, no block cloning is used ..

Is this because I have used the same drive letter and veeam still "thinks" windows dedup is enabled ?
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by soncscy » 1 person likes this post

@Pkaufmann

As I get it, you missed doing either the Compact Full or a new Active Full.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

> After you have enabled Fast Clone for existing repositories as described in the previous paragraph or have moved backup chains to backup repositories with Fast Clone support, you must create active full backups for backup chains stored in / moved to the repositories to activate Fast Clone. You can also schedule the backup file compact operation instead of performing active full backup.

Under the windows section. The base full backup has to be compacted or it needs to be a new full backup; as I remember from some literature I found awhile back, it has to do with the block alignment of the backup file? I don't remember it exactly, but I do remember from the Microsoft ReFS docs if the data leaves a ReFS volume, all block cloning benefits are lost until you start a new dedicated write, and I guess that's how Veeam starts the block cloning.
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Safe to update to V11? (dedup + compression)

Post by PKaufmann »

Thanks, that makes sense - i will give it a try..
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 190 guests