Safe to update to V11? (dedup + compression)

[aschepers]

Hi,

when V11 was released there was an issue with Windows 2019 ReFS with dedup and compression enabled. Backups were very slow or didn't even work at all IIRC.

Now I see the following in the top issues tracker:

#2: Backup hangs at 0KB processed (usually when Windows deduplication is enabled on the repository)
Symptoms: Backup sits at 0KB on the hard disk read step. Potentially, any process that interacts with a backup file may hang when trying to open a backup file.
Cause: Issue with the new high-perf backup backup file interaction engine logic that can happen if a backup storage is very slow to respond to a request to open a backup file. Issue ID 299730.
Status: Fixed in P20210319 or later. If you previously created the DisableHtAsyncIo registry key as a temporary workaround, remember to delete it from the backup repository server.

Does this mean I can safely upgrade to V11? Disabling deduplication is not an option for us as we need the space.

Thanks!

[HannesK]

Hello,
It's recommended do disable windows deduplication on REFS in any case (forum search finds you comments on performance drawbacks of that approach). There is a good reason, that the combination of REFS block cloning and windows deduplication is only available via a registry key and that the feature is marked as "experimental".

To answer the question: from that bug perspective it's safe.

Best regards,
Hannes

[aschepers]

Hi Hannes,

thank you for your reply. However, I'm not sure what you mean by Deduplication only accessible by registry key?

I can choose Deduplication from Server Manager:


Are we still talking about the same issue here?

Many thanks.

[HannesK]

Hello,
we are talking about the same issue. Everything fine with that.

I just tried to improve the performance of your environment while maintaining disk space efficiency Except you are using "active full" for whatever reason, I recommend re-thinking the design to use bock-cloning instead of Windows deduplication. If you are not aware of any reg key, then you are not using block cloning right now.

https://www.veeam.com/blog/advanced-ref ... suite.html

I also recommend to check out the Sticky FAQ section about deduplication and REFS to get an idea of the space savings with block cloning http://forums.veeam.com/veeam-backup-re ... tml#p95276

Best regards,
Hannes

[aschepers]

Thank you again Hannes!

Seems like I have some thinking to do. Admittedly, I'm a little overwhelmed at all the options and their implications.

For instance, you make it seem like active full backups are a bad thing Why is that? And if I disable deduplication, will I have enough space or do I need to provide temp storage for backups to seed to while I clean the old respositories?

Apologies if this goes way deeper than the forums are for. If that's the case I will ask our VAR to take a look as well.

[Gostev]

Active fulls are not necessarily a bad thing, they just take a long time and put a significant load on your production storage. Most of the larger customers don't use them, simply because they have so much data that active full backup of their entire environment takes a few days to finish.

We did not even have Active Fulls as an option in the product until v4, when we added them for compliance reasons (some customers had this as a strict requirement).

[HannesK]

nothing do add from my side. active full vs. synthetic full is also covered in a section of the FAQ I posted above (search for "backup mode")

as you sound happy with your todays solution, I would keep it for simplicity. My concern was that you might be running the default setting (synthetic full) on deduplicated REFS. That would be useless then.

Hard to say whether you need temporary storage - maybe https://social.technet.microsoft.com/Fo ... n-a-volume

[aschepers]

Thanks guys!

I will stop using Active Fulls in any case, as it provides no benefit to us. I'm not sure why I even enabled them in the first place anymore.

Then I will have to decide if I go to V11 first or first disable deduplication (and how). I have a lot of thinking to do

[PKaufmann]

Hi,

I'd like to know how you are going to proceed as I am in the same situation..

When I set up the repository, I followed the Windows Server Deduplication Part of this guide:
https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html


Also if I disable dedup now and enflate all the files, my disk would be full
Capacity : 21.83 TB
FreeSpace : 11.07 TB
UsedSpace : 10.76 TB
UnoptimizedSize : 21.06 TB
SavedSpace : 10.3 TB

[jbraes]

The problem I have with synthetic full backups is that I don't have the same options for scheduling in Veeam V10.
I can only specify a day of the week , not a specific month, week or day. Things I can do when I schedule an Active Full Backup.

[Priit]

Hi,

I'd like to know how you are going to proceed as I am in the same situation..

When I set up the repository, I followed the Windows Server Deduplication Part of this guide:
https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html


Also if I disable dedup now and enflate all the files, my disk would be full
Capacity : 21.83 TB
FreeSpace : 11.07 TB
UsedSpace : 10.76 TB
UnoptimizedSize : 21.06 TB
SavedSpace : 10.3 TB

I followed the same article you did when setting our backups up and as it mentioned in the article "In WS2019 dedup is supported for ReFS but will only dedup the first 4TB of a file.", I enabled that also. Been running that for over 2 years now that way. Still running V10 currently and I would not be able to inflate my backups no matter how much I wanted. I have 2 stores with backup data on them and the deduplication rate is very high:
Volume1:
Capacity : 36.38 TB
FreeSpace : 29.81 TB
UsedSpace : 6.58 TB
UnoptimizedSize : 55.78 TB
SavedSpace : 49.2 TB

Volume2:
Capacity : 36.38 TB
FreeSpace : 28.98 TB
UsedSpace : 7.4 TB
UnoptimizedSize : 117.11 TB
SavedSpace : 109.7 TB

Specially on the second volume, the savings are so substantial from the deduplication that there is no options to inflate and keep that drive without deduplication.
Looking at the recommendations here mentioning issues with the ReFS and deduplication, I have run into that issue twice in 2 years where a backup gets that problem. I have set my deduplication delay to 2 days, so the last 2 day backups dont get deduplicated and if the issue rears its head, I move the affected file off the drive and copy it back, this leaves that file without deduplication for 2 days and allows Veeam to do its operations with it just fine.
If the issue is fixed, that is great, but I just want to verify that the article: https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html is still valid and running ReFS with deduplication is a supported solution as per "In WS2019 dedup is supported for ReFS but will only dedup the first 4TB of a file."?

Priit

[aschepers]

I would like to know as well. I disabled active full backups and now my weekly backups are still merging (65 hours and counting) and disk space usage is through the roof, almost filling up one of my 2 repositories. I'm wishing I hadn't made this change now.

[HannesK]

Hello,
everyone using active full: 3rd party deduplication is fine (whether Windows deduplication on NTFS or REFS or anything else)

for everyone who is happy with his current setup: no need to change anything right now.

As far as I understand, the article I linked earlier, there is no inflation.

Synthetic full on REFS in combination with block cloning: "experimental support" (see reg key in V9.5U4 what's new document). https://www.veeam.com/kb2976 - that will probably never change, because the setup has too many downsides. https://www.veeam.com/veeam_backup_9_5_whats_new_wn.pdf has the information on the reg key.

The best practice guide article is about REFS without block cloning.

Best regards,
Hannes

[soncscy]

I disagree with "best practice" regarding Refs + Dedup; I consider it essential to pick one or the other. I've tested this on dozens of client servers with varying specs, from "budget storage box" to "absolute beast of a machine", and ReFS + Dedup just doesn't work. It's useless at best, and a liability at worst. I've seen the processing rate increase by 100+% by simply disabling dedup and the change in on-disk data usage when both are enabled is not even enough to move the needle sometimes.

I'm not sure who is selling the idea of ReFS+Dedup, but it's a sham as best I can tell, and it simply defies physics. Avoid it, abandon the dream of violating the laws of entropy and getting even more reduced space savings, and just stick to one data-reduction technology. I've yet to see an environment that has "good" ReFS + Dedup that did accurate performance testing against a Dedup/ReFS exclusive environment at production scale; everything is just lab simulations; once you get to production level values (> a few TB of user generated data), the situation become untenable, or at the very least, not a valid challenger for straight dedup or ReFS.

[aschepers]

I disagree concscy. I have a Veeam B&R server with a Xeon 4110 (16 core) processor at 2.1GHz, 64GB of RAM and a MSA1050 disk array attached to it and I am getting (well, was, until I started messing with no active fulls) ~7Gbit per second throughput during a full backup. This B&R server is fed by 2 proxies in different DC's simultaneously. I also had major space savings as you can see from my post of March 29th and this is all production data.

Meanwhile, my weekend backupjobs are still merging. I thought this was supposed to be lightning fast with ReFS block cloning? Am I now hitting some roadblock concscy hinted at, did I do something wrong or are my expectations just plain wrong? I'm getting more and more confused every day ...

[HannesK]

I consider it essential to pick one or the other.

yes, correct. As mentioned, the best practice guide article is without block cloning.

I'm not sure who is selling the idea of ReFS+Dedup

at least not Veeam What partners / customers to is sometimes hard to influence...

I thought this was supposed to be lightning fast with ReFS block cloning?

that's what I tried to emphasize multiple times already: Veeam deactivates block-cloning per default, if someone configures deduplication on REFS. See my posts and documentation links above.

[aschepers]

I could have sworn back in 2018, when this environment was spun up, ReFS + dedup was the way to go when Server 2019 was released. I looked through our documentation but could not find any hint on where we got this impression from, unfortunately.

Appreciate the replies though, even as a long time customer you learn new stuff every day! Now to see how to get out of this hole I dug myself into ...

[PKaufmann]

I could have sworn back in 2018, when this environment was spun up, ReFS + dedup was the way to go when Server 2019 was released

Me, too - that's why I build it like this

Thanks for all the answers by the way ..

To Sum this up:
If I did not activate block cloning manually it will be disabled by default once enabling deduplication in windows.
And with that config, it's safe to update to v11 - correct?

How can I compare Veeam Dedup with Windows Dedup from a space savings perspective?

Merging takes a while in our environment, too, but best practice with windows dedup and refs is using active full anyway.

[aschepers]

Merging takes a while in our environment, too, but best practice with windows dedup and refs is using active full anyway.

Can you tell me where you found this best practice?

[HannesK]

@aschepers: I don't believe that this "best practice" is written down anywhere directly. It's just a the logic of "if you have no block cloning, then go with active full to avoid merges".

With the "speed per GB" of spinning disks during the last years, active full is usually the only way to use them.

I also linked an example of REFS disk space savings in the sticky FAQ: post95276.html#p95276 / post368308.html#p368308

[aschepers]

I see, that makes sense now that I think about it. I reverted the change so we are now doing active fulls once again. When the merge is done (still running ..) I will look into moving away from Windows dedup.

[PKaufmann]

Can you tell me where you found this best practice?

Sure, on the veeam best practice page

https://bp.veeam.com/vbr/VBP/3_Build_st ... block.html
In the part "Windows Server Deduplication" you can find this line:
"Use backup jobs configured to perform Active full with Incrementals"

[aschepers]

I see, thanks!

[PKaufmann]

Does not help me as I only keep the last 7 days on this server, so no active fulls possible as it will ruin the retention..

[Gostev]

1. Performance
2. Disk space usage
3. Cost

Choose any two.

[PKaufmann]

Quick question: I have

- evacuated all backups from the performance extend
- deleted the old repository
- disabled windows dedup on volume
- deleted the Volume in Windows
- created a new bigger volume (same drive letter as before)
- created new repository and added to SOBR

Still, no block cloning is used ..

Is this because I have used the same drive letter and veeam still "thinks" windows dedup is enabled ?

[soncscy]

@Pkaufmann

As I get it, you missed doing either the Compact Full or a new Active Full.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

> After you have enabled Fast Clone for existing repositories as described in the previous paragraph or have moved backup chains to backup repositories with Fast Clone support, you must create active full backups for backup chains stored in / moved to the repositories to activate Fast Clone. You can also schedule the backup file compact operation instead of performing active full backup.

Under the windows section. The base full backup has to be compacted or it needs to be a new full backup; as I remember from some literature I found awhile back, it has to do with the block alignment of the backup file? I don't remember it exactly, but I do remember from the Microsoft ReFS docs if the data leaves a ReFS volume, all block cloning benefits are lost until you start a new dedicated write, and I guess that's how Veeam starts the block cloning.

[PKaufmann]

Thanks, that makes sense - i will give it a try..