Host-based backup of Microsoft Hyper-V VMs.
Post Reply
jzilak
Influencer
Posts: 19
Liked: 1 time
Joined: May 10, 2017 9:01 am
Full Name: Josef Zilak
Contact:

Option for controling ReFS integrity stream

Post by jzilak »

Because Integrity stream is now optional for ReFS and is very resource expensive (degrade 50% write performance and consume many CPU resource 25% or more) it would be nice to have control about this feature. for example option to enable/disable this resource hungry feature per backup job.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Option for controling ReFS integrity stream

Post by dellock6 »

Hi Josef,
integrity streams are enabled on ReFS used for backups to guarantee that files are not corrupted, be immediately warned about bit rot (and potentially solve the issue automatically with storage spaces) and so restore operations can ultimately be reliable. As a data protection solution, we believe this is a strong proposition. Would you give away this advantage?
Also, in all the tests I've seen and read about, I've never seen such a degradation like the one you are quoting, may I ask you were these results are coming from?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
jzilak
Influencer
Posts: 19
Liked: 1 time
Joined: May 10, 2017 9:01 am
Full Name: Josef Zilak
Contact:

Re: Option for controling ReFS integrity stream

Post by jzilak »

Hello Luca,

I will throw this advantage without any problem. First there is many other option for backup validation, so this feature is not only possible solution. Second performance issues on Refs 3.1 with Veeam are huge (major issues is MS problem, but partly it is VEEAM problem in way how fastclone is implemented). We are running VBR 9.5U1 and backup VMs with more than 10TB storage attached to single VM (few VMs has currently allocated ~50TB). For this VMs, write performance is much more important than early info about archive corruption.
Reason is simple, even I'm aware about archive corruption during read operation, without Storage Spaces there is no way how to correct this corruption and archive is lost anyway. So there is simple no benefit or guaranties that archive will be OK for sure. Only real benefit is, that you receive info, that archive is corrupted and that's all.

What is visible in system for this large fastcloned archives, that storage has many performance problems even for read operations (because of hi archive fragmentation and integrity stream checks) and it is much worst for write operations, again due to hi fragmentation and integrity stream computation.

For ReFS 1.2 (integrity streams disabled)/NTFS storage is able to read ~500Mb/s and write ~250MB/s. But with ReFs 3.1 with integrity streams enabled, storage performance drop ~100MB/s for write and ~150MB/s for read

So give users possibility to chose what is better for his system is much better solution than believe, that only you are right and your test cover all situations and scenarios that your customers use. (and "ReFS 4k horror story" on your forum is nice example how badly you validate ReFS 3.1 features)

Regards
josef
jzilak
Influencer
Posts: 19
Liked: 1 time
Joined: May 10, 2017 9:01 am
Full Name: Josef Zilak
Contact:

Re: Option for controling ReFS integrity stream

Post by jzilak »

Here is part of MS documentation about integrity streams performance:

ReFS integrity streams
https://docs.microsoft.com/en-us/window ... ty-streams

Performance
Though integrity streams provides greater data integrity for the system, it also incurs a performance cost. There are a couple different reasons for this:
+
If integrity streams are enabled, all write operations become allocate-on-write operations. Though this avoids the read-modify-write bottleneck, data frequently becomes fragmented, which delays reads.
Depending on the workload and underlying storage of the system, the computational cost of computing and validating the checksum can cause IO latency to increase.
+
Because integrity streams carries a performance cost, we recommend leaving integrity streams disabled on highly performant systems.
Post Reply

Who is online

Users browsing this forum: No registered users and 29 guests