How do i disable Integrity Streams?

[mkretzer]

Hello,

as i stated before we are just about to give up REFS because of its terrible write and merge performance several weeks after active fulls.

As a last hope i searched for REFS performance and found https://www.starwindsoftware.com/blog/refs-performance . What they describe about the declinig performance with integrity streams looks exactly like what we experience when doing active fulls.

The question now is: Is there a way to disable integrity streams for new backups? Our disk array itself ensures consistency so we do not need the additional protection. This is our last hope to "fix" the issues we have before we go back to NTFS...

Markus

[tdewin]

This is set automatically per file by Veeam. Btw, you can not unset it on one file in the chain because for block cloning to work, all files need to have the same settings (at least that was my experience while playing manually with the api)

[mkretzer]

Hello,

thats why i said "for new backups". I want to create an active full and see if it works better!

But how??

Markus

[foggy]

Markus, see examples at the bottom of this article.

[mkretzer]

Would Veeam not get problems when i simply disable integrity streams for existing files? And would an active full not re-enable integrity streams right away?

[kubimike]

@mkretzer I thought the latest REFS solved your merge issues? Or was that someone else Im thinking of.

[nmdange]

Are you certain integrity streams are actually enabled? From what I've seen, when you format ReFS on Windows Server 2016, integrity streams are not enabled by default.

If you run "fsutil fsinfo refsinfo <volume>" what does it say for Checksum type? If it says "CHECKSUM_TYPE_NONE" then integrity streams are disabled on the volume.

[mkretzer]

@kubimike: Its better. But merge speed after 4 weeks is still 8 times or so of the original merge speed. We were optimistic as we tested with a volume that was not written to several weeks and there merge speed was fast again.....
@nmdage according to Get-FileIntegrity it is enable by Veeam per file. fsutil shows CHECKSUM_TYPE_NONE. What is the correct info?

[foggy]

Integrity streams will be enabled for each new backup file, as Timothy has mentioned above. If not all the files in the chain have this setting enabled, block cloning will not be possible (so basically, it is required for REFS integration).

[mkretzer]

But why is this needed for block cloning??

[tdewin]

This is an MS requirement:
https://docs.microsoft.com/en-us/window ... ck-cloning
The source and destination files must have the same Integrity Streams setting.

[nmdange]

If integrity streams are disabled at the volume level, why is Veeam turning it on per-file? Ok two block cloned files have to match, but the original file shouldn't have needed integrity streams enabled in the first place.

[mkretzer]

@tdewin AGAIN: I want to do NEW active fulls. I am willing to to anything to get this to work. But Veeam is not allowing this right now and enables integrity for every file!!

[Gostev]

I am fairly sure there was a registry key to disable integrity streams, need to check with the devs.

[foggy]

The registry value is RefsVirtualSyntheticDisabled, however, it disables FastClone itself as well, so is applicable in scenario with active fulls, but not if you're still going to use synthetic ones.

[mkretzer]

Thats what support told me as well. But without fastclone why in the world would i use REFS in the current state?
So there is no solution?

[foggy]

Currently integrity streams cannot be disabled without disabling fastclone.

[nmdange]

I think it should really be added as an option to allow integrity streams to be disabled while keeping fastclone enabled. When you are not using storage spaces, integrity streams don't provide much benefit and definitely have a performance overhead. Honestly, I think Veeam should just use whatever default setting for integrity streams is set to on the volume itself.

[jonboy]

We are experiencing poor read performance issues with REFS on Windows 2016 Server. The backing storage is RAID 10 (HPE P441 with 4GB Cache and 24 x 7.2k 4TB disks). Backups are OK and FastClone works well however we are seeing the Source as the bottleneck when writing to LTO8 drives. (We have been using LTO5 drives until recently and so had not noticed that read is maxing out at around 150MB/s.)

In our case, integrity is being dealt with by the RAID controller. We are not using Storage Spaces so the Integrity Stream at file system level does not offer us a benefit. Or at least not a benefit sufficient to make the poor read performance a valid compromise.

The Microsoft requirement is: "The source and destination files must have the same Integrity Streams setting."

We would like to be able to switch off Integrity Streams on Veeam backup files. Please add this option.

[mkretzer]

Bad read performance to LTO8 is not necessarily a REFS issue. We are having the same issues just because Veeam reads data so terribly inefficient with only one stream. We have a Hitachi G200 Storage system with 192 7,2k disks and disk latency alone will make it impossible to get LTO 8 streaming. On the other hand if we use multiple streams our storage can provide up to 2 GB/s (yes, Byte/s)... Here is another thread about that: tape-f29/another-question-about-tape-ba ... 60-15.html

[Gostev]

Yes, I would say the real issue is what @mkretzer explains. In fact, it's one of my own favorite feature request to enable multi-threaded read from backup files for the operations where it makes sense.

As for ReFS integrity stream - if anything, we actually saw it HELP performance on slower backup storage, because of how they change write pattern - also making reads faster.

[mkretzer]

@Gostev Is there any hope for multi-threaded read?
We have a longer case open with our storage vendor and they verified multiple times that with single streams and certain kinds of storage pooling the read cannot get faster without using SSD... So for LTO9 everyone has to get a SSD backend behind Veeam??

[Gostev]

There's always hope and your example with LTO-8 helps me raise the importance of this issue internally.

[mkretzer]

FYI: Case 03303203, 03402893, 03472515 (current case). In the last case i submitted the storage array vendor analysis.

[mkretzer]

@Gostev i just saw that you implemented DisableRefsIntegrityStreams

Did you do any testing how much performance that brings?

In theory disabling this would not prevent Veeam detecting a flipped bit when reading a backup file, correct?

[Gostev]

Yes we tested and it's not very significant. I don't remember exactly but I think it was no more than 30% in best case scenario. But I'm also not sure about the long-term performance impact for example on fragmentation, because with data integrity streams enabled ReFS behaves VERY differently, like a log-structured file system (which in particular eliminates I/O blender effect).

Veeam can only detect a bit rot if you're lucky enough that it lands in the block belonging to the latest state. So in general, ReFS is a few times more likely to warn you about storage issues sooner.

[mkretzer]

I always wondered one thing - how can ReFS warn me about a corrupted block in one of the older restore points if it does not scan the points in backround (which from all i saw until now it does not seem to do)?

[tsightler]

ReFS does have a background scrubber to scan infrequently used data. I believe it's run by default every 4 weeks and that you can see it in Task Manager under the Data integrity scanner category. The description for the task is something like "Scans fault tolerant volumes" but this seems to include ReFS with integrity streams enabled, at least, it seems to run on my volumes.

[mkretzer]

According to the task planner the job only runs for seconds in our systems... Is that different in your system?

[tsightler]

Unfortunately I've killed all of the ReFS repos in my lab as I'm all Linux all the time these days (hey, finally back to my roots, even while still working at Veeam). But, somewhere along the way it worked because I even managed to get an alert from it on some old backups at one point a few years ago after a power outage caused an unexpected shutdown on my storage system. Thinking about it though, I don't remember seeing the I/O load recently. Maybe I did something special to get it to run in 2016. I've seen conflicting reports of it running only for Storage Spaces or for all ReFS, but the last I was told, it ran for both.