How do i disable Integrity Streams?

[mkretzer]

I will just ask MS devs. Lets see what they say about it...

[mkretzer]

MS will respond to this thread shortly

[tsightler]

Sounds good. I've been racking my brain trying to remember if maybe I was actually using storage spaces at the time!

[mkretzer]

Spoiler: you must have according to MS...

[Gostev]

This definitely can't be true: I saw ReFS detecting a storage corruption issue on a regular volume with Veeam backup files. I needed a screenshot of the Windows Event Log error for my presentation, and QC folks purposely corrupted some bits in VMDK to trigger one.

[mkretzer]

Gostev, if what you say is true it would mean that ReFS would have to do backround IO on all data on the volume.

Our monitoring cannot see any backround IO when there is no backup running.

I could understand if it could detect files that are just beeing accessed but not files which are just are on the disk but not accessed...

[Gostev]

All I'm saying is that data integrity checks do definitely work even on basic volumes (doesn't require Storage Spaces). I don't know when or how often the check is performed though.

But in any case, for me the biggest benefits of data integrity streams has always been performance benefits due to the change of I/O pattern that are beneficial for slow backup storage with poor random I/O capacity, which is what we see used most commonly by our SMB customers.

[tsightler]

Well, there's no doubt that simply reading the file can generate the event log error message. I've definitely seen events created during, for example, Veeam health checks when there was active corruption in a tested file, even on simple volumes.

But I think @mkretzer is talking more about a background scrub to detect corruption. Honestly, a simple script that just walked the directories and read every file should be enough.

[mkretzer]

Yes, thats what i meant. 30 % better performance sounds good if there is no big benefit - in know the IO pattern changes then, at least it is an option without a big risk missing corruption if that is really true....

[soncscy]

https://docs.microsoft.com/en-us/window ... y-scrubber

Isn't this the automated process being talked about? There should be a scheduled task running every so often to catch this issues on ReFS in the background, regardless of the volume type.

[mkretzer]

@soncscy yes thats what i am talking about.

But: Did you ever see the process taking more than a few seconds? We have 600 TB of ReFS on our primary system, nearly full. For the scan to work at 100 MB/s (single stream i guess, slow rotating media) the scan would take 69 days. I see no IO whatsoever when there are no backups running.

This leaves two options:
- Eighter i am right (as microsoft ReFS devs have confirmed and hopefully will write a statement here) and integrity scan does nothing on non-storagespace volumes - this means integrity streams does not have any benefit on ReFS other than altering the IO pattern which might bring some kind of benefit - or not
- Or my servers behave incorrectly and i need to open a ticket with MS

Markus

[Larissa@MSFT]

DISCLAIMER: I work for Microsoft as a Program Manager on the Storage and File Systems team - specifically ReFS.

First, thank you all for using ReFS and sharing your feedback! It’s always great to hear first-hand user experience and areas for improvement. I wanted to follow up on a few topics that have been discussed on this thread:

ReFS Integrity Streams provide data integrity. When enabled, it computes and stores data checksums on write and validates data against stored checksums on read. This functionality can be enabled irrespective of the physical media backing the ReFS volume.

If ReFS is on top of resilient Windows Storage Spaces (i.e. Mirror, Parity), ReFS will periodically scrub data with integrity checksums to detect and correct corruption. This scrub function is a monthly scheduled task named “Data Integrity Scan”. On non resilient media, or non-Windows Storage Spaces media this functionality is unavailable.

As Mkretzer mentioned, if an application maintains its own data checksums, and is not on resilient Windows Storage Spaces media, ReFS integrity streams do not provide any additional benefit over regular (non-integrity) streams.

As Gostev mentioned, when ReFS is on a non-resilient/simple Storage Space or bare drive, ReFS Integrity Streams will detect data corruption on read, log an event, but would not be able to correct it.

Therefore, ReFS Integrity Streams work best with mirrored or parity Storage Spaces.

I’m considering a Blog post to cover this in more detail, but feel free to ask any further questions you might have.

[mkretzer]

Larissa,

thank you very much for the clarification. Gostev mentioned a change in IO pattern caused by enabling integrity streams for files.

Is there a way to get this "benefit" without the additional load caused by integrity streams enabled?

Markus

[Larissa@MSFT]

When Integrity Streams are enabled, any overwrite becomes a copy-on-write operation which changes the IO pattern. When Integrity Streams are not enabled, any overwrite is performed in place and does not change the IO pattern.

Copy-on-write operations have additional costs:
1.Pathlength - Modifying metadata, allocation in IO codepath.
2.IOPS - dirty metadata, and log records have to be written onto the disk.

Whether the changed IO pattern justifies the cost fundamentally depends on the type of media ReFS is on. We are unable to make blanket recommendation one way or the other.

[Larissa@MSFT]

DISCLAIMER: I work for Microsoft as a Program Manager on the Storage and File Systems team - specifically ReFS.

First, thank you all for using ReFS and sharing your feedback! It’s always great to hear first-hand user experience and areas for improvement. I wanted to follow up on a few topics that have been discussed on this thread:

ReFS Integrity Streams provide data integrity. When enabled, it computes and stores data checksums on write and validates data against stored checksums on read. This functionality can be enabled irrespective of the physical media backing the ReFS volume.

If ReFS is on top of resilient Windows Storage Spaces (i.e. Mirror, Parity), ReFS will periodically scrub data with integrity checksums to detect and correct corruption. This scrub function is a monthly scheduled task named “Data Integrity Scan”. On non resilient media, or non-Windows Storage Spaces media this functionality is unavailable.

As Mkretzer mentioned, if an application maintains its own data checksums, and is not on resilient Windows Storage Spaces media, ReFS integrity streams do not provide any additional benefit over regular (non-integrity) streams.

As Gostev mentioned, when ReFS is on a non-resilient/simple Storage Space or bare drive, ReFS Integrity Streams will detect data corruption on read, log an event, but would not be able to correct it.

Therefore, ReFS Integrity Streams work best with mirrored or parity Storage Spaces.

I’m considering a Blog post to cover this in more detail, but feel free to ask any further questions you might have.

Also something to note is ReFS prefers to update in-place when possible. Update in-place does not have any additional metadata cost and is the normal codepath for non-integrity streams.
So when Integrity Streams are enabled, any overwrite becomes a copy-on-write operation.

[yara12]

In the description of what's new in 11 Veeam it is mentioned about creating a key in the registry, but I don't find something in the docs:

• Disable ReFS integrity streams - You can now disable ReFS integrity streams for Veeam backup files
using the DisableRefsIntegrityStreams (DWORD, 1) registry value on the backup server. While we do not
recommend disabling integrity streams because data integrity is paramount in data protection, some
customers insisted on having the ability to control this setting to improve performance.

Question: Which is the path for this key?

[veremin]

All Veeam Backup & Replication registry keys should go to the standard HKLM\SOFTWARE\Veeam\Veeam Backup and Replication hive, unless specifically stated otherwise. Thanks!

[yara12]

One more question)
HKLM\SOFTWARE\Veeam\Veeam Backup and Replication on VBR or on Repo Server?
What about case of using cloud connect server - is this parameter passed from the tenant?

[veremin]

On the backup server, and yes it works for CC deployments as well:

Disable ReFS integrity streams - You can now disable ReFS integrity streams for Veeam backup files using the DisableRefsIntegrityStreams (DWORD, 1) registry value on the backup server.

Thanks!