Comprehensive data protection for all workloads
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

How do i disable Integrity Streams?

Post by mkretzer »

Hello,

as i stated before we are just about to give up REFS because of its terrible write and merge performance several weeks after active fulls.

As a last hope i searched for REFS performance and found https://www.starwindsoftware.com/blog/refs-performance . What they describe about the declinig performance with integrity streams looks exactly like what we experience when doing active fulls.

The question now is: Is there a way to disable integrity streams for new backups? Our disk array itself ensures consistency so we do not need the additional protection. This is our last hope to "fix" the issues we have before we go back to NTFS...

Markus
tdewin
Veeam Software
Posts: 1775
Liked: 646 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: How do i disable Integrity Streams?

Post by tdewin »

This is set automatically per file by Veeam. Btw, you can not unset it on one file in the chain because for block cloning to work, all files need to have the same settings (at least that was my experience while playing manually with the api)
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

Hello,

thats why i said "for new backups". I want to create an active full and see if it works better!

But how??

Markus
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How do i disable Integrity Streams?

Post by foggy »

Markus, see examples at the bottom of this article.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

Would Veeam not get problems when i simply disable integrity streams for existing files? And would an active full not re-enable integrity streams right away?
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: How do i disable Integrity Streams?

Post by kubimike »

@mkretzer I thought the latest REFS solved your merge issues? Or was that someone else Im thinking of.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: How do i disable Integrity Streams?

Post by nmdange »

Are you certain integrity streams are actually enabled? From what I've seen, when you format ReFS on Windows Server 2016, integrity streams are not enabled by default.

If you run "fsutil fsinfo refsinfo <volume>" what does it say for Checksum type? If it says "CHECKSUM_TYPE_NONE" then integrity streams are disabled on the volume.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

@kubimike: Its better. But merge speed after 4 weeks is still 8 times or so of the original merge speed. We were optimistic as we tested with a volume that was not written to several weeks and there merge speed was fast again.....
@nmdage according to Get-FileIntegrity it is enable by Veeam per file. fsutil shows CHECKSUM_TYPE_NONE. What is the correct info?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How do i disable Integrity Streams?

Post by foggy »

Integrity streams will be enabled for each new backup file, as Timothy has mentioned above. If not all the files in the chain have this setting enabled, block cloning will not be possible (so basically, it is required for REFS integration).
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

But why is this needed for block cloning??
tdewin
Veeam Software
Posts: 1775
Liked: 646 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: How do i disable Integrity Streams?

Post by tdewin »

This is an MS requirement:
https://docs.microsoft.com/en-us/window ... ck-cloning
The source and destination files must have the same Integrity Streams setting.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: How do i disable Integrity Streams?

Post by nmdange »

If integrity streams are disabled at the volume level, why is Veeam turning it on per-file? Ok two block cloned files have to match, but the original file shouldn't have needed integrity streams enabled in the first place.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

@tdewin AGAIN: I want to do NEW active fulls. I am willing to to anything to get this to work. But Veeam is not allowing this right now and enables integrity for every file!!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How do i disable Integrity Streams?

Post by Gostev »

I am fairly sure there was a registry key to disable integrity streams, need to check with the devs.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How do i disable Integrity Streams?

Post by foggy »

The registry value is RefsVirtualSyntheticDisabled, however, it disables FastClone itself as well, so is applicable in scenario with active fulls, but not if you're still going to use synthetic ones.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

Thats what support told me as well. But without fastclone why in the world would i use REFS in the current state?
So there is no solution?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How do i disable Integrity Streams?

Post by foggy »

Currently integrity streams cannot be disabled without disabling fastclone.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: How do i disable Integrity Streams?

Post by nmdange » 1 person likes this post

I think it should really be added as an option to allow integrity streams to be disabled while keeping fastclone enabled. When you are not using storage spaces, integrity streams don't provide much benefit and definitely have a performance overhead. Honestly, I think Veeam should just use whatever default setting for integrity streams is set to on the volume itself.
jonboy
Novice
Posts: 5
Liked: never
Joined: Jan 23, 2015 12:33 pm
Contact:

Re: How do i disable Integrity Streams?

Post by jonboy »

We are experiencing poor read performance issues with REFS on Windows 2016 Server. The backing storage is RAID 10 (HPE P441 with 4GB Cache and 24 x 7.2k 4TB disks). Backups are OK and FastClone works well however we are seeing the Source as the bottleneck when writing to LTO8 drives. (We have been using LTO5 drives until recently and so had not noticed that read is maxing out at around 150MB/s.)

In our case, integrity is being dealt with by the RAID controller. We are not using Storage Spaces so the Integrity Stream at file system level does not offer us a benefit. Or at least not a benefit sufficient to make the poor read performance a valid compromise.

The Microsoft requirement is: "The source and destination files must have the same Integrity Streams setting."

We would like to be able to switch off Integrity Streams on Veeam backup files. Please add this option.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

Bad read performance to LTO8 is not necessarily a REFS issue. We are having the same issues just because Veeam reads data so terribly inefficient with only one stream. We have a Hitachi G200 Storage system with 192 7,2k disks and disk latency alone will make it impossible to get LTO 8 streaming. On the other hand if we use multiple streams our storage can provide up to 2 GB/s (yes, Byte/s)... Here is another thread about that: tape-f29/another-question-about-tape-ba ... 60-15.html
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How do i disable Integrity Streams?

Post by Gostev » 1 person likes this post

Yes, I would say the real issue is what @mkretzer explains. In fact, it's one of my own favorite feature request to enable multi-threaded read from backup files for the operations where it makes sense.

As for ReFS integrity stream - if anything, we actually saw it HELP performance on slower backup storage, because of how they change write pattern - also making reads faster.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

@Gostev Is there any hope for multi-threaded read?
We have a longer case open with our storage vendor and they verified multiple times that with single streams and certain kinds of storage pooling the read cannot get faster without using SSD... So for LTO9 everyone has to get a SSD backend behind Veeam??
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How do i disable Integrity Streams?

Post by Gostev »

There's always hope ;) and your example with LTO-8 helps me raise the importance of this issue internally.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

FYI: Case 03303203, 03402893, 03472515 (current case). In the last case i submitted the storage array vendor analysis.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

@Gostev i just saw that you implemented DisableRefsIntegrityStreams :-)

Did you do any testing how much performance that brings?

In theory disabling this would not prevent Veeam detecting a flipped bit when reading a backup file, correct?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How do i disable Integrity Streams?

Post by Gostev »

Yes we tested and it's not very significant. I don't remember exactly but I think it was no more than 30% in best case scenario. But I'm also not sure about the long-term performance impact for example on fragmentation, because with data integrity streams enabled ReFS behaves VERY differently, like a log-structured file system (which in particular eliminates I/O blender effect).

Veeam can only detect a bit rot if you're lucky enough that it lands in the block belonging to the latest state. So in general, ReFS is a few times more likely to warn you about storage issues sooner.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

I always wondered one thing - how can ReFS warn me about a corrupted block in one of the older restore points if it does not scan the points in backround (which from all i saw until now it does not seem to do)?
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: How do i disable Integrity Streams?

Post by tsightler » 1 person likes this post

ReFS does have a background scrubber to scan infrequently used data. I believe it's run by default every 4 weeks and that you can see it in Task Manager under the Data integrity scanner category. The description for the task is something like "Scans fault tolerant volumes" but this seems to include ReFS with integrity streams enabled, at least, it seems to run on my volumes.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: How do i disable Integrity Streams?

Post by mkretzer »

According to the task planner the job only runs for seconds in our systems... Is that different in your system?
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: How do i disable Integrity Streams?

Post by tsightler »

Unfortunately I've killed all of the ReFS repos in my lab as I'm all Linux all the time these days (hey, finally back to my roots, even while still working at Veeam). But, somewhere along the way it worked because I even managed to get an alert from it on some old backups at one point a few years ago after a power outage caused an unexpected shutdown on my storage system. Thinking about it though, I don't remember seeing the I/O load recently. Maybe I did something special to get it to run in 2016. I've seen conflicting reports of it running only for Storage Spaces or for all ReFS, but the last I was told, it ran for both.
Post Reply

Who is online

Users browsing this forum: ante_704, Bing [Bot], rweis, Semrush [Bot] and 227 guests