Comprehensive data protection for all workloads
Locked
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

Could it be the amount of fragmentation thats causing this? I checked our 4K volume before we deleted it and defrag showed 100 % fragmentation...
Gostev
Chief Product Officer
Posts: 31429
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » 5 people like this post

All, I have some good news from Microsoft about this whole issue, stay tuned for the update soon!
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » 2 people like this post

What a tease
j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe »

Hi.
Sorry for the delay in answering
I have changed my setup to NTFS volumes and since then there has been no crash, or freeze.

@Mike Resseler
I have only installed the patches provided by automatic updates, I haven’t downloaded and installed anything. The Veeam support has been very slow at the beginning, took them 48h to respond. He told me the setup is correct, I should use the ReFS volume again to provoke a server frees and create a server memory dump for the support to analyse. He told as well that they may not be able to analyse anything and I end up opening a MS case, which I won't do.

@kubimike
I don't see much besides that the server is freezing. I have some Eventlogs regarding ReFS timeouts...
I have two rather small ReFS repositorys, 15TB at a local SAS attached HP P440ar controller. It is running in HBA mode and I used Storage Spaces to create a parity volume.
The iSCSI ReFS volume is 10TB connected with two 1Gb connections.

Both are 64k and it worked great for about 2 weeks. Now it turns the rest of my hair grey.

John
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » 1 person likes this post

@j.forsythe GEN9 box ? See this post about updating the UEFI veeam-backup-replication-f2/server-2016 ... ml#p231993
j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe »

@kubimike
Yeah, it is an Gen9.
No I have not seen the post. I am downloading at the moment and will give it a try.
Thanks for the hint!
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

@j.forsythe awesome, want to see if that solves your issues.
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: REFS 4k horror story

Post by skumflum »

I have been running 4K REFS on my copy job repository (DR site replication with two restore points) and ran into BSOD and other horrible stuff. I reformatted my volume to 64K a week ago and so far no issue.

The volume is large – 45TB

However, I noticed that the volume is fragmented (88%). Is defragmentation needed on REFS?... I have a feeling that running defragmentation will not go well on 45TB.
Gostev
Chief Product Officer
Posts: 31429
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev »

You should treat ReFS volume where block cloning was used as a sort of deduplication storage. You can never have it NOT fragmented due to the nature of the process (different files sharing the same physical block). As such, there's absolutely no point to perform defragmentation - just like there's no point in defragmenting a deduplicating storage. In fact, the only way to truly defragment any given file would require "inflating" one by recreating the file without leveraging block cloning. Does it make sense?
j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe »

Hello.
I have installed the HP patch from the following link, as mentioned by @kubimike
http://h20564.www2.hpe.com/hpsc/swd/pub ... nvOid=4231

The servers Intelligent Provisioning version is now 2.50.164, but to be honest I haven't checked before the update which version had been installed....
A small test backup didn't cause any problem with the ReFS repository, tonight there will be a bigger one and I will see in the morning if the backup server froze during the night.
Richardrichard
Novice
Posts: 5
Liked: never
Joined: Mar 07, 2017 5:57 am
Full Name: Rich
Contact:

Re: REFS 4k horror story

Post by Richardrichard »

What was the MIcrosoft update please Gostev? I'm currently looking at Veeam and need to work out a best fit repository, need to know if ReFS is on the table.
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD »

Gostev wrote:All, I have some good news from Microsoft about this whole issue, stay tuned for the update soon!
Hi Gostev, any news to share yet? Need to install a ReFS repository this week and it would be nice if I can avoid any pitfalls...
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS 4k horror story

Post by Mike Resseler »

Wim,

I can be wrong but I believe he shared it in his weekly digest. But I don't think there is actually a fix at this moment
Gostev
Chief Product Officer
Posts: 31429
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » 1 person likes this post

Sorry folks, I've got into a little trouble for sharing those news too early in the weekly forum digest, so I need to limit the disclosure to that newsletter for now ;) but in short, the issue is now fully understood and will be addressed by Microsoft soon - the official word from the ReFS dev team should be posted any day now.
graham8
Enthusiast
Posts: 59
Liked: 20 times
Joined: Dec 14, 2016 1:56 pm
Contact:

Re: REFS 4k horror story

Post by graham8 »

Regardless of the specifics of when it will be resolved, it's at least good news just to hear that the actual dev team has become aware of the problem. That takes care of my main concern, which is that this might become one of those "unresolved for years" sorts of problems because the development team is simply never made aware of a serious issue by end-users via frontline support.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

I think I missed the e-mail from Gostev, what did it say ? I've been waiting! :)
j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe » 1 person likes this post

Hello.
Just as a quick update.
Since I installed the HP intelligent provisioning update, there has been no issue with accessing the ReFS volume.
I have changed two jobs, one is writing backups to the RefS iSCSI target, the other one to the local ReFS SAS volume.
Reading data for the "to tape job", is no problem as well.
The last time it worked for about two weeks and not all of the jobs are writing to the ReFS volumes.
But, as I wasn't even able to rescan a repository, this is already a progress.
I will change the last two jobs on Friday, so they can create a new full backup during the weekend.
I will keep you guys updated and hope that there is a MS fix soon too.

John
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD »

Also anxiously awaiting the MS update. Just deployed two proxies with ReFS repositories and have migrated a lot of jobs.
For the moment backup speeds are screaming but they haven't built up their configured restore points yet.
When the jobs start their synthetic operations I hope not to run into any issues or have a fix from MS by then.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

@j.forsynthe are you also running verifier ? Sad to say my box froze today, even with verifier on it failed to create a dump file. It was just frozen at the login screen (Press CTRL + ALT + DEL to login).
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

@Gostev: Do you know if the update released today contains REFS fixes?

EDIT: i just saw there is a new REFS.SYS version 10.0.14393.953 from this month!
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

Isn't it patch Tuesday ? Where is the goods
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

Thanks!
Of the fixed listed which addresses our problems with REFS ?
•Addressed issue with multipath I/O failure that can lead to data corruption or application failures.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

No i think this might fit:

Addressed additional issues with updated time zone information, Internet Explorer, file server and clustering, wireless networking, Map apps, mobile upgrades for IoT, display rendering, USB 2.0 safe removal, multimedia, Direct3D, Microsoft Edge, enterprise security, Windows Server Update Services, storage networking, Remote Desktop, clustering, Windows Hyper-V, and Credential Guard.

As i said above there is a much newer refs.sys in there
graham8
Enthusiast
Posts: 59
Liked: 20 times
Joined: Dec 14, 2016 1:56 pm
Contact:

Re: REFS 4k horror story

Post by graham8 » 2 people like this post

Gotta love how Microsoft has offered less and less information about changes they make over the years. 10 years from now their KBs will just contain $null
Gostev
Chief Product Officer
Posts: 31429
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » 1 person likes this post

Presumably, it fixes one issue around ReFS, but not "our" issue. I am planning to ping ReFS team this week unless they post the blog they were going to soon.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

Oh :(
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS 4k horror story

Post by Mike Resseler »

But it does fixes an issue around ReFS that could lead to corrupted metadata so I would say a rather important update, even if it doesn't fix our issue.
david.buchanan
Service Provider
Posts: 42
Liked: 8 times
Joined: Jun 02, 2015 12:44 am
Full Name: David
Contact:

Re: REFS 4k horror story

Post by david.buchanan »

I haven't read the whole 6 pages but from the sounds of it I should stick with my current Server 2012 R2 NTFS until these issues are resolved or is Server 2016 ReFS 64K the way to go?

Dave
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS 4k horror story

Post by Mike Resseler »

David,

The potential of having issues with 64K are much lower then with 4K. But we have already a report here that it can happen with a very large 64K ReFS volume also. On the other hand, there are many that have 64K volumes with no issues. If you have the possibility to wait until MSFT has solved it. Why not, but it would be a shame if that would put your ReFS 2016 plans on hold for a long time since the benefits of block-cloning API are really cool.

My 2 cents
Locked

Who is online

Users browsing this forum: Bing [Bot], Mildur, uk1984 and 107 guests