Comprehensive data protection for all workloads
Locked
mkretzer
Expert
Posts: 566
Liked: 127 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer » Mar 03, 2017 6:12 am

Could it be the amount of fragmentation thats causing this? I checked our 4K volume before we deleted it and defrag showed 100 % fragmentation...

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » Mar 03, 2017 4:45 pm 5 people like this post

All, I have some good news from Microsoft about this whole issue, stay tuned for the update soon!

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 03, 2017 8:06 pm 2 people like this post

What a tease

j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe » Mar 07, 2017 8:58 am

Hi.
Sorry for the delay in answering
I have changed my setup to NTFS volumes and since then there has been no crash, or freeze.

@Mike Resseler
I have only installed the patches provided by automatic updates, I haven’t downloaded and installed anything. The Veeam support has been very slow at the beginning, took them 48h to respond. He told me the setup is correct, I should use the ReFS volume again to provoke a server frees and create a server memory dump for the support to analyse. He told as well that they may not be able to analyse anything and I end up opening a MS case, which I won't do.

@kubimike
I don't see much besides that the server is freezing. I have some Eventlogs regarding ReFS timeouts...
I have two rather small ReFS repositorys, 15TB at a local SAS attached HP P440ar controller. It is running in HBA mode and I used Storage Spaces to create a parity volume.
The iSCSI ReFS volume is 10TB connected with two 1Gb connections.

Both are 64k and it worked great for about 2 weeks. Now it turns the rest of my hair grey.

John

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 07, 2017 3:09 pm 1 person likes this post

@j.forsythe GEN9 box ? See this post about updating the UEFI veeam-backup-replication-f2/server-2016 ... ml#p231993

j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe » Mar 07, 2017 3:27 pm

@kubimike
Yeah, it is an Gen9.
No I have not seen the post. I am downloading at the moment and will give it a try.
Thanks for the hint!

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 07, 2017 4:45 pm

@j.forsythe awesome, want to see if that solves your issues.

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: REFS 4k horror story

Post by skumflum » Mar 08, 2017 10:24 am

I have been running 4K REFS on my copy job repository (DR site replication with two restore points) and ran into BSOD and other horrible stuff. I reformatted my volume to 64K a week ago and so far no issue.

The volume is large – 45TB

However, I noticed that the volume is fragmented (88%). Is defragmentation needed on REFS?... I have a feeling that running defragmentation will not go well on 45TB.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » Mar 08, 2017 2:26 pm

You should treat ReFS volume where block cloning was used as a sort of deduplication storage. You can never have it NOT fragmented due to the nature of the process (different files sharing the same physical block). As such, there's absolutely no point to perform defragmentation - just like there's no point in defragmenting a deduplicating storage. In fact, the only way to truly defragment any given file would require "inflating" one by recreating the file without leveraging block cloning. Does it make sense?

j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe » Mar 08, 2017 3:11 pm

Hello.
I have installed the HP patch from the following link, as mentioned by @kubimike
http://h20564.www2.hpe.com/hpsc/swd/pub ... nvOid=4231

The servers Intelligent Provisioning version is now 2.50.164, but to be honest I haven't checked before the update which version had been installed....
A small test backup didn't cause any problem with the ReFS repository, tonight there will be a bigger one and I will see in the morning if the backup server froze during the night.

Richardrichard
Novice
Posts: 5
Liked: never
Joined: Mar 07, 2017 5:57 am
Full Name: Rich
Contact:

Re: REFS 4k horror story

Post by Richardrichard » Mar 08, 2017 3:26 pm

What was the MIcrosoft update please Gostev? I'm currently looking at Veeam and need to work out a best fit repository, need to know if ReFS is on the table.

WimVD
Service Provider
Posts: 53
Liked: 18 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD » Mar 08, 2017 4:25 pm

Gostev wrote:All, I have some good news from Microsoft about this whole issue, stay tuned for the update soon!
Hi Gostev, any news to share yet? Need to install a ReFS repository this week and it would be nice if I can avoid any pitfalls...

Mike Resseler
Product Manager
Posts: 5760
Liked: 616 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS 4k horror story

Post by Mike Resseler » Mar 08, 2017 4:31 pm

Wim,

I can be wrong but I believe he shared it in his weekly digest. But I don't think there is actually a fix at this moment

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » Mar 08, 2017 5:58 pm 1 person likes this post

Sorry folks, I've got into a little trouble for sharing those news too early in the weekly forum digest, so I need to limit the disclosure to that newsletter for now ;) but in short, the issue is now fully understood and will be addressed by Microsoft soon - the official word from the ReFS dev team should be posted any day now.

graham8
Enthusiast
Posts: 59
Liked: 20 times
Joined: Dec 14, 2016 1:56 pm
Contact:

Re: REFS 4k horror story

Post by graham8 » Mar 08, 2017 6:35 pm

Regardless of the specifics of when it will be resolved, it's at least good news just to hear that the actual dev team has become aware of the problem. That takes care of my main concern, which is that this might become one of those "unresolved for years" sorts of problems because the development team is simply never made aware of a serious issue by end-users via frontline support.

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 09, 2017 9:36 pm

I think I missed the e-mail from Gostev, what did it say ? I've been waiting! :)

j.forsythe
Influencer
Posts: 15
Liked: 4 times
Joined: Jan 06, 2016 10:26 am
Full Name: John P. Forsythe
Contact:

Re: REFS 4k horror story

Post by j.forsythe » Mar 14, 2017 7:26 am 1 person likes this post

Hello.
Just as a quick update.
Since I installed the HP intelligent provisioning update, there has been no issue with accessing the ReFS volume.
I have changed two jobs, one is writing backups to the RefS iSCSI target, the other one to the local ReFS SAS volume.
Reading data for the "to tape job", is no problem as well.
The last time it worked for about two weeks and not all of the jobs are writing to the ReFS volumes.
But, as I wasn't even able to rescan a repository, this is already a progress.
I will change the last two jobs on Friday, so they can create a new full backup during the weekend.
I will keep you guys updated and hope that there is a MS fix soon too.

John

WimVD
Service Provider
Posts: 53
Liked: 18 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD » Mar 14, 2017 10:05 am

Also anxiously awaiting the MS update. Just deployed two proxies with ReFS repositories and have migrated a lot of jobs.
For the moment backup speeds are screaming but they haven't built up their configured restore points yet.
When the jobs start their synthetic operations I hope not to run into any issues or have a fix from MS by then.

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 14, 2017 2:34 pm

@j.forsynthe are you also running verifier ? Sad to say my box froze today, even with verifier on it failed to create a dump file. It was just frozen at the login screen (Press CTRL + ALT + DEL to login).

mkretzer
Expert
Posts: 566
Liked: 127 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer » Mar 14, 2017 8:14 pm

@Gostev: Do you know if the update released today contains REFS fixes?

EDIT: i just saw there is a new REFS.SYS version 10.0.14393.953 from this month!

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 14, 2017 8:28 pm

Isn't it patch Tuesday ? Where is the goods

mkretzer
Expert
Posts: 566
Liked: 127 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer » Mar 14, 2017 8:44 pm


kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 14, 2017 8:51 pm

Thanks!
Of the fixed listed which addresses our problems with REFS ?
•Addressed issue with multipath I/O failure that can lead to data corruption or application failures.

mkretzer
Expert
Posts: 566
Liked: 127 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer » Mar 14, 2017 9:00 pm

No i think this might fit:

Addressed additional issues with updated time zone information, Internet Explorer, file server and clustering, wireless networking, Map apps, mobile upgrades for IoT, display rendering, USB 2.0 safe removal, multimedia, Direct3D, Microsoft Edge, enterprise security, Windows Server Update Services, storage networking, Remote Desktop, clustering, Windows Hyper-V, and Credential Guard.

As i said above there is a much newer refs.sys in there

graham8
Enthusiast
Posts: 59
Liked: 20 times
Joined: Dec 14, 2016 1:56 pm
Contact:

Re: REFS 4k horror story

Post by graham8 » Mar 14, 2017 9:06 pm 2 people like this post

Gotta love how Microsoft has offered less and less information about changes they make over the years. 10 years from now their KBs will just contain $null

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » Mar 14, 2017 11:32 pm 1 person likes this post

Presumably, it fixes one issue around ReFS, but not "our" issue. I am planning to ping ReFS team this week unless they post the blog they were going to soon.

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Mar 14, 2017 11:50 pm

Oh :(

Mike Resseler
Product Manager
Posts: 5760
Liked: 616 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS 4k horror story

Post by Mike Resseler » Mar 15, 2017 7:06 am

But it does fixes an issue around ReFS that could lead to corrupted metadata so I would say a rather important update, even if it doesn't fix our issue.

david.buchanan
Service Provider
Posts: 42
Liked: 8 times
Joined: Jun 02, 2015 12:44 am
Full Name: David
Contact:

Re: REFS 4k horror story

Post by david.buchanan » Mar 15, 2017 7:32 am

I haven't read the whole 6 pages but from the sounds of it I should stick with my current Server 2012 R2 NTFS until these issues are resolved or is Server 2016 ReFS 64K the way to go?

Dave

Mike Resseler
Product Manager
Posts: 5760
Liked: 616 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS 4k horror story

Post by Mike Resseler » Mar 15, 2017 7:41 am

David,

The potential of having issues with 64K are much lower then with 4K. But we have already a report here that it can happen with a very large 64K ReFS volume also. On the other hand, there are many that have 64K volumes with no issues. If you have the possibility to wait until MSFT has solved it. Why not, but it would be a shame if that would put your ReFS 2016 plans on hold for a long time since the benefits of block-cloning API are really cool.

My 2 cents

Locked

Who is online

Users browsing this forum: Bing [Bot] and 31 guests