REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Raleigh » Fri Jul 13, 2018 10:29 pm

Thank you for the reply, Gostev.

Point taken regarding opening support cases with Microsoft regarding ReFS issues. If having many Veeam customers open cases with Microsoft will better motivate them to resolve the issues, then I was happy to participate with that. Microsoft Support has not yet admitted to me that my issue is the result of any known bug yet (they are *still* in the process of analyzing my memory.dmp file). So I may have to push at them on that.

You are also correct: I have no idea what facilities and resources Veeam engineers have engaged on this issue. The Veeam support technicians I worked with never mentioned that Veeam was working directly with MS to resolve the ReFS issues. Actually, the first Veeam Support tech that I worked with when I opened the ticket (this was back in early April) told me that it was her understanding that the ReFS issues were resolved by the February Windows Updates. So apparently, she was not aware of any ongoing initiative with Microsoft either, or at least didn’t feel it was relevant to my issue.

Yes, it’s true that I created my login to the Veeam Community Forum only several weeks ago, but I have been reading this topic thread since my problem began. The first Veeam Support tech told me about this forum topic. I did not need to create an account until I wanted to submit a post. I only wish I had done that much sooner. I will not make that mistake again, since this forum is where the solution to my repo server issue came from.

Finally, I want to be clear that I offered my suggestions for constructive purposes. I do not mean to come off like I’m simply “bagging” on Veeam. I would truly like to help make it better. FYI, prior to becoming a Veeam customer at the end of March, we (for many, many years) were a Symantec Backup Exec shop. I just got tired of that product. I felt like I was constantly babysitting the system, dealing with agent updates on servers, dealing with backups that failed for this reason or that reason, dealing with (IMHO) a very poor support, and simply having to work with a product that simply was not designed from the beginning to work with VMware VMs. So yes, I hope you can appreciate that I was a bit frustrated when I found myself babysitting my shiny new Veeam backup system only two weeks into using it, and I am sure that frustration came through in my post. But I intend to be a Veeam customer for the foreseeable future, so if I do comment, it is meant constructively. And do feel free to correct me when I’m wrong or misinformed. I can take it!

Thanks,
Raleigh
Raleigh
Novice
 
Posts: 7
Liked: never
Joined: Tue Jun 26, 2018 11:33 pm
Full Name: Raleigh

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Fri Jul 13, 2018 10:54 pm

Hi, Raleigh - no worries, I understand. And thank you for understanding!
Gostev
Veeam Software
 
Posts: 22400
Liked: 2675 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby JimmyO » Mon Jul 16, 2018 7:02 am

So - we have som confirmations that the latest refs.sys does the trick. Have we got some figures that indicates we´re back to same performance as before?
JimmyO
Enthusiast
 
Posts: 51
Liked: 7 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby reaperhammer » Mon Jul 16, 2018 9:23 am

When will Veeam feature RAM requirements for Refs block clone on the the official system requirements page?
reaperhammer
Enthusiast
 
Posts: 25
Liked: 7 times
Joined: Thu Aug 18, 2016 7:59 pm
Full Name: Will S

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Humphro » Mon Jul 16, 2018 10:41 am

I can confirm, for our environment, that applying KB4338814 the refs driver changed from 10.0.14393.2312 to 10.0.14393.2363. After this update was applied to both the source (Veeam server) and remote repository the time taken for full backup merge to complete dropped from over 60 hours, eventually, after a few iterations, down to less than 3 hours, which is near enough to what the job was taking before.
Humphro
Novice
 
Posts: 4
Liked: 1 time
Joined: Thu Mar 09, 2017 1:35 pm
Full Name: Matthew Humphreys

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby LBegnaud » Mon Jul 16, 2018 6:59 pm

Just throwing out our experience here. Probably not worth much without some additional info, but after fighting for 4 days I feel like sharing regardless.

We have an SoBR with 200TB+ of usable storage spread across 7 physical servers and 12 extents (we try not to have our ReFS volumes be larger than 20TB, because of issues in the past). Of these 7 servers, 6 have had their performance improved after the update. We updated because we were having issues with one of the server's performance. This server having issues is actually identical hardware-wise with one of the others in the SoBR, but it TANKED after the update. Would become unstable after ~2 hours of running small backups. Seems like a warning sign for these ReFS issues is an ever-growing value for "Modified" RAM (not sure if that was mentioned in this thread already).

Image

Anyway, Modified RAM would go higher and higher, then RPC / WMI would start failing on the repo (same old story throughout this thread). You'll notice the graph actually started going down, this is because around 2am the majority of our jobs were outright failed and past the 3 retries, so operations were mostly stopped on the offender, rs-bkptar-1.

Just replaced the newest refs.sys with refs.sys version 2097 on that single server and the server is now rock solid. Catching up from the failed backups last night at record pace. Before, we were seeing disk response times measured in seconds, now everthing is sitting pretty at <50ms. Running more concurrent jobs than we were when it would slow to a crawl.

I don't quite understand how refs.sys can be interchangeable like this, but I really hope it doesn't cause some silent corruption that pops up 3 weeks from now...
LBegnaud
Influencer
 
Posts: 10
Liked: 3 times
Joined: Wed Jan 24, 2018 12:08 am

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Mon Jul 16, 2018 9:59 pm

Did you check to see if this misbehaving server has some software installed that other servers don't? And my other guess, just by looking at server naming (likely your oldest ReFS repository), perhaps it still has some former ReFS tweaks left in the registry? I would try to reinstall Windows on that server first and foremost, as indeed something is very wrong looking at how healthy all other backup repositories are.
Gostev
Veeam Software
 
Posts: 22400
Liked: 2675 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Mgamerz » Tue Jul 17, 2018 3:33 pm

Newest server 2016 update (July 16th) now contains DHCP fix. Installing update now. I need to learn to not do this before support calls with companies working on my server though, it never works out for me doing this :)
Mgamerz
Enthusiast
 
Posts: 27
Liked: 5 times
Joined: Fri Sep 29, 2017 8:07 pm

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: nmdange, Ron_Jon and 58 guests