REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Wed Mar 21, 2018 5:43 pm 2 people like this post

Hi Iain,

Yes, according to the feedback so far, with the latest fixes ReFS is now usable for the majority of users. Some still have issues, which are suspected to be caused by lack of RAM on the backup repository server. You can read the detailed feedback on the last few pages of this thread.

You need to make sure you have KB4088787 installed (this update will be downloaded and installed automatically from Windows Update, but you can also download it from Microsoft Update Catalog). This is 2nd week of March update, so all the future updates will include the required fix as well.

Normally, there's no need for any reg changes.

Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby JaySt » Wed Mar 21, 2018 8:24 pm 1 person likes this post

Gostev, all good news, but i still don't get why there's no Veeam KB describing the current status. Some kind of advise on the minimum requirement of KB4088787 and , as you mentioned, things like "use 64K for now" is good stuff to get out there if you ask me.
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 77
Liked: 17 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Iain_Green » Thu Mar 22, 2018 6:34 am

Hi Gostev,
Thanks for the confirmation, however regarding:
Gostev wrote:Some still have issues, which are suspected to be caused by lack of RAM on the backup repository server. You can read the detailed feedback on the last few pages of this thread.

I have gone back over ten pages, and unless I have missed it, there is no real specific information.

I assume the online documentation is relevant to REFS?

Hardware
CPU: x86 processor (x86-64 recommended).
Memory: 4 GB RAM, plus up to 2 GB RAM (32-bit OS) or up to 4 GB RAM (64-bit OS) for each concurrent job depending on backup chain’s length and backup files sizes. For more information, see Limitation of Concurrent Tasks.
Network: 1 Gbps or faster for on-site backup and replication, and 1 Mbps or faster for off-site backup and replication. High latency and reasonably unstable WAN links are supported.
Many thanks

Iain Green
Iain_Green
Service Provider
 
Posts: 107
Liked: 6 times
Joined: Fri Dec 05, 2014 2:13 pm
Full Name: Iain Green

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Fri Mar 23, 2018 2:57 pm 1 person likes this post

JaySt wrote:Gostev, all good news, but i still don't get why there's no Veeam KB describing the current status. Some kind of advise on the minimum requirement of KB4088787 and , as you mentioned, things like "use 64K for now" is good stuff to get out there if you ask me.

Because the KB was literally just published, and we can't possibly have enough information to make any recommendations. Please don't forget we're not the vendor behind ReFS, so all we can base our recommendations on is empiric evidence. We don't have the luxury of approaching this scientifically, by looking at the ReFS code to calculate total metadata array sizes, consider cleanup algorithms peculiarities etc.

We don't ask Michelin for a recommendation on oil type for our car, just because we use their tiers, right? And yet, pressing Veeam for recommendations on Microsoft ReFS block size and memory footprint became so normal people "don't get it" why Veeam does not have the KB mere few days after the major ReFS patch :D this doesn't seem very fair to me!

Anyway, please just give us some time to collect some real-world data. There's little use of semi-scientific guesstimates anyway, which is the only thing we would be able to provide without performing long-term stress testing of the new ReFS code. Thank you!
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Fri Mar 23, 2018 4:44 pm 1 person likes this post

My run time still looking good. I haven't updated. This veeam box is still running BETA2

Image
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Giacomo_N » Fri Mar 23, 2018 5:12 pm

Hi all, it’s time to refresh our repositories, I want to replace current two Nas with a server and windows 2016.
I’ve just read some posts, but I’m Not sure if ReFs can increase any kind of backup method or only synthetic full.
Actually I make a full during Sunday and a daily incremental, I tried in the past reverse and forever incremental, but both was slowly compare to standard incremental and we don’t have a space repository problem.
Can ReFs boost also standard incremental? And, the most important things is it now stable??
Thanks!
Giacomo_N
Enthusiast
 
Posts: 53
Liked: 5 times
Joined: Fri Feb 15, 2013 1:56 pm
Location: Vicenza, Italy
Full Name: Giacomo

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Fri Mar 23, 2018 5:43 pm

No, classic incremental backup does not benefit from ReFS block cloning, so you can use any file system at all.
As for stability, I just answered this question to someone else (see the first post on this very page). Thanks!
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Giacomo_N » Fri Mar 23, 2018 6:32 pm

Gostev wrote:No, classic incremental backup does not benefit from ReFS block cloning, so you can use any file system at all.
As for stability, I just answered this question to someone else (see the first post on this very page). Thanks!


And forever incremental?
Thank you gostev for the reply and for all, my choice to replace the Nas with a general purpose server come after reading your “last word” newsletter, it’s the only one vendor newsletter that does not trash since 2013 :D
Giacomo_N
Enthusiast
 
Posts: 53
Liked: 5 times
Joined: Fri Feb 15, 2013 1:56 pm
Location: Vicenza, Italy
Full Name: Giacomo

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby mkaec » Fri Mar 23, 2018 9:32 pm 1 person likes this post

I quite enjoy Gostev's newsletter. It was an unexpected perk of licensing Veeam.
mkaec
Expert
 
Posts: 217
Liked: 49 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Fri Mar 23, 2018 10:48 pm

Thank you for your kind words, guys :oops:

Giacomo_N wrote:And forever incremental?

Yes, every backup mode updates an existing full backup file, or creates new synthetic full backup files does benefits from fast cloning.
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby CloudMSP » Sat Mar 24, 2018 10:45 am

Gostev wrote:Thank you for your kind words, guys :oops:
Yes, every backup mode updates an existing full backup file, or creates new synthetic full backup files does benefits from fast cloning.


Guys, I just migrated my CC Repositories, by stopping all Veeam services, moving the data to new disks, reformatting with ReFS 64KB, and moved the data back, restarted services. Everything is working, but will Veeam automatically see that it ReFS and fast clone capable now?

Also I have another thread about how I'm still spinning up Windows 2012 R2 for VM hosts due to all the snapshot problems. Should I be using ReFS on Hyper-V hosts (about 6 TB of data )? If so, does that mean I should definitely be using 2016 now? Do you know if the production snapshot issues are better now?

Thanks.
CloudMSP
Service Provider
 
Posts: 26
Liked: 10 times
Joined: Sun Jul 16, 2017 5:39 am
Full Name: Veeam MSP

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Sat Mar 24, 2018 4:04 pm

CloudMSP wrote:Everything is working, but will Veeam automatically see that it ReFS and fast clone capable now?

Yes. You will start seeing benefits after the next active or synthetic full backup.
Yes, the new Hyper-V 2016 snapshots are MUCH better.
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby mweissen13 » Mon Mar 26, 2018 8:22 am 1 person likes this post

Gostev wrote:Yes. You will start seeing benefits after the next active or synthetic full backup.

Hi Gostev. We did exactly the same thing as CloudMSP and it did not work that way for us. The support told me it would work that way but it actually didn't. The solution was to create a new Repository pointing to the very same directory after moving the data back. My guess is that Veeam doesn't re-check the block alignment after the repository was created and also it will not automatically activate the associated checkbox (Align backup file data blocks) in the repository's advanced settings. See Support Case ID 02308704. But maybe this was fixed in 9.5 Update 3?
mweissen13
Service Provider
 
Posts: 11
Liked: 3 times
Joined: Thu Dec 28, 2017 3:22 pm
Full Name: Michael Weissenbacher

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Mon Mar 26, 2018 9:09 am

That is correct, you cannot just edit the existing repository - you have to create a new one, even if it points to the same volume. Thanks for reminding me of this!
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Fri Apr 06, 2018 1:39 pm

Following up with everyone, has the latest Microsoft KB solved this issue for you guys ?
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 43 guests