Availability for the Always-On Enterprise
Locked
Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Mar 21, 2018 5:43 pm 2 people like this post

Hi Iain,

Yes, according to the feedback so far, with the latest fixes ReFS is now usable for the majority of users. Some still have issues, which are suspected to be caused by lack of RAM on the backup repository server. You can read the detailed feedback on the last few pages of this thread.

You need to make sure you have KB4088787 installed (this update will be downloaded and installed automatically from Windows Update, but you can also download it from Microsoft Update Catalog). This is 2nd week of March update, so all the future updates will include the required fix as well.

Normally, there's no need for any reg changes.

Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!

JaySt
Service Provider
Posts: 98
Liked: 20 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by JaySt » Mar 21, 2018 8:24 pm 1 person likes this post

Gostev, all good news, but i still don't get why there's no Veeam KB describing the current status. Some kind of advise on the minimum requirement of KB4088787 and , as you mentioned, things like "use 64K for now" is good stuff to get out there if you ask me.
Veeam Certified Engineer

Iain_Green
Service Provider
Posts: 137
Liked: 8 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green » Mar 22, 2018 6:34 am

Hi Gostev,
Thanks for the confirmation, however regarding:
Gostev wrote:Some still have issues, which are suspected to be caused by lack of RAM on the backup repository server. You can read the detailed feedback on the last few pages of this thread.
I have gone back over ten pages, and unless I have missed it, there is no real specific information.

I assume the online documentation is relevant to REFS?
Hardware
CPU: x86 processor (x86-64 recommended).
Memory: 4 GB RAM, plus up to 2 GB RAM (32-bit OS) or up to 4 GB RAM (64-bit OS) for each concurrent job depending on backup chain’s length and backup files sizes. For more information, see Limitation of Concurrent Tasks.
Network: 1 Gbps or faster for on-site backup and replication, and 1 Mbps or faster for off-site backup and replication. High latency and reasonably unstable WAN links are supported.
Many thanks

Iain Green

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Mar 23, 2018 2:57 pm 1 person likes this post

JaySt wrote:Gostev, all good news, but i still don't get why there's no Veeam KB describing the current status. Some kind of advise on the minimum requirement of KB4088787 and , as you mentioned, things like "use 64K for now" is good stuff to get out there if you ask me.
Because the KB was literally just published, and we can't possibly have enough information to make any recommendations. Please don't forget we're not the vendor behind ReFS, so all we can base our recommendations on is empiric evidence. We don't have the luxury of approaching this scientifically, by looking at the ReFS code to calculate total metadata array sizes, consider cleanup algorithms peculiarities etc.

We don't ask Michelin for a recommendation on oil type for our car, just because we use their tiers, right? And yet, pressing Veeam for recommendations on Microsoft ReFS block size and memory footprint became so normal people "don't get it" why Veeam does not have the KB mere few days after the major ReFS patch :D this doesn't seem very fair to me!

Anyway, please just give us some time to collect some real-world data. There's little use of semi-scientific guesstimates anyway, which is the only thing we would be able to provide without performing long-term stress testing of the new ReFS code. Thank you!

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Mar 23, 2018 4:44 pm 1 person likes this post

My run time still looking good. I haven't updated. This veeam box is still running BETA2

Image

Giacomo_N
Enthusiast
Posts: 65
Liked: 5 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Vicenza, Italy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Giacomo_N » Mar 23, 2018 5:12 pm

Hi all, it’s time to refresh our repositories, I want to replace current two Nas with a server and windows 2016.
I’ve just read some posts, but I’m Not sure if ReFs can increase any kind of backup method or only synthetic full.
Actually I make a full during Sunday and a daily incremental, I tried in the past reverse and forever incremental, but both was slowly compare to standard incremental and we don’t have a space repository problem.
Can ReFs boost also standard incremental? And, the most important things is it now stable??
Thanks!

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Mar 23, 2018 5:43 pm

No, classic incremental backup does not benefit from ReFS block cloning, so you can use any file system at all.
As for stability, I just answered this question to someone else (see the first post on this very page). Thanks!

Giacomo_N
Enthusiast
Posts: 65
Liked: 5 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Vicenza, Italy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Giacomo_N » Mar 23, 2018 6:32 pm

Gostev wrote:No, classic incremental backup does not benefit from ReFS block cloning, so you can use any file system at all.
As for stability, I just answered this question to someone else (see the first post on this very page). Thanks!
And forever incremental?
Thank you gostev for the reply and for all, my choice to replace the Nas with a general purpose server come after reading your “last word” newsletter, it’s the only one vendor newsletter that does not trash since 2013 :D

mkaec
Expert
Posts: 277
Liked: 58 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mkaec » Mar 23, 2018 9:32 pm 1 person likes this post

I quite enjoy Gostev's newsletter. It was an unexpected perk of licensing Veeam.

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Mar 23, 2018 10:48 pm

Thank you for your kind words, guys :oops:
Giacomo_N wrote:And forever incremental?
Yes, every backup mode updates an existing full backup file, or creates new synthetic full backup files does benefits from fast cloning.

CloudMSP
Service Provider
Posts: 30
Liked: 11 times
Joined: Jul 16, 2017 5:39 am
Full Name: Veeam MSP
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by CloudMSP » Mar 24, 2018 10:45 am

Gostev wrote:Thank you for your kind words, guys :oops:
Yes, every backup mode updates an existing full backup file, or creates new synthetic full backup files does benefits from fast cloning.
Guys, I just migrated my CC Repositories, by stopping all Veeam services, moving the data to new disks, reformatting with ReFS 64KB, and moved the data back, restarted services. Everything is working, but will Veeam automatically see that it ReFS and fast clone capable now?

Also I have another thread about how I'm still spinning up Windows 2012 R2 for VM hosts due to all the snapshot problems. Should I be using ReFS on Hyper-V hosts (about 6 TB of data )? If so, does that mean I should definitely be using 2016 now? Do you know if the production snapshot issues are better now?

Thanks.

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Mar 24, 2018 4:04 pm

CloudMSP wrote:Everything is working, but will Veeam automatically see that it ReFS and fast clone capable now?
Yes. You will start seeing benefits after the next active or synthetic full backup.
Yes, the new Hyper-V 2016 snapshots are MUCH better.

mweissen13
Service Provider
Posts: 18
Liked: 7 times
Joined: Dec 28, 2017 3:22 pm
Full Name: Michael Weissenbacher
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mweissen13 » Mar 26, 2018 8:22 am 1 person likes this post

Gostev wrote: Yes. You will start seeing benefits after the next active or synthetic full backup.
Hi Gostev. We did exactly the same thing as CloudMSP and it did not work that way for us. The support told me it would work that way but it actually didn't. The solution was to create a new Repository pointing to the very same directory after moving the data back. My guess is that Veeam doesn't re-check the block alignment after the repository was created and also it will not automatically activate the associated checkbox (Align backup file data blocks) in the repository's advanced settings. See Support Case ID 02308704. But maybe this was fixed in 9.5 Update 3?

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Mar 26, 2018 9:09 am

That is correct, you cannot just edit the existing repository - you have to create a new one, even if it points to the same volume. Thanks for reminding me of this!

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Apr 06, 2018 1:39 pm

Following up with everyone, has the latest Microsoft KB solved this issue for you guys ?

Locked

Who is online

Users browsing this forum: Ildar, ivica.vujovic and 66 guests