REFS issues (server lockups, high CPU, high RAM)

[Gostev]

Hi Iain,

Yes, according to the feedback so far, with the latest fixes ReFS is now usable for the majority of users. Some still have issues, which are suspected to be caused by lack of RAM on the backup repository server. You can read the detailed feedback on the last few pages of this thread.

You need to make sure you have KB4088787 installed (this update will be downloaded and installed automatically from Windows Update, but you can also download it from Microsoft Update Catalog). This is 2nd week of March update, so all the future updates will include the required fix as well.

Normally, there's no need for any reg changes.

Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!

[JaySt]

Gostev, all good news, but i still don't get why there's no Veeam KB describing the current status. Some kind of advise on the minimum requirement of KB4088787 and , as you mentioned, things like "use 64K for now" is good stuff to get out there if you ask me.

[Iain_Green]

Hi Gostev,
Thanks for the confirmation, however regarding:

Some still have issues, which are suspected to be caused by lack of RAM on the backup repository server. You can read the detailed feedback on the last few pages of this thread.

I have gone back over ten pages, and unless I have missed it, there is no real specific information.

I assume the online documentation is relevant to REFS?

Hardware
CPU: x86 processor (x86-64 recommended).
Memory: 4 GB RAM, plus up to 2 GB RAM (32-bit OS) or up to 4 GB RAM (64-bit OS) for each concurrent job depending on backup chain’s length and backup files sizes. For more information, see Limitation of Concurrent Tasks.
Network: 1 Gbps or faster for on-site backup and replication, and 1 Mbps or faster for off-site backup and replication. High latency and reasonably unstable WAN links are supported.

[Gostev]

Gostev, all good news, but i still don't get why there's no Veeam KB describing the current status. Some kind of advise on the minimum requirement of KB4088787 and , as you mentioned, things like "use 64K for now" is good stuff to get out there if you ask me.

Because the KB was literally just published, and we can't possibly have enough information to make any recommendations. Please don't forget we're not the vendor behind ReFS, so all we can base our recommendations on is empiric evidence. We don't have the luxury of approaching this scientifically, by looking at the ReFS code to calculate total metadata array sizes, consider cleanup algorithms peculiarities etc.

We don't ask Michelin for a recommendation on oil type for our car, just because we use their tiers, right? And yet, pressing Veeam for recommendations on Microsoft ReFS block size and memory footprint became so normal people "don't get it" why Veeam does not have the KB mere few days after the major ReFS patch this doesn't seem very fair to me!

Anyway, please just give us some time to collect some real-world data. There's little use of semi-scientific guesstimates anyway, which is the only thing we would be able to provide without performing long-term stress testing of the new ReFS code. Thank you!

[kubimike]

My run time still looking good. I haven't updated. This veeam box is still running BETA2

[Giacomo_N]

Hi all, it’s time to refresh our repositories, I want to replace current two Nas with a server and windows 2016.
I’ve just read some posts, but I’m Not sure if ReFs can increase any kind of backup method or only synthetic full.
Actually I make a full during Sunday and a daily incremental, I tried in the past reverse and forever incremental, but both was slowly compare to standard incremental and we don’t have a space repository problem.
Can ReFs boost also standard incremental? And, the most important things is it now stable??
Thanks!

[Gostev]

No, classic incremental backup does not benefit from ReFS block cloning, so you can use any file system at all.
As for stability, I just answered this question to someone else (see the first post on this very page). Thanks!

[Giacomo_N]

No, classic incremental backup does not benefit from ReFS block cloning, so you can use any file system at all.
As for stability, I just answered this question to someone else (see the first post on this very page). Thanks!

And forever incremental?
Thank you gostev for the reply and for all, my choice to replace the Nas with a general purpose server come after reading your “last word” newsletter, it’s the only one vendor newsletter that does not trash since 2013

[mkaec]

I quite enjoy Gostev's newsletter. It was an unexpected perk of licensing Veeam.

[Gostev]

Thank you for your kind words, guys

And forever incremental?

Yes, every backup mode updates an existing full backup file, or creates new synthetic full backup files does benefits from fast cloning.

[CloudMSP]

Thank you for your kind words, guys
Yes, every backup mode updates an existing full backup file, or creates new synthetic full backup files does benefits from fast cloning.

Guys, I just migrated my CC Repositories, by stopping all Veeam services, moving the data to new disks, reformatting with ReFS 64KB, and moved the data back, restarted services. Everything is working, but will Veeam automatically see that it ReFS and fast clone capable now?

Also I have another thread about how I'm still spinning up Windows 2012 R2 for VM hosts due to all the snapshot problems. Should I be using ReFS on Hyper-V hosts (about 6 TB of data )? If so, does that mean I should definitely be using 2016 now? Do you know if the production snapshot issues are better now?

Thanks.

[Gostev]

Everything is working, but will Veeam automatically see that it ReFS and fast clone capable now?

Yes. You will start seeing benefits after the next active or synthetic full backup.
Yes, the new Hyper-V 2016 snapshots are MUCH better.

[mweissen13]

Yes. You will start seeing benefits after the next active or synthetic full backup.

Hi Gostev. We did exactly the same thing as CloudMSP and it did not work that way for us. The support told me it would work that way but it actually didn't. The solution was to create a new Repository pointing to the very same directory after moving the data back. My guess is that Veeam doesn't re-check the block alignment after the repository was created and also it will not automatically activate the associated checkbox (Align backup file data blocks) in the repository's advanced settings. See Support Case ID 02308704. But maybe this was fixed in 9.5 Update 3?

[Gostev]

That is correct, you cannot just edit the existing repository - you have to create a new one, even if it points to the same volume. Thanks for reminding me of this!

[kubimike]

Following up with everyone, has the latest Microsoft KB solved this issue for you guys ?

[JimmyO]

I´ve only installed it on one of my repos but it works really well so I plan to reformat my other ones to ReFS in the next couple of months.

[myFist]

Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!

Hello,

is the memory optimization included in April Update KB4093119?

[stoal76]

Hi everyone,

When our Veeam backups run out of disk space on our Server 2016 REFS volume, the server stops allowing logins via RDP and on the console gets stuck at this message

Setting up personalized settings for web platform customizations

I can get to task manager on the console and see some services stopping and starting, if I try to logout it gets stuck, only option is to force the server off.

This message appears in the logs around the time after the backup runs and fulls the disk:

tcp/ip failed to establish an outgoing connection - and it looks like lots of Veeam event logs message about File is locked by 1 processes

I upgraded the server to the March cumulative update mentioned above but it still had the same problem, maybe it will be fixed in the April update.

We are only using reverse incremental backups but it does seem to be using this Fast cloning Dedupe thing as the total of the backups is more than the total disk size

Does anyone know if you can disable Veeam from using this new REFS Fast cloning feature without rolling back to NTFS??

Thanks

[Bacon]

I still have the problem on the target repo for my backpcopy job. It only happen when veeam is trying to merge the oldest restore point to the full backup file. The merge is starting, after 4-6 hours the servers loads up the cpu to 100% and is stuck. I can see it on the vmware performance overview, the cpu load goes to 100% the disk down to 0%.

I installed the April updates but no luck there.

[Gostev]

is the memory optimization included in April Update KB4093119?

I haven't heard the news but I am expecting the update we're waiting as "download only" update at first (available only via Microsoft Update Catalog). This one looks to be published on Windows Update.

[dive7]

Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?

[kubimike]

@dive7 what issue ?

[dive7]

Sorry, I should have clarified. We were seeing our repo server lock up and become unresponsive to RDP and HP ILO console. I originally typed out a post with more info but it got denied for not having a support case to reference. The engineer we worked with suggested offloading the proxy function to a different server. Previously we allowed the jobs to automatically select a proxy, and each job would auto-select the repo server as the proxy. Issue hasn't come up since the last server reboot, and we have since set some of our jobs to go to a new proxy.

We have never used 4k block size ReFS volumes, but we were seeing similiar symptoms with our 64k block size volumes (server lockups/high RAM).


case 02767982

[DonZoomik]

is the memory optimization included in April Update KB4093119?

I extracted both KB4093119 and KB4093120 and refs.sys is still at 10.14393.2097 (from February).

I don't use ReFS and therefor don't have any issues but I've been following this thread for a long time.

[Cicadymn]

Anybody seen any word on the April update that's supposed to help out a bit too?

I'm doing really good now with the previous update, but I'm almost scared to patch it and fall back into the world of ReFS horror!

[BigJack]

Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?

I am currently experiencing this issue. I have a pair of 64-disk arrays, each formatted as ReFS with 64K blocks. Fully updated, these Windows 2016 servers still suffer terrible performance with fast-clone operations. The onsite repository has enough time to perform backups with weekly synthetic fulls. The offsite repository can't process all of our backup copy jobs. The data copies from onsite to offsite quickly, but merge operations take forever. Veeam support suggested enabling "Defragment and compact full backup file" within the backup copy jobs. Not only did this not help, but it added to the copy job lengths. I don't understand how this option would benefit an ReFS volume anyway. Also, I think our off-site copy repository is in such bad shape because I believe copy jobs can only use the "Forward Incremental-Forever" storage method, which triggers a fast-clone (SLOW) operation with every backup copy instead of just during synthetic fulls. Help!

[EricJ]

Has anyone heard anything new about the ReFS memory fix due out in April?

We don't have server lockups anymore, but during nightly backup copy jobs, the filesystem gets bogged down enough that ongoing SQL transaction jobs fail. Each night we'd reduce the transfer limit on each repo, and had to crank it down to 30MB/sec to keep the jobs from failing. It's a band-aid, but backup windows are much larger and we're not getting anywhere near the max performance of the disk arrays that we paid all this money for.

Hoping the rumored "April fix" will address our issue.

Thanks,
Eric

[Gostev]

I already asked my contact for an update about a week ago, but got OOO back as he's on vacation. I will update once I hear anything back... after he's back. Thanks!

[DonZoomik]

KB4103723 has patched refs.sys to 10.0.14393.2248 (April 28th) but nothing in release notes.

[EzE]

Fully updated, these Windows 2016 servers still suffer terrible performance with fast-clone operations.

Just rebuilt my backup server and having trouble with ReFS fast-clone slowness as well. I figured with the MS patches out, I could move forward with ReFS. The server is fully up to date, and ReFS.sys file is at version 10.0.14393.2273, which comes down as part of CU KB4103720. I'm using 64k clusters as recommended, but I see terrible merge times for both local disk repo and external USB repo (temporary while changing some cloud provider stuff). Local RAID 10 repo takes about 1 hour per TB for fast clone backup chain transformation. So disappointed that even after all the fixes, I need to consider going back to NTFS before I get too far down the rabbit hole. I hate to add yet another ReFS ticket to the support queue... Anyone see performance degrade with the newer version of ReFS? Maybe MS fixed it, and then broke it again?