9.5/ReFS/Server 2016 Memory Consumption

Availability for the Always-On Enterprise

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby dellock6 » Mon Feb 06, 2017 8:31 pm

rgarvelink wrote:As stated in this thread, 4Gb of memory per core is the recommendation wouldn't OP need 64 Gb just for the Veeam operations assuming he's at 16 threads and is hitting the recommendation of 1 core for every thread? We know that ReFS prioritizes data availability over everything else and it appears to do so via memory consumption. We might just need to take that into consideration when sizing repositories.

I just wanted to highlight this part, as you are spot on and some people often overlook this part. 4GB per job (sorry my bad as I wrote core following the reply, thanks Anton) is the requested sizing for Veeam repository services, not the entire system. If you have additional services, like the ReFS components, you need to add these additional requirements into the sum of the needed memory. For 16 cores, thus handling usually 16 concurrent jobs, you need at least 64GB for just Veeam, plus the OS and its services.

Thanks for remiding this!

Luca
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 4876
Liked: 1280 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Gostev » Mon Feb 06, 2017 8:57 pm

Guys, you probably meant 4GB per concurrent job (not per core) for the repository. One core = one task (if you follow our recommendation), however a single job usually uses many tasks.
Gostev
Veeam Software
 
Posts: 21166
Liked: 2305 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby suprnova » Wed Feb 08, 2017 2:55 pm

I added a new 64k repository and we had this issue again. The VM was reset and appears to be responsive, when before it would freeze after 30 seconds.

Here's the error message from the job that was running:

Synthetic full backup creation failed Error: Agent: Failed to process method {Transform.CompileFIB}: A file system block being referenced has already reached the maximum reference count and can't be referenced any further. Failed to duplicate extent
suprnova
Service Provider
 
Posts: 8
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Gostev » Thu Feb 09, 2017 3:00 pm

The error is quite strange as it would require having around 8000 of backup files in the chain to trigger. Perhaps it happens due to system instability, rather than running into the actual BlockClone API limitation. Please let me know the support case ID for this issue, as our devs would like to collect more information from debug logs. Thanks!
Gostev
Veeam Software
 
Posts: 21166
Liked: 2305 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby matteo.tosino » Tue Mar 21, 2017 10:09 am

hi, i have the same error (the maximum reference, from the last reply of suprnova) while Copy job merge, after 2 months of backup chain (7 Days, 4 Weeks, 3 Months, of 5 vms in one backup job).
This is the second time: the first time i open a case but with no solution
The support try to:
1. remove configuration of the copy job, and import again creaeting a new backup repository in a new location, impossible in my case (i have only one volume; i have try within the same volume, the error remain)
2. a volume defrag, but the defrag activity hung after few minutes with no disk or CPU activity).
3. refer to windows error code 0xC000048C STATUS_BLOCK_TOO_MANY_REFERENCES, but there is no documentation
I have closed the ticket because i read (don't remember where, sorry) to use ReFS with Storage Space as backup repository for Veeam. I try it for some days, but it's too slow respect of an hardware (hp smart array) management of my backup storage (JBOD).

Now i have again the same problem, what do you suggest?
I'm sure is a ReFS related problem, because it's born with this change in my infrastructure. Before to come back to NTFS, do you know some trick to solve this problem?
matteo.tosino
Novice
 
Posts: 3
Liked: never
Joined: Tue Mar 21, 2017 9:52 am
Full Name: Matteo Tosino

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby kubimike » Tue Mar 21, 2017 2:14 pm

@matteo.tosino can I chime in and ask if you've tried doing an active-full on the job to see if it goes away ?
kubimike
Expert
 
Posts: 143
Liked: 20 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby matteo.tosino » Mon Mar 27, 2017 8:24 am

good idea! i have started an active full just now, i'll report here
thanks
matteo.tosino
Novice
 
Posts: 3
Liked: never
Joined: Tue Mar 21, 2017 9:52 am
Full Name: Matteo Tosino

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby mickyv » Tue Apr 04, 2017 11:48 pm

Hey matteo.tosino, did this issue go away for you?
mickyv
Novice
 
Posts: 9
Liked: never
Joined: Tue Apr 04, 2017 2:38 am
Location: Adelaide, Australia
Full Name: Michael V

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby matteo.tosino » Mon Apr 10, 2017 8:32 am

no, i gave up and i went back to NTFS :(
matteo.tosino
Novice
 
Posts: 3
Liked: never
Joined: Tue Mar 21, 2017 9:52 am
Full Name: Matteo Tosino

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Ctek » Wed Apr 26, 2017 8:35 pm

suprnova wrote:I added a new 64k repository and we had this issue again. The VM was reset and appears to be responsive, when before it would freeze after 30 seconds.

Here's the error message from the job that was running:

Synthetic full backup creation failed Error: Agent: Failed to process method {Transform.CompileFIB}: A file system block being referenced has already reached the maximum reference count and can't be referenced any further. Failed to duplicate extent


This exact error message issue started happening in our test environment overtime. ReFS 64K volume.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 36
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby sitruk » Sun May 21, 2017 2:22 pm

I am so happy I found this thread. I have been battling this issue for a couple weeks and it took me this long to narrow down the cause. I even reimaged my physical server thinking Windows was corrupt.

Physical Server
Win Server 2016 x64
Intel Xeon Quad Core
32GB RAM
2x ReFS Repositories 10TB and 20TB both formatted with 64K cluster
Added to virtual Veeam B&R 6.5 update 2 server as a proxy

I have one job with all my VMs where most are very small. I ordered the VMs to backup smallest first and they all complete successfully. Performance is great, but once I get to the large VM the RAM is slowly consumed until full. Then the physical server locks up due to a lack of resources. The job fails and the RAM is left consumed. I can sometimes remote in and reboot the server to reclaim the RAM.

Windows recently released a fix for ReFS memory usage. https://support.microsoft.com/en-us/hel ... windows-10

Unfortunately, when I attempt to install this fix it says it is not applicable to my computer. I am leaning more towards formatting my repositories with NTFS and calling it a day.
sitruk
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 01, 2016 4:54 pm
Full Name: Kurtis

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby graham8 » Mon May 22, 2017 2:22 pm

sitruk wrote:once I get to the large VM the RAM is slowly consumed until full. Then the physical server locks up due to a lack of resources. The job fails and the RAM is left consumed. I can sometimes remote in and reboot the server to reclaim the RAM.

Well, if you look over at veeam-backup-replication-f2/refs-4k-horror-story-t40629.html you'll find that plenty have had this issue with the default of 4k. Apparently, from comments, 64k has been an issue as well - just much less frequently. What you're describing is definitely what I was seeing with 4k. What's the size of your largest job?

sitruk wrote:Windows recently released a fix for ReFS memory usage. https://support.microsoft.com/en-us/hel ... windows-10 .. Unfortunately, when I attempt to install this fix it says it is not applicable to my computer.

Odd about not being applicable. YMMV, but in our case, even with all the options enabled for us at their most strict settings, the problem still occurred.

sitruk wrote:I am leaning more towards formatting my repositories with NTFS and calling it a day.

I've had a MS case open about the issue for months now. I got a bunch of manually-initiated (when the server locks due to resource consumption) memory dumps sent to them, and then after that point I had to reload with something that I could rely on since this wasn't a test server. It's been about a month, I think, since I got the memory dumps sent to MS. I got a reply after a few weeks that they had analyzed them, saying that they think they knew what is going on, and that they wanted to do testing on our server. I had to point out that this is a production server and that Microsoft needs to do their testing internally. I asked what the cause was, though, because we have other ReFS+2016+Veeam servers (though those have more ram and have been more stable), but I haven't gotten a reply after asking a few times.

So...if you can deal with the backup space inflation of NTFS, then yes, I think that's probably wise. They may fix the issue "soon", but I personally wouldn't want to bet on it.
graham8
Enthusiast
 
Posts: 54
Liked: 20 times
Joined: Wed Dec 14, 2016 1:56 pm

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby sitruk » Tue May 23, 2017 11:05 pm

graham8 wrote:Well, if you look over at veeam-backup-replication-f2/refs-4k-horror-story-t40629.html you'll find that plenty have had this issue with the default of 4k. Apparently, from comments, 64k has been an issue as well - just much less frequently. What you're describing is definitely what I was seeing with 4k. What's the size of your largest job?

Odd about not being applicable. YMMV, but in our case, even with all the options enabled for us at their most strict settings, the problem still occurred.

I've had a MS case open about the issue for months now. I got a bunch of manually-initiated (when the server locks due to resource consumption) memory dumps sent to them, and then after that point I had to reload with something that I could rely on since this wasn't a test server. It's been about a month, I think, since I got the memory dumps sent to MS. I got a reply after a few weeks that they had analyzed them, saying that they think they knew what is going on, and that they wanted to do testing on our server. I had to point out that this is a production server and that Microsoft needs to do their testing internally. I asked what the cause was, though, because we have other ReFS+2016+Veeam servers (though those have more ram and have been more stable), but I haven't gotten a reply after asking a few times.

So...if you can deal with the backup space inflation of NTFS, then yes, I think that's probably wise. They may fix the issue "soon", but I personally wouldn't want to bet on it.

My largest job is almost 12TB but the RAM issue occurs on a 1TB backup. I think that update isn't applicable because it was superseded by another one. However, the registry keys mentioned in the KB are not present on my server. Were you able to try changing those REG keys as per the KB? Also, do you know if the ReFS storage needs to be configured within Storage Spaces for it to function properly?

The space savings seem to be significant as well as the incremental full performance. So getting ReFS to work would be pretty awesome.
sitruk
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 01, 2016 4:54 pm
Full Name: Kurtis

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby graham8 » Wed May 24, 2017 1:28 am

You have to create the registry keys - they aren't there by default.
graham8
Enthusiast
 
Posts: 54
Liked: 20 times
Joined: Wed Dec 14, 2016 1:56 pm

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby sitruk » Wed May 24, 2017 1:57 am

I realized that after looking over the page again. I actually ended up passing my disks through and I'm giving Storage Spaces a shot.
sitruk
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 01, 2016 4:54 pm
Full Name: Kurtis

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], EIvanov, Google [Bot] and 18 guests