9.5/ReFS/Server 2016 Memory Consumption

Availability for the Always-On Enterprise

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby sitruk » Wed May 24, 2017 11:23 am

graham8 wrote:You have to create the registry keys - they aren't there by default.


Using storage spaces seems to prevent memory from filling up but it writes to the disk in bursts which results in horrible write performance.
sitruk
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 01, 2016 4:54 pm
Full Name: Kurtis

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Hauke » Sat May 27, 2017 9:00 pm

Same issue here.
The storage with ReFS crashed today, not responding to connections anymore, Veeam jobs are hanging endless.
After reset of the storage it works - for a few minutes, then the memory growed to 100% usage, CPU too, and again, not responding anymore.
After a next reboot the same.
And after another reboot - now the ReFS drive is gone. Trying to access it -> freese. Opening disk management -> freese.
Other disks on the same raid are working, so this is not a drive issue.
The Windows 2016 was fully patched to the latest version.

Will now switch back to NTFS. With 2012 R2. Server 2016 is still much to buggy and not ready for production, I had a lot of other issues with it too.
Hauke
Novice
 
Posts: 8
Liked: 1 time
Joined: Thu Apr 16, 2015 11:25 am
Full Name: Hauke Ihnen

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby rkovhaev » Wed Jun 21, 2017 3:31 pm 2 people like this post

Just for visibility I am sharing details for the error that was mentioned a few times in this thread
"A file system block being referenced has already reached the maximum reference count and can't be referenced any further."

We execute DeviceIoControl() with FSCTL_DUPLICATE_EXTENTS_TO_FILE control code to clone block and Windows kernel returns to us
ERROR_BLOCK_TOO_MANY_REFERENCES (0x0000015b).
We reproduced issue without our software with a few lines of code
https://github.com/rustylife/apitest/bl ... sclone.cpp

This one looks like undocumented REFS limitation.
Case with MS support has been opened.
rkovhaev
Veeam Software
 
Posts: 33
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby dellock6 » Fri Jun 23, 2017 12:10 pm

Great finding Rustam!
Sounds like one block can only be referenced X times, did you find what's this limit? It's interesting as GFS or synthetic fulls may hit this limit.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1334 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby rkovhaev » Mon Jun 26, 2017 6:09 am

dellock6 wrote: did you find what's this limit?

no, still waiting for MS
rkovhaev
Veeam Software
 
Posts: 33
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby christopher-swe » Tue Jun 27, 2017 6:21 am

Hi,

I also have run across this problem. We have two Veeam B&R servers with two storage nodes running Windows 2016 and ReFS. Both setup have identical hardware.
I only have this problem with one job, and this job unfortunately contains a whopping 240VM's
Other jobs on the same storage have about 50VM's and I haven't seen this problem with them.

The jobs runs once per night and are set to create synthetic full every Sunday and has been running for two month now.

Can the amount of VM per job affect this?
christopher-swe
Novice
 
Posts: 5
Liked: 1 time
Joined: Wed Dec 14, 2016 6:54 am
Full Name: Christopher Svensson

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby dellock6 » Tue Jun 27, 2017 8:30 pm

I feel like no, especially with per-vm chains enabled. Each VM has its own chain, so blocks can only be cloned inside the same chain. Having 1 vm or 1000 vm should not make any difference if I'm not mistaken. It may make much more difference the retention that has been selected for a given job, and even more inside this chain how many times a block is cloned, like with synthetic fulls or GFS points.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1334 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby mkretzer » Wed Jun 28, 2017 4:21 am

@dellock6 With per-VM our deletion of restore points took hours and blocked the storage completely. Without per-VM we did not have that specific issue.
mkretzer
Expert
 
Posts: 310
Liked: 69 times
Joined: Thu Dec 17, 2015 7:17 am

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby ivordillen » Fri Jun 30, 2017 8:37 am

I have this issue on one job with a retention of 120 and I am thinking I have 110 synthetic fulls...
ivordillen
Enthusiast
 
Posts: 61
Liked: never
Joined: Thu Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Delo123 » Fri Jun 30, 2017 8:50 am 1 person likes this post

If there really is a hard limit, it would mean ReFS would pracically be useless for retention. I assume Veeam is already in contact with Microsoft regarding this?
Delo123
Expert
 
Posts: 351
Liked: 97 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby ivordillen » Fri Jun 30, 2017 1:37 pm

After a few failures I brought the retention back to 100 and all is working well.
ivordillen
Enthusiast
 
Posts: 61
Liked: never
Joined: Thu Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby rkovhaev » Thu Jul 06, 2017 1:40 pm

Per Microsoft premier support (case no 117062915966967), maximum reference count in REFS kernel module is 8175, which is really low limit
https://docs.microsoft.com/en-us/window ... ck-cloning

Chances to get ERROR_BLOCK_TOO_MANY_REFERENCES (0x0000015b) during REFS fast-clone are pretty high actually.
rkovhaev
Veeam Software
 
Posts: 33
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby nmdange » Thu Jul 06, 2017 2:15 pm 1 person likes this post

Wouldn't that mean you'd have to have do 8175 separate fast clones? In a typical once-a-day backup that'd be 8175 days or 22 years! Or can a single fast clone reference the same block multiple times? Even if you were doing a backup once per hour, you'd need to have almost a years worth of hourly backups saved. Typically, I'd expect you only keep hourly/daily backups for a month or two, and long term you'd have GFS backups monthly/quarterly/yearly which are much smaller in number.
nmdange
Expert
 
Posts: 194
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby ds2 » Thu Jul 06, 2017 2:19 pm

What will veeam do in this case? You know how offten you map this block. So you can write this block to a new location short before runing against this limit.
ds2
Enthusiast
 
Posts: 76
Liked: 16 times
Joined: Thu Jul 16, 2015 6:31 am
Full Name: Rene Keller

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby rkovhaev » Thu Jul 06, 2017 2:57 pm

per-vm backup chain and disabled "inline dedup" in job properties should help to avoid this issue.
if you are already affected by it you can always temporarily disable REFS fast-clone and let the job do merge with ReadFile() WriteFile() instead of DeviceIoControl()
rkovhaev
Veeam Software
 
Posts: 33
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Yahoo [Bot] and 20 guests