REFS without per-VM

Availability for the Always-On Enterprise

REFS without per-VM

Veeam Logoby mkretzer » Wed Nov 09, 2016 5:51 pm

Hello,

per-VM backup files had one big advantage for us: The speed of merges and so on is much faster. But it had some major problems in 9.0 as we got problems with database deadlocks.

Now with 9.5 and REFS is per-VM for performance still necessarry?

Markus
mkretzer
Expert
 
Posts: 252
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS without per-VM

Veeam Logoby Gostev » Thu Nov 10, 2016 12:12 am

Hello,

Yes you'd still want per-VM for better performance due to multiple I/O streams with separate backup files vs. single I/O stream with a single backup file. ReFS does not change anything at all as it comes to per-VM backup file chains, it is just a different file system.

The good news is that 9.5 fixes these database issues you were facing.

Thanks!
Gostev
Veeam Software
 
Posts: 21166
Liked: 2305 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: REFS without per-VM

Veeam Logoby mkretzer » Thu Nov 10, 2016 11:24 am

Gostev,

that would be very nice... I guess we will have to test if per-VM really works, it should have been fixed with update2 and it was not.
What was changed to fix this?

Markus
mkretzer
Expert
 
Posts: 252
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS without per-VM

Veeam Logoby v.Eremin » Thu Nov 10, 2016 11:38 am

Version 9.5 will bring numerous enhancements to let Veeam Backup & Replication scale for the largest environments and guarantee efficient processing of jobs containing thousands of VMs or millions of files.

You will find more information regarding the improvements in 9.5 Release Notes, once those are published.

Thanks.
v.Eremin
Veeam Software
 
Posts: 12703
Liked: 920 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: REFS without per-VM

Veeam Logoby mkretzer » Thu Jan 26, 2017 12:43 pm

Hello,

this all sounded good in theory, but now in reality it is quite a desaster (case 02045891): Removing backups because retention is over takes 1-2 hours for every backup which is slightly bigger (~100 VMs). Whats even worse is that this process blocks other backups processes like snapshot commands to VSphere, starting backups and so on.

We are thinking about going back to non-per VM again - i just hope at some point Veeam gets this finally working...

Markus
mkretzer
Expert
 
Posts: 252
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS without per-VM

Veeam Logoby Gostev » Thu Jan 26, 2017 10:37 pm

mkretzer wrote:I just hope at some point Veeam gets this finally working

I don't think it's possible to make it working in your environment. Because if I understand debug logs correctly, your backup storage needs 1-2 seconds to delete a single file, up to 30 seconds to perform basic check for file existence etc. Sure thing, these times will add up to hours when retention policy needs to process hundreds of files! Normally, these basic file operations should take milliseconds...
Gostev
Veeam Software
 
Posts: 21166
Liked: 2305 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: REFS without per-VM

Veeam Logoby tsightler » Fri Jan 27, 2017 12:27 am

I'm curious if you are using 4K or 64K clusters on your ReFS volume? I've found that deleting files with ReFS is an "interesting" process, especially when they are sharing lots of block cloned data. Deleting a file that includes cloned blocks appears to start a background process that slowly frees the blocks and I wonder, I wonder if deleting a lot of files at once starts many of these processes. Have you noted if the disk itself is busy during this time?

Regardless, this process is MUCH slower on volumes with 4K clusters, like 16x slower (not a total surprise since there's 16x as many cluster). Note that I'm not sure that's what is causing your issue, but I've spent a lot of time trying to understand ReFS behavior in the lab and when I saw your case notes and how long deletes were taking, that's the first thing that crossed my mind because it's very similar to some behavior I was seeing. I guess I need to get my capacity lab spun back up!
tsightler
Veeam Software
 
Posts: 4687
Liked: 1698 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: REFS without per-VM

Veeam Logoby mkretzer » Fri Jan 27, 2017 7:47 am

4K. To be honest this was my theory as well. The thing is that with performance monitor from microsoft i do not have the feeling that i am seeing all the IO there is regarding to REFS operations. But on the storage side i also do not see that there is significant load at the time. Perhaps this is just sub-optimal implementation in REFS when using 4K? Sadly, Veeam support never seemed to think of a REFS problem so this is the reason i dismissed this theory... :-/

The good thing is that this is our "temporary" migration storage to REFS. So our even faster, bigger storage which we then will reformat can be formatted with 64k. I just hope we "survive" the month we have to wait until we can do this.
mkretzer
Expert
 
Posts: 252
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS without per-VM

Veeam Logoby mkretzer » Fri Jan 27, 2017 8:05 am

@gostev: I checked performance monitor again. Even under der high load of several active fulls with 16 streams and target as bottleneck the IO latency is < 100 ms. I do not get how basic checks for file existence can take 30 seconds in less-loaded situations.

Is microsoft aware that their new default setting for the filesystem causes such issues?

Edit: We also have no CPU or Memory bottleneck. CPU is at ~50 % and Memory at 71 GB of 128 Gb (SQL takes a big heap of that)
mkretzer
Expert
 
Posts: 252
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS without per-VM

Veeam Logoby mkretzer » Fri Jan 27, 2017 6:24 pm

Here is a nice screenshot while retention points are beeing deleted:

http://bayimg.com/nAIcGAAGM

The system definately could do more with its physical ressources!
mkretzer
Expert
 
Posts: 252
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], mkretzer and 16 guests