REFS without per-VM

[mkretzer]

Hello,

per-VM backup files had one big advantage for us: The speed of merges and so on is much faster. But it had some major problems in 9.0 as we got problems with database deadlocks.

Now with 9.5 and REFS is per-VM for performance still necessarry?

Markus

[Gostev]

Hello,

Yes you'd still want per-VM for better performance due to multiple I/O streams with separate backup files vs. single I/O stream with a single backup file. ReFS does not change anything at all as it comes to per-VM backup file chains, it is just a different file system.

The good news is that 9.5 fixes these database issues you were facing.

Thanks!

[mkretzer]

Gostev,

that would be very nice... I guess we will have to test if per-VM really works, it should have been fixed with update2 and it was not.
What was changed to fix this?

Markus

[veremin]

Version 9.5 will bring numerous enhancements to let Veeam Backup & Replication scale for the largest environments and guarantee efficient processing of jobs containing thousands of VMs or millions of files.

You will find more information regarding the improvements in 9.5 Release Notes, once those are published.

Thanks.

[mkretzer]

Hello,

this all sounded good in theory, but now in reality it is quite a desaster (case 02045891): Removing backups because retention is over takes 1-2 hours for every backup which is slightly bigger (~100 VMs). Whats even worse is that this process blocks other backups processes like snapshot commands to VSphere, starting backups and so on.

We are thinking about going back to non-per VM again - i just hope at some point Veeam gets this finally working...

Markus

[Gostev]

I just hope at some point Veeam gets this finally working

I don't think it's possible to make it working in your environment. Because if I understand debug logs correctly, your backup storage needs 1-2 seconds to delete a single file, up to 30 seconds to perform basic check for file existence etc. Sure thing, these times will add up to hours when retention policy needs to process hundreds of files! Normally, these basic file operations should take milliseconds...

[tsightler]

I'm curious if you are using 4K or 64K clusters on your ReFS volume? I've found that deleting files with ReFS is an "interesting" process, especially when they are sharing lots of block cloned data. Deleting a file that includes cloned blocks appears to start a background process that slowly frees the blocks and I wonder, I wonder if deleting a lot of files at once starts many of these processes. Have you noted if the disk itself is busy during this time?

Regardless, this process is MUCH slower on volumes with 4K clusters, like 16x slower (not a total surprise since there's 16x as many cluster). Note that I'm not sure that's what is causing your issue, but I've spent a lot of time trying to understand ReFS behavior in the lab and when I saw your case notes and how long deletes were taking, that's the first thing that crossed my mind because it's very similar to some behavior I was seeing. I guess I need to get my capacity lab spun back up!

[mkretzer]

4K. To be honest this was my theory as well. The thing is that with performance monitor from microsoft i do not have the feeling that i am seeing all the IO there is regarding to REFS operations. But on the storage side i also do not see that there is significant load at the time. Perhaps this is just sub-optimal implementation in REFS when using 4K? Sadly, Veeam support never seemed to think of a REFS problem so this is the reason i dismissed this theory... :-/

The good thing is that this is our "temporary" migration storage to REFS. So our even faster, bigger storage which we then will reformat can be formatted with 64k. I just hope we "survive" the month we have to wait until we can do this.

[mkretzer]

@gostev: I checked performance monitor again. Even under der high load of several active fulls with 16 streams and target as bottleneck the IO latency is < 100 ms. I do not get how basic checks for file existence can take 30 seconds in less-loaded situations.

Is microsoft aware that their new default setting for the filesystem causes such issues?

Edit: We also have no CPU or Memory bottleneck. CPU is at ~50 % and Memory at 71 GB of 128 Gb (SQL takes a big heap of that)

[mkretzer]

Here is a nice screenshot while retention points are beeing deleted:

http://bayimg.com/nAIcGAAGM

The system definately could do more with its physical ressources!

[EzE]

mkretzer, Would you be able to update on your scenario? Were you able to fix the slowness with 64k clusters? Did you notice more space was needed since Veeam cannot dedup across the per-vm backup files?

[mkretzer]

Hello,

we gave up on REFS. And from what i read in the other posts it was the best thing we can do.

Now with per-VM and NTFS it works perfectly in 9.5 U2, we have no deadlock issues anymore!

Markus