Current status of REFS and 9.5 update 3

[mkretzer]

Hello,

after 6 months of again and again trying to implement REFS in production the current state (since KB4025334) is usable but not perfect. Merges after 5 weekly backups still take ~8 times the amount of time they took with the first merge but are still a little bit faster than NTFS.

My question is: will there be optimisations in update 3 from Veeam side (different block clone block size or something else) to make it more stable / get more predictable performance?

Markus

[mkretzer]

Still no info about U3 & REFS?

[ptoro]

When you say merge, do you mean synthetic full backups?

If you do mean that, did you remember to create an active full after changing over to ReFS?

[tsightler]

I'm not aware of any changes coming in 9.5 U3 except for hopefully the fix for the "ERROR_BLOCK_TOO_MANY_REFERENCES" issue as this is caused by a Veeam bug/behavior.

For the other ReFS issues we are largely dependent on Microsoft for those fixes. Most customers that I've worked with are having good success with the Microsoft hotfixes + registry edits, but that is really a workaround while waiting for all of the proper fixes to get into the mainline Windows releases. Some of those fixes have made their way there, but not all of them. For now, running the hotfix + registry key changes, while having plenty of available memory, seems to be the best option.

[mkretzer]

Hello,

but none of the fixes really adress performance. We have had 0 crashes in the last 4 months, all our issues are about performance with merges and general write IO.

Markus

[tsightler]

I understand, but it doesn't change my answer. Do you have support cases open for these issues? While I've worked with some customers that saw slow merge over time, this is far from universal, and it's not currently clear to me why it happens in only some cases. I have a suspicion that it is related the the cache fix which Microsoft is still working on, as far as I know, but I don't see how anything could be changed on the Veeam side to solve this. The reason I suspect this is because, in one case, we found that if we rebooted the server on the afternoon before the merge, performance would be very fast, but would again slow over the course of weeks.

I'd be happy to look at your environment and compare to the others that I've worked with that don't have issues, see if we can find anything common. If you are interested please PM me.

[DaveWatkins]

Tom, any comments about a Veeam equivalent to Option 3 listed here
https://support.microsoft.com/en-us/hel ... erver-2016

It's something specifically for DPM but presumably applies for Veeam too

[kubimike]

I understand, but it doesn't change my answer. Do you have support cases open for these issues? While I've worked with some customers that saw slow merge over time, this is far from universal, and it's not currently clear to me why it happens in only some cases. I have a suspicion that it is related the the cache fix which Microsoft is still working on, as far as I know, but I don't see how anything could be changed on the Veeam side to solve this. The reason I suspect this is because, in one case, we found that if we rebooted the server on the afternoon before the merge, performance would be very fast, but would again slow over the course of weeks.

I'd be happy to look at your environment and compare to the others that I've worked with that don't have issues, see if we can find anything common. If you are interested please PM me.

That is the same thing I see as well!!

[kubimike]

Tom, any comments about a Veeam equivalent to Option 3 listed here

https://support.microsoft.com/en-us/hel ... erver-2016

It's something specifically for DPM but presumably applies for Veeam too

Does anyone even know what a Large duplicate extents call is ?

[Gostev]

Veeam uses large block size already, so this is DPM-specific issue. We work directly with the ReFS dev team and they never saw the issue with our block size.