ReFS strange performance issues

[dbr]

Hi guys,

we're running VB&R 9.5 U4b and installed a new repository server

- Supermicro
- Windows Server 2019 1803 Build 17763.737
- 256GB ram
- 200TB ReFS backup volume

It seems that we have perfomance issues on ReFS volume. I noticed that the server has limited throughput when reading from ReFS volume. For testing disk performance I usally use diskspd also mentioned in KB2014. If I run a 100% read diskspd on system partition (diskspd.exe -c1G -b512K -Sh -d15 c:\temp\diskspd\testfile.dat) I get a throughput around 5GB/s. Once I run the same test on the ReFS partition I get only around 150MB/s even though there are hardly any other tasks accessing the partition. The underlying storage is the same (raid 6, one virtual drive, two partitions), Defender / virus scanner is disabled, quite current build... The stranges thing is, that in rare diskspd runs throughput is near throughput on system partition without changing any parameters. Did anyone have similar strange issues regarging performance?

Thanks, Daniel.

[Gostev]

Hi, Daniel. There are known performance issues with Server 2019 LTSC which may affect other other I/O on the same volume. Thanks!

[dbr]

Hi Gostev,

I noticed that thread, but hoped that my issues are not related to that because we have performance issues even when doing simple reads (with diskspd) on ReFS. However, I'm afraid you're right and my issues are related to the known issues, too. I will subscribe that thread and will do some tests on a temporary ReFS repository.

While it seems that there are still massive problems when using ReFS, do you recommend to use Windows Server 2016 with ReFS, wait for the issues to be fixed or just stick to using ntfs?

[Gostev]

If you can live with the current performance for some time, then at this point I recommend waiting out, because Microsoft has the "final" fix in hands that (unlike the previous one) has been validated by multiple Veeam customers, so at this point it's just a matter of how fast they can take it through the servicing team to ship in a Windows update.

Windows Server 2016 ReFS has been solid for our customers.

NTFS would be my last choice.

[mkretzer]

... don't forget 1903!!! This is still extremly fast!

[Gostev]

Or 1909

[mkretzer]

@gostev with Veeam 10 yes. Will try it out soon. Do you already have any experience with 1909? Did you do any internal test with ReFS on that version? I am kind of worried that there might be a regression.
What's good is that we will have an alternative very soon

[Steve-nIP]

People have confirmed several times that 1903 and 1909 don't suffer from performance issues like 2019, but there are of course some "gotchas" with running those core-only builds that are on a different support lifecycle.

[mkretzer]

Sometimes i feel like i am one of very few 1903 users. And since 1909 is supported for 2 weeks now i find it highly unlikely that many use it already.

[poulpreben]

We have been using 1909 with both 9.5.4 (even though it isn’t officially supported) and 10. Works great.

I never had the same success with 1903 as you did, but 1909 has been flawless.

[flcap]

Do you reinstall Windows every 6-12 months when you have your repos on SAC?

[poulpreben]

SAC is supported for 24 months.

I’d say that most of our Windows estate isn’t very automated, but since SAC is Core only, it’s been a great learning platform for automating Windows configuration using PowerShell.

Even with iSCSI/MPIO setup for Direct SAN, we can perform a full reinstall and config in less than 30 minutes. So the need for periodical reinstalls isn’t really an issue.

[mkretzer]

Is there no way to inplace upgrade from 1903 to 1909?

[dbr]

Meanwhile the patch was released and I've just installed it (4531331). Even though the next synthetic will run on Saturday, I still encounter massive performance issues on when testing with diskspd and following settings as recommended by Andrew@MSFT (veeam-backup-replication-f2/windows-201 ... ml#p364022) and restarted the server afterwards:

- fsutil behavior set disabledeletenotify Refs 1
- reg add HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1

Before installing the patch I added some more disks to the server and configured them in a separate raid. At the beginning with no backups on the refs test partition I had no issues running diskspd. But with some synthetic backups on it throughput varies between full speed (5GB/s) and extremely slow (120MB/s). I haven't seen any performance issues running on drive c: (ntfs). Could any please run a diskspd with following command on a partition where already some synthetic backup reside?

As said above I tested with following syntax:

diskspd.exe -c1G -b512K -Sh -d15 t:\temp\diskspd\testfile.dat

[PeterC]

We are using a HPE Apollo 4200 (server 2019 with 1 200 TB volume as repository for Veeam backups.
We were having BIG issues with the performance during backups. When we started investigating the issue we also did some diskspd tests.

We actually were unable to complete these tests because the Apollo would completely freeze after a few minutes and we needed to perform a cold reboot to get it running again. The boot volume (NTFS) never gave any problem.
We created a case with HPE, but they did a few tests and said nothing was wrong the the server. This would only occur when we used diskspd.

Now that we have installed KB4531331 the performance issues have completely been resolved. However we never tested with diskspd again after the update so can't tell you if the same problems with diskspd would occur again.