Comprehensive data protection for all workloads
Post Reply
dbr
Enthusiast
Posts: 77
Liked: 5 times
Joined: Apr 06, 2017 9:48 am
Full Name: Daniel Brase
Contact:

ReFS strange performance issues

Post by dbr »

Hi guys,

we're running VB&R 9.5 U4b and installed a new repository server

- Supermicro
- Windows Server 2019 1803 Build 17763.737
- 256GB ram
- 200TB ReFS backup volume

It seems that we have perfomance issues on ReFS volume. I noticed that the server has limited throughput when reading from ReFS volume. For testing disk performance I usally use diskspd also mentioned in KB2014. If I run a 100% read diskspd on system partition (diskspd.exe -c1G -b512K -Sh -d15 c:\temp\diskspd\testfile.dat) I get a throughput around 5GB/s. Once I run the same test on the ReFS partition I get only around 150MB/s even though there are hardly any other tasks accessing the partition. The underlying storage is the same (raid 6, one virtual drive, two partitions), Defender / virus scanner is disabled, quite current build... The stranges thing is, that in rare diskspd runs throughput is near throughput on system partition without changing any parameters. Did anyone have similar strange issues regarging performance?

Thanks, Daniel.

Gostev
SVP, Product Management
Posts: 26917
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS strange performance issues

Post by Gostev »

Hi, Daniel. There are known performance issues with Server 2019 LTSC which may affect other other I/O on the same volume. Thanks!

dbr
Enthusiast
Posts: 77
Liked: 5 times
Joined: Apr 06, 2017 9:48 am
Full Name: Daniel Brase
Contact:

Re: ReFS strange performance issues

Post by dbr »

Hi Gostev,

I noticed that thread, but hoped that my issues are not related to that because we have performance issues even when doing simple reads (with diskspd) on ReFS. However, I'm afraid you're right and my issues are related to the known issues, too. I will subscribe that thread and will do some tests on a temporary ReFS repository.

While it seems that there are still massive problems when using ReFS, do you recommend to use Windows Server 2016 with ReFS, wait for the issues to be fixed or just stick to using ntfs?

Gostev
SVP, Product Management
Posts: 26917
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS strange performance issues

Post by Gostev »

If you can live with the current performance for some time, then at this point I recommend waiting out, because Microsoft has the "final" fix in hands that (unlike the previous one) has been validated by multiple Veeam customers, so at this point it's just a matter of how fast they can take it through the servicing team to ship in a Windows update.

Windows Server 2016 ReFS has been solid for our customers.

NTFS would be my last choice.

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: ReFS strange performance issues

Post by mkretzer »

... don't forget 1903!!! This is still extremly fast!

Gostev
SVP, Product Management
Posts: 26917
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS strange performance issues

Post by Gostev »

Or 1909 :D

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: ReFS strange performance issues

Post by mkretzer »

@gostev with Veeam 10 yes. Will try it out soon. Do you already have any experience with 1909? Did you do any internal test with ReFS on that version? I am kind of worried that there might be a regression.
What's good is that we will have an alternative very soon :-)

Steve-nIP
Service Provider
Posts: 73
Liked: 21 times
Joined: Feb 06, 2018 10:08 am
Full Name: Steve
Contact:

Re: ReFS strange performance issues

Post by Steve-nIP » 1 person likes this post

People have confirmed several times that 1903 and 1909 don't suffer from performance issues like 2019, but there are of course some "gotchas" with running those core-only builds that are on a different support lifecycle.

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: ReFS strange performance issues

Post by mkretzer »

Sometimes i feel like i am one of very few 1903 users. And since 1909 is supported for 2 weeks now i find it highly unlikely that many use it already.

poulpreben
Veeam Vanguard
Posts: 1009
Liked: 433 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: ReFS strange performance issues

Post by poulpreben » 1 person likes this post

We have been using 1909 with both 9.5.4 (even though it isn’t officially supported) and 10. Works great.

I never had the same success with 1903 as you did, but 1909 has been flawless.

flcap
Lurker
Posts: 2
Liked: never
Joined: Feb 06, 2020 9:25 am
Full Name: Alex Petersen
Contact:

Re: ReFS strange performance issues

Post by flcap »

Do you reinstall Windows every 6-12 months when you have your repos on SAC?

poulpreben
Veeam Vanguard
Posts: 1009
Liked: 433 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: ReFS strange performance issues

Post by poulpreben »

SAC is supported for 24 months.

I’d say that most of our Windows estate isn’t very automated, but since SAC is Core only, it’s been a great learning platform for automating Windows configuration using PowerShell.

Even with iSCSI/MPIO setup for Direct SAN, we can perform a full reinstall and config in less than 30 minutes. So the need for periodical reinstalls isn’t really an issue.

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: ReFS strange performance issues

Post by mkretzer »

Is there no way to inplace upgrade from 1903 to 1909?

dbr
Enthusiast
Posts: 77
Liked: 5 times
Joined: Apr 06, 2017 9:48 am
Full Name: Daniel Brase
Contact:

Re: ReFS strange performance issues

Post by dbr »

Meanwhile the patch was released and I've just installed it (4531331). Even though the next synthetic will run on Saturday, I still encounter massive performance issues on when testing with diskspd and following settings as recommended by Andrew@MSFT (veeam-backup-replication-f2/windows-201 ... ml#p364022) and restarted the server afterwards:

- fsutil behavior set disabledeletenotify Refs 1
- reg add HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1

Before installing the patch I added some more disks to the server and configured them in a separate raid. At the beginning with no backups on the refs test partition I had no issues running diskspd. But with some synthetic backups on it throughput varies between full speed (5GB/s) and extremely slow (120MB/s). I haven't seen any performance issues running on drive c: (ntfs). Could any please run a diskspd with following command on a partition where already some synthetic backup reside?

As said above I tested with following syntax:

diskspd.exe -c1G -b512K -Sh -d15 t:\temp\diskspd\testfile.dat

PeterC
Enthusiast
Posts: 25
Liked: 7 times
Joined: Apr 10, 2018 2:24 pm
Full Name: Peter Camps
Contact:

Re: ReFS strange performance issues

Post by PeterC » 1 person likes this post

We are using a HPE Apollo 4200 (server 2019 with 1 200 TB volume as repository for Veeam backups.
We were having BIG issues with the performance during backups. When we started investigating the issue we also did some diskspd tests.

We actually were unable to complete these tests because the Apollo would completely freeze after a few minutes and we needed to perform a cold reboot to get it running again. The boot volume (NTFS) never gave any problem.
We created a case with HPE, but they did a few tests and said nothing was wrong the the server. This would only occur when we used diskspd.

Now that we have installed KB4531331 the performance issues have completely been resolved. However we never tested with diskspd again after the update so can't tell you if the same problems with diskspd would occur again.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], darmarko, smirza and 57 guests