Windows 2019, large REFS and deletes

[yara12]

However, trim/unmap have effect only on thin provisioned storages... I think.

[mkretzer]

Which we do have. This might at least help a bit. But every little thing helps in our situation.

[soehl]

.... not too big volumes (the 600 Tb we have seem to be a little bit too much) was their recommendation for now.

Which size is the MS recommendation?!

Thanks!

[mkretzer]

There was no specific recommendation as they are just building their own bigger test systems right now. For us we will set the new limit at ~150 TB.

All the problems should happen per-volumes and not server-global.

But this seem to have one other effect: Right now we have our new system with windows server 1903 and we did first tests. If the backend storage is not fast enough REFS memory usage does not only go to ~300 GB but ~600 GB with two REFS volumes.

So more REFS volumes might benefit from (i am not sure about "need") more RAM.

[Gostev]

[trim/unmap have effect only on thin provisioned storages] Which we do have.

Just wanted to note that Microsoft explicitly does not support thin provisioning on the storage device layer with ReFS.

[mkretzer]

How is the storage being thin provisioned or not relevat to the filesystem? It should have no effect on the filesystem...

Do you have a link to that information?

[Gostev]

https://docs.microsoft.com/en-us/window ... s-overview

Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

[bretesq]

We just received new hardware and will switch to 1903 Server to test if that helps. I will keep this thread updated but expect results in 4-6 weeks (after first deletes).

Hey Mike,

Are you having any better luck with 1903? Hows the experience been thus far?

Thanks

[mkretzer]

Hello,

its Markus
It gets interesting next week. Then we will have the first deletions.

[bretesq]

Sounds great. Look forward to hearing about it. Sorry about that. I don't know how I even got Mike

[EvoGeek]

Hi Markus:

How goes your system after the deletes?

Also, regarding version 1903. Since you mentioned being LTSC which is based off 1809, how did you go about getting 1903? Did you have to switch to Semi-Annual Channel (SAC)?

[mkretzer]

So far so good. Fírst deletes are through but those are rather small. But i am quite optimistic!

In 1-2 weeks the real big deletes are coming.

We completely reinstalled windows server core 1903.

[YoMarK]

Subscribed to this topic.
Just installed three 60TB repo's(one 180TB scale out repo) based on Windows 2019 with ReFS(64k blocks), and am not really looking forward to ReFS issues again.

I'm not sure what would be the benefit of using Windows server 1903(for Veeam/RefS) compared to Windows 2016 given microsoft support cycles.

[mkretzer]

According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.

We were in direct contact with the REFS devs...

If the issue is still there we should have felt at least some issues by now. As I said I am quite optimistic for the first time in 2 years.

[rhys.hammond]

We are experiencing similar issues with ReFS + large deletions causing severe performance degradation on our win server 2019 servers (build 1809) repository servers.
Synthetic Full operations that were previously completed in under 10 hours are now taking 48+ hours or more. Also seeing REFS writes from jobs hang, then transfer data, hang again, after restore points are deleted.

We're currently on 1809 so we are now considering upgrading to 1903.

For those curious, the specs of our repository servers are as follows,
-Windows Server 2019 (version 1809)
-dual socket Intel Xeon-Gold 6130 (2.1Ghz 16 core)
-128GB DDR4
-2 x 480GB SSD for running Windows
- HPE Smart Array P408i-p SR Gen 10 Raid Controller with 2GB Cache
-26 x 8TB SATA 6Gbps 7.2K rpm 3.5”
The filesystem is ReFS with 64K cluster size

Regarding upgrading from 1809 to 1903,
-If you're on CB (SAC), it should be an in-place upgrade
-If you're on LTSC, to go to 1903 it will be a wipe / reload.

btw, our Veeam support case currently open for this issue is, 03699027.

I'm curious to understand if there is a 'quality update / monthly cumulative' update for 1809 to remediate this fix or does this fix rely on new functionality only available in the feature update (1903).
Are we required to reformat the ReFS volume for the fix to work?
I'm also very keen to hear more from mkretzer, if the ReFS deletion issue has well and truly been put to rest.

[YoMarK]

According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.

I'm not sure if this is a reply to my post, but what I meant to ask is what the advantage of 1903 is compared to Windows 2016 specifically for Veeam.
If Windows 2019 has issues, then it's not a option right now. But 2016 still is, and is supported longer then 1903 then why use the Semi-Annual Channel releases(some consider those the public beta testing releases of windows server).
1903 ReFS will support Dedub and compression as well, but given the way it works, that is not really a killer feature.

As Windows 2016 has support until 12 January 2027(much longer then all Semi-Annual Channel releases like 1903), I would think it would be the more obvious choice for something "simple"(should just "work") like a fileserver.

[mkretzer]

@rhys.hammond Right now there is no update for 2019/1809 as far as i understand microsoft. They told me newest REFS driver builds should work with 2019/1809 but then this is no "fully supported" solution anymore.

I would gladly give you a "all is fixed" already but we are very caucious with such a statement before we have a full month of backups and the bigger backups get deleted. In the last 2 years we had many situation where we believed "problem is finally fixed" and the whole thing came down one week later.

But besides the first successfull smaller deletes there is one other good thing with 1903: Synthetics after 3 weeks were just as fast as in the first week. That was one thing that was always an issue: synthetics got slower the more data was on the repo!

[rhys.hammond]

That sounds promising mkretzer, thank you for the update.

If you happen to speak to the Microsoft devs again could you possibly confirm if we need to format ReFS volume containing veeam backups files when moving from 1809 to 1903, specifically during a wipe/reload as required for LTSC to CB (SAC) please? Given the mention of the architecture changes to ReFS from 1809 to 1903 I thought it would be prudent to confirm this.

[mkretzer]

I can ask but i am also 95% certain you don't need to reformat as they offered to replace the driver in 2019/1809 with their dev driver which is even newer - without reformat.

[rhys.hammond]

Hey mkretzer,

That's certainly good news regarding the reformat likely not being necessary for 1809 to 1903.
Regarding this dev driver, iI'm guessing this is not available to the general public?

[mkretzer]

No it is not. Also they told me that there is a certain risk involved.

If you want to hear my take on it: Try the 1903 driver in 2019. It should work and then at least you have a driver which is in itself fit for production (but don't ask me to fix it if all goes wrong )

[nikpolini]

Hi All
I see the update "August 17, 2019—KB4512534 (OS Build 17763.720)" the refs.dll has changed.
For 1809 Windows Server 2019 that is.
No mention in release notes though...
Nik

[mkretzer]

To be honest thats the most confusing about the whole thing: the refs.sys from 1903 was not updates once since release.
And the 2019 version is all the time.

Still, i was told that 1903 version is superior to the 2019 version...

[rhys.hammond]

I've been making some tweaks in preparation for the weekend when our performance appears to nosedive off a cliff.

-Uninstalled SCCM client as per https://www.veeam.com/kb2792 (the GUI was bugged and refusing to start which wasn't a good sign)
-very small tweak to our raid cache policy as per 'HPE Reference Architecture for Veeam Availability Suite with HPE Apollo backup target' document
-removed concurrent task limit on the backup repository as per 'HPE Reference Architecture for Veeam Availability Suite with HPE Apollo backup target' document
-applied all windows updates available and restarted (we were on vanilla 2019 with zero updates)

Interesting that HPE is recommending Windows Server 2019 in their Veeam Apollo Repository Server Reference Architecture as well.

Now we need to wait for the weekend to see if how it goes... fingers crossed.

[mkretzer]

Yesterday we had our first set of large deletes. The best thing: Memory usage did not increase at all. With 2019 memory usage fluctuated heavily when there was something deleted. Also there was ZERO impact to other backups.
The next 2-3 days will get VERY interesting as now the HUGE deletes are coming!

[mkretzer]

Ok. In the last 2 days there where ~10000 restore points being deleted.

And it seems without any impact AT ALL!

I will ask MS support if there is any hope for creating a backport....

[rhys.hammond]

So I'm happy to report that our poor performance issue did not reoccur on the weekend. Synthetic fulls / GFS took the usual few hours to complete, not days to complete.
Whether it was the "August 17, 2019—KB4512534 (OS Build 17763.720)" update resolved the issue or perhaps just the removal of the Microsoft Config Manager client we're not quite sure.
We ended up implementing quite a few improvements to try and solve these 'deletion' triggered performance issues so hard to pinpoint the exact fix.

We are now reinstalling Microsoft Config Manager client and will report back after next weekend passes, fingers crossed the issue doesn't appear so we can stick on build 1807.

[GACcc]

Dear all
I'm having same issues since patchday 07/2019
We're using ReFS on a Server 2016 (10.0.14393)
The snapshot of the single vm's takes some minutes but then for the synthetic backup it takes up to 4 hours and before about 1hour..
I just checked the refs.sys ans it was changed wed 7th of august 2019 the last time. I've patched the server last friday but it didn't help.

[GACcc]

Today I found this KB (https://support.microsoft.com/en-us/hel ... is-running) which exactly describes our problem and it was updated Jul 22, 2019. Does anyone have the same problem? Server have the newest patches installed - I am at a loss...

[mkretzer]

That article is quite old, in our case none of the settings really helped.