Comprehensive data protection for all workloads
yara12
Service Provider
Posts: 9
Liked: 1 time
Joined: Oct 25, 2018 11:33 am
Full Name: Yaroslav
Contact:

Re: Windows 2019, large REFS and deletes

Post by yara12 » Jul 13, 2019 4:34 am

However, trim/unmap have effect only on thin provisioned storages... I think.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 13, 2019 7:24 am

Which we do have. This might at least help a bit. But every little thing helps in our situation.

soehl
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 09, 2011 12:43 pm
Full Name: Sebastian
Location: Germany
Contact:

Re: Windows 2019, large REFS and deletes

Post by soehl » Jul 13, 2019 9:37 am

mkretzer wrote:
Jun 27, 2019 5:34 pm
.... not too big volumes (the 600 Tb we have seem to be a little bit too much) was their recommendation for now.
Which size is the MS recommendation?!

Thanks!

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 13, 2019 12:38 pm

There was no specific recommendation as they are just building their own bigger test systems right now. For us we will set the new limit at ~150 TB.

All the problems should happen per-volumes and not server-global.

But this seem to have one other effect: Right now we have our new system with windows server 1903 and we did first tests. If the backend storage is not fast enough REFS memory usage does not only go to ~300 GB but ~600 GB with two REFS volumes.

So more REFS volumes might benefit from (i am not sure about "need") more RAM.

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » Jul 14, 2019 9:15 pm

mkretzer wrote:
Jul 13, 2019 7:24 am
[trim/unmap have effect only on thin provisioned storages] Which we do have.
Just wanted to note that Microsoft explicitly does not support thin provisioning on the storage device layer with ReFS.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 14, 2019 10:04 pm

How is the storage being thin provisioned or not relevat to the filesystem? It should have no effect on the filesystem...

Do you have a link to that information?

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » Jul 15, 2019 5:28 pm

https://docs.microsoft.com/en-us/window ... s-overview
Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

bretesq
Service Provider
Posts: 16
Liked: 2 times
Joined: Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel
Contact:

Re: Windows 2019, large REFS and deletes

Post by bretesq » Jul 29, 2019 5:01 pm

mkretzer wrote:
Jul 10, 2019 4:17 am
We just received new hardware and will switch to 1903 Server to test if that helps. I will keep this thread updated but expect results in 4-6 weeks (after first deletes).
Hey Mike,

Are you having any better luck with 1903? Hows the experience been thus far?

Thanks

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 29, 2019 5:51 pm 1 person likes this post

Hello,

its Markus :-)
It gets interesting next week. Then we will have the first deletions.

bretesq
Service Provider
Posts: 16
Liked: 2 times
Joined: Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel
Contact:

Re: Windows 2019, large REFS and deletes

Post by bretesq » Jul 29, 2019 5:53 pm

Sounds great. Look forward to hearing about it. Sorry about that. I don't know how I even got Mike :?

EvoGeek
Novice
Posts: 5
Liked: never
Joined: Jun 26, 2019 5:55 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by EvoGeek » Aug 14, 2019 5:39 pm

Hi Markus:

How goes your system after the deletes?

Also, regarding version 1903. Since you mentioned being LTSC which is based off 1809, how did you go about getting 1903? Did you have to switch to Semi-Annual Channel (SAC)?

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 15, 2019 5:16 am 1 person likes this post

So far so good. Fírst deletes are through but those are rather small. But i am quite optimistic!

In 1-2 weeks the real big deletes are coming.

We completely reinstalled windows server core 1903.

YoMarK
Enthusiast
Posts: 50
Liked: 5 times
Joined: Jul 13, 2009 12:50 pm
Full Name: Mark
Location: The Netherlands
Contact:

Re: Windows 2019, large REFS and deletes

Post by YoMarK » Aug 15, 2019 3:22 pm

Subscribed to this topic.
Just installed three 60TB repo's(one 180TB scale out repo) based on Windows 2019 with ReFS(64k blocks), and am not really looking forward to ReFS issues again. :)

I'm not sure what would be the benefit of using Windows server 1903(for Veeam/RefS) compared to Windows 2016 given microsoft support cycles.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 15, 2019 5:15 pm 3 people like this post

According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.

We were in direct contact with the REFS devs...

If the issue is still there we should have felt at least some issues by now. As I said I am quite optimistic for the first time in 2 years.

rhys.hammond
Veeam Vanguard
Posts: 58
Liked: 9 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 18, 2019 11:59 pm

We are experiencing similar issues with ReFS + large deletions causing severe performance degradation on our win server 2019 servers (build 1809) repository servers.
Synthetic Full operations that were previously completed in under 10 hours are now taking 48+ hours or more. Also seeing REFS writes from jobs hang, then transfer data, hang again, after restore points are deleted.

We're currently on 1809 so we are now considering upgrading to 1903.

For those curious, the specs of our repository servers are as follows,
-Windows Server 2019 (version 1809)
-dual socket Intel Xeon-Gold 6130 (2.1Ghz 16 core)
-128GB DDR4
-2 x 480GB SSD for running Windows
- HPE Smart Array P408i-p SR Gen 10 Raid Controller with 2GB Cache
-26 x 8TB SATA 6Gbps 7.2K rpm 3.5”
The filesystem is ReFS with 64K cluster size

Regarding upgrading from 1809 to 1903,
-If you're on CB (SAC), it should be an in-place upgrade
-If you're on LTSC, to go to 1903 it will be a wipe / reload.

btw, our Veeam support case currently open for this issue is, 03699027.

I'm curious to understand if there is a 'quality update / monthly cumulative' update for 1809 to remediate this fix or does this fix rely on new functionality only available in the feature update (1903).
Are we required to reformat the ReFS volume for the fix to work?
I'm also very keen to hear more from mkretzer, if the ReFS deletion issue has well and truly been put to rest.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

YoMarK
Enthusiast
Posts: 50
Liked: 5 times
Joined: Jul 13, 2009 12:50 pm
Full Name: Mark
Location: The Netherlands
Contact:

Re: Windows 2019, large REFS and deletes

Post by YoMarK » Aug 19, 2019 9:25 am

mkretzer wrote:
Aug 15, 2019 5:15 pm
According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.
I'm not sure if this is a reply to my post, but what I meant to ask is what the advantage of 1903 is compared to Windows 2016 specifically for Veeam.
If Windows 2019 has issues, then it's not a option right now. But 2016 still is, and is supported longer then 1903 then why use the Semi-Annual Channel releases(some consider those the public beta testing releases of windows server).
1903 ReFS will support Dedub and compression as well, but given the way it works, that is not really a killer feature.

As Windows 2016 has support until 12 January 2027(much longer then all Semi-Annual Channel releases like 1903), I would think it would be the more obvious choice for something "simple"(should just "work") like a fileserver.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 19, 2019 9:50 am 1 person likes this post

@rhys.hammond Right now there is no update for 2019/1809 as far as i understand microsoft. They told me newest REFS driver builds should work with 2019/1809 but then this is no "fully supported" solution anymore.

I would gladly give you a "all is fixed" already but we are very caucious with such a statement before we have a full month of backups and the bigger backups get deleted. In the last 2 years we had many situation where we believed "problem is finally fixed" and the whole thing came down one week later.

But besides the first successfull smaller deletes there is one other good thing with 1903: Synthetics after 3 weeks were just as fast as in the first week. That was one thing that was always an issue: synthetics got slower the more data was on the repo!

rhys.hammond
Veeam Vanguard
Posts: 58
Liked: 9 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 20, 2019 3:54 am

That sounds promising mkretzer, thank you for the update.

If you happen to speak to the Microsoft devs again could you possibly confirm if we need to format ReFS volume containing veeam backups files when moving from 1809 to 1903, specifically during a wipe/reload as required for LTSC to CB (SAC) please? Given the mention of the architecture changes to ReFS from 1809 to 1903 I thought it would be prudent to confirm this.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 20, 2019 4:46 am

I can ask but i am also 95% certain you don't need to reformat as they offered to replace the driver in 2019/1809 with their dev driver which is even newer - without reformat.

rhys.hammond
Veeam Vanguard
Posts: 58
Liked: 9 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 20, 2019 9:50 am

Hey mkretzer,

That's certainly good news regarding the reformat likely not being necessary for 1809 to 1903.
Regarding this dev driver, iI'm guessing this is not available to the general public?
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 20, 2019 11:55 am

No it is not. Also they told me that there is a certain risk involved.

If you want to hear my take on it: Try the 1903 driver in 2019. It should work and then at least you have a driver which is in itself fit for production (but don't ask me to fix it if all goes wrong :-))

nikpolini
Enthusiast
Posts: 30
Liked: 2 times
Joined: May 28, 2015 3:23 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by nikpolini » Aug 21, 2019 10:37 pm

Hi All
I see the update "August 17, 2019—KB4512534 (OS Build 17763.720)" the refs.dll has changed.
For 1809 Windows Server 2019 that is.
No mention in release notes though...
Nik

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 22, 2019 5:38 am

To be honest thats the most confusing about the whole thing: the refs.sys from 1903 was not updates once since release.
And the 2019 version is all the time.

Still, i was told that 1903 version is superior to the 2019 version...

rhys.hammond
Veeam Vanguard
Posts: 58
Liked: 9 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 23, 2019 12:13 am

I've been making some tweaks in preparation for the weekend when our performance appears to nosedive off a cliff.

-Uninstalled SCCM client as per https://www.veeam.com/kb2792 (the GUI was bugged and refusing to start which wasn't a good sign)
-very small tweak to our raid cache policy as per 'HPE Reference Architecture for Veeam Availability Suite with HPE Apollo backup target' document
-removed concurrent task limit on the backup repository as per 'HPE Reference Architecture for Veeam Availability Suite with HPE Apollo backup target' document
-applied all windows updates available and restarted (we were on vanilla 2019 with zero updates)

Interesting that HPE is recommending Windows Server 2019 in their Veeam Apollo Repository Server Reference Architecture as well.

Now we need to wait for the weekend to see if how it goes... fingers crossed.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 23, 2019 6:16 am 1 person likes this post

Yesterday we had our first set of large deletes. The best thing: Memory usage did not increase at all. With 2019 memory usage fluctuated heavily when there was something deleted. Also there was ZERO impact to other backups.
The next 2-3 days will get VERY interesting as now the HUGE deletes are coming!

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 24, 2019 6:15 am 4 people like this post

Ok. In the last 2 days there where ~10000 restore points being deleted.

And it seems without any impact AT ALL!

I will ask MS support if there is any hope for creating a backport....

rhys.hammond
Veeam Vanguard
Posts: 58
Liked: 9 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 26, 2019 12:23 am 1 person likes this post

So I'm happy to report that our poor performance issue did not reoccur on the weekend. Synthetic fulls / GFS took the usual few hours to complete, not days to complete.
Whether it was the "August 17, 2019—KB4512534 (OS Build 17763.720)" update resolved the issue or perhaps just the removal of the Microsoft Config Manager client we're not quite sure.
We ended up implementing quite a few improvements to try and solve these 'deletion' triggered performance issues so hard to pinpoint the exact fix.

We are now reinstalling Microsoft Config Manager client and will report back after next weekend passes, fingers crossed the issue doesn't appear so we can stick on build 1807.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

GACcc
Novice
Posts: 7
Liked: never
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc » Aug 26, 2019 9:39 am

Dear all
I'm having same issues since patchday 07/2019
We're using ReFS on a Server 2016 (10.0.14393)
The snapshot of the single vm's takes some minutes but then for the synthetic backup it takes up to 4 hours and before about 1hour..
I just checked the refs.sys ans it was changed wed 7th of august 2019 the last time. I've patched the server last friday but it didn't help.

GACcc
Novice
Posts: 7
Liked: never
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc » Aug 26, 2019 1:15 pm

Today I found this KB (https://support.microsoft.com/en-us/hel ... is-running) which exactly describes our problem and it was updated Jul 22, 2019. Does anyone have the same problem? Server have the newest patches installed - I am at a loss...

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 26, 2019 7:12 pm

That article is quite old, in our case none of the settings really helped.

Post Reply

Who is online

Users browsing this forum: nmdange and 38 guests