Comprehensive data protection for all workloads
yara12
Service Provider
Posts: 7
Liked: 1 time
Joined: Oct 25, 2018 11:33 am
Full Name: Yaroslav
Contact:

Re: Windows 2019, large REFS and deletes

Post by yara12 » Jul 13, 2019 4:34 am

However, trim/unmap have effect only on thin provisioned storages... I think.

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 13, 2019 7:24 am

Which we do have. This might at least help a bit. But every little thing helps in our situation.

soehl
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 09, 2011 12:43 pm
Full Name: Sebastian
Location: Germany
Contact:

Re: Windows 2019, large REFS and deletes

Post by soehl » Jul 13, 2019 9:37 am

mkretzer wrote:
Jun 27, 2019 5:34 pm
.... not too big volumes (the 600 Tb we have seem to be a little bit too much) was their recommendation for now.
Which size is the MS recommendation?!

Thanks!

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 13, 2019 12:38 pm

There was no specific recommendation as they are just building their own bigger test systems right now. For us we will set the new limit at ~150 TB.

All the problems should happen per-volumes and not server-global.

But this seem to have one other effect: Right now we have our new system with windows server 1903 and we did first tests. If the backend storage is not fast enough REFS memory usage does not only go to ~300 GB but ~600 GB with two REFS volumes.

So more REFS volumes might benefit from (i am not sure about "need") more RAM.

Gostev
SVP, Product Management
Posts: 24431
Liked: 3406 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » Jul 14, 2019 9:15 pm

mkretzer wrote:
Jul 13, 2019 7:24 am
[trim/unmap have effect only on thin provisioned storages] Which we do have.
Just wanted to note that Microsoft explicitly does not support thin provisioning on the storage device layer with ReFS.

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 14, 2019 10:04 pm

How is the storage being thin provisioned or not relevat to the filesystem? It should have no effect on the filesystem...

Do you have a link to that information?

Gostev
SVP, Product Management
Posts: 24431
Liked: 3406 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » Jul 15, 2019 5:28 pm

https://docs.microsoft.com/en-us/window ... s-overview
Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

bretesq
Service Provider
Posts: 16
Liked: 2 times
Joined: Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel
Contact:

Re: Windows 2019, large REFS and deletes

Post by bretesq » Jul 29, 2019 5:01 pm

mkretzer wrote:
Jul 10, 2019 4:17 am
We just received new hardware and will switch to 1903 Server to test if that helps. I will keep this thread updated but expect results in 4-6 weeks (after first deletes).
Hey Mike,

Are you having any better luck with 1903? Hows the experience been thus far?

Thanks

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 29, 2019 5:51 pm 1 person likes this post

Hello,

its Markus :-)
It gets interesting next week. Then we will have the first deletions.

bretesq
Service Provider
Posts: 16
Liked: 2 times
Joined: Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel
Contact:

Re: Windows 2019, large REFS and deletes

Post by bretesq » Jul 29, 2019 5:53 pm

Sounds great. Look forward to hearing about it. Sorry about that. I don't know how I even got Mike :?

EvoGeek
Lurker
Posts: 1
Liked: never
Joined: Jun 26, 2019 5:55 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by EvoGeek » Aug 14, 2019 5:39 pm

Hi Markus:

How goes your system after the deletes?

Also, regarding version 1903. Since you mentioned being LTSC which is based off 1809, how did you go about getting 1903? Did you have to switch to Semi-Annual Channel (SAC)?

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 15, 2019 5:16 am 1 person likes this post

So far so good. Fírst deletes are through but those are rather small. But i am quite optimistic!

In 1-2 weeks the real big deletes are coming.

We completely reinstalled windows server core 1903.

YoMarK
Enthusiast
Posts: 50
Liked: 5 times
Joined: Jul 13, 2009 12:50 pm
Full Name: Mark
Location: The Netherlands
Contact:

Re: Windows 2019, large REFS and deletes

Post by YoMarK » Aug 15, 2019 3:22 pm

Subscribed to this topic.
Just installed three 60TB repo's(one 180TB scale out repo) based on Windows 2019 with ReFS(64k blocks), and am not really looking forward to ReFS issues again. :)

I'm not sure what would be the benefit of using Windows server 1903(for Veeam/RefS) compared to Windows 2016 given microsoft support cycles.

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 15, 2019 5:15 pm 3 people like this post

According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.

We were in direct contact with the REFS devs...

If the issue is still there we should have felt at least some issues by now. As I said I am quite optimistic for the first time in 2 years.

rhys.hammond
Veeam Vanguard
Posts: 47
Liked: 6 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 18, 2019 11:59 pm

We are experiencing similar issues with ReFS + large deletions causing severe performance degradation on our win server 2019 servers (build 1809) repository servers.
Synthetic Full operations that were previously completed in under 10 hours are now taking 48+ hours or more. Also seeing REFS writes from jobs hang, then transfer data, hang again, after restore points are deleted.

We're currently on 1809 so we are now considering upgrading to 1903.

For those curious, the specs of our repository servers are as follows,
-Windows Server 2019 (version 1809)
-dual socket Intel Xeon-Gold 6130 (2.1Ghz 16 core)
-128GB DDR4
-2 x 480GB SSD for running Windows
- HPE Smart Array P408i-p SR Gen 10 Raid Controller with 2GB Cache
-26 x 8TB SATA 6Gbps 7.2K rpm 3.5”
The filesystem is ReFS with 64K cluster size

Regarding upgrading from 1809 to 1903,
-If you're on CB (SAC), it should be an in-place upgrade
-If you're on LTSC, to go to 1903 it will be a wipe / reload.

btw, our Veeam support case currently open for this issue is, 03699027.

I'm curious to understand if there is a 'quality update / monthly cumulative' update for 1809 to remediate this fix or does this fix rely on new functionality only available in the feature update (1903).
Are we required to reformat the ReFS volume for the fix to work?
I'm also very keen to hear more from mkretzer, if the ReFS deletion issue has well and truly been put to rest.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

YoMarK
Enthusiast
Posts: 50
Liked: 5 times
Joined: Jul 13, 2009 12:50 pm
Full Name: Mark
Location: The Netherlands
Contact:

Re: Windows 2019, large REFS and deletes

Post by YoMarK » Aug 19, 2019 9:25 am

mkretzer wrote:
Aug 15, 2019 5:15 pm
According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.
I'm not sure if this is a reply to my post, but what I meant to ask is what the advantage of 1903 is compared to Windows 2016 specifically for Veeam.
If Windows 2019 has issues, then it's not a option right now. But 2016 still is, and is supported longer then 1903 then why use the Semi-Annual Channel releases(some consider those the public beta testing releases of windows server).
1903 ReFS will support Dedub and compression as well, but given the way it works, that is not really a killer feature.

As Windows 2016 has support until 12 January 2027(much longer then all Semi-Annual Channel releases like 1903), I would think it would be the more obvious choice for something "simple"(should just "work") like a fileserver.

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 19, 2019 9:50 am 1 person likes this post

@rhys.hammond Right now there is no update for 2019/1809 as far as i understand microsoft. They told me newest REFS driver builds should work with 2019/1809 but then this is no "fully supported" solution anymore.

I would gladly give you a "all is fixed" already but we are very caucious with such a statement before we have a full month of backups and the bigger backups get deleted. In the last 2 years we had many situation where we believed "problem is finally fixed" and the whole thing came down one week later.

But besides the first successfull smaller deletes there is one other good thing with 1903: Synthetics after 3 weeks were just as fast as in the first week. That was one thing that was always an issue: synthetics got slower the more data was on the repo!

rhys.hammond
Veeam Vanguard
Posts: 47
Liked: 6 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 20, 2019 3:54 am

That sounds promising mkretzer, thank you for the update.

If you happen to speak to the Microsoft devs again could you possibly confirm if we need to format ReFS volume containing veeam backups files when moving from 1809 to 1903, specifically during a wipe/reload as required for LTSC to CB (SAC) please? Given the mention of the architecture changes to ReFS from 1809 to 1903 I thought it would be prudent to confirm this.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 20, 2019 4:46 am

I can ask but i am also 95% certain you don't need to reformat as they offered to replace the driver in 2019/1809 with their dev driver which is even newer - without reformat.

rhys.hammond
Veeam Vanguard
Posts: 47
Liked: 6 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 20, 2019 9:50 am

Hey mkretzer,

That's certainly good news regarding the reformat likely not being necessary for 1809 to 1903.
Regarding this dev driver, iI'm guessing this is not available to the general public?
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

mkretzer
Expert
Posts: 532
Liked: 113 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 20, 2019 11:55 am

No it is not. Also they told me that there is a certain risk involved.

If you want to hear my take on it: Try the 1903 driver in 2019. It should work and then at least you have a driver which is in itself fit for production (but don't ask me to fix it if all goes wrong :-))

Post Reply

Who is online

Users browsing this forum: Bing [Bot], eric@sip.be, lasse@itogco.dk, tdewin and 62 guests