Comprehensive data protection for all workloads
yara12
Service Provider
Posts: 7
Liked: 1 time
Joined: Oct 25, 2018 11:33 am
Full Name: Yaroslav
Contact:

Re: Windows 2019, large REFS and deletes

Post by yara12 » Jul 13, 2019 4:34 am

However, trim/unmap have effect only on thin provisioned storages... I think.

mkretzer
Expert
Posts: 525
Liked: 111 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 13, 2019 7:24 am

Which we do have. This might at least help a bit. But every little thing helps in our situation.

soehl
Enthusiast
Posts: 52
Liked: 8 times
Joined: May 09, 2011 12:43 pm
Full Name: Sebastian
Location: Germany
Contact:

Re: Windows 2019, large REFS and deletes

Post by soehl » Jul 13, 2019 9:37 am

mkretzer wrote:
Jun 27, 2019 5:34 pm
.... not too big volumes (the 600 Tb we have seem to be a little bit too much) was their recommendation for now.
Which size is the MS recommendation?!

Thanks!

mkretzer
Expert
Posts: 525
Liked: 111 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 13, 2019 12:38 pm

There was no specific recommendation as they are just building their own bigger test systems right now. For us we will set the new limit at ~150 TB.

All the problems should happen per-volumes and not server-global.

But this seem to have one other effect: Right now we have our new system with windows server 1903 and we did first tests. If the backend storage is not fast enough REFS memory usage does not only go to ~300 GB but ~600 GB with two REFS volumes.

So more REFS volumes might benefit from (i am not sure about "need") more RAM.

Gostev
SVP, Product Management
Posts: 24421
Liked: 3402 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » Jul 14, 2019 9:15 pm

mkretzer wrote:
Jul 13, 2019 7:24 am
[trim/unmap have effect only on thin provisioned storages] Which we do have.
Just wanted to note that Microsoft explicitly does not support thin provisioning on the storage device layer with ReFS.

mkretzer
Expert
Posts: 525
Liked: 111 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 14, 2019 10:04 pm

How is the storage being thin provisioned or not relevat to the filesystem? It should have no effect on the filesystem...

Do you have a link to that information?

Gostev
SVP, Product Management
Posts: 24421
Liked: 3402 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » Jul 15, 2019 5:28 pm

https://docs.microsoft.com/en-us/window ... s-overview
Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

bretesq
Service Provider
Posts: 16
Liked: 2 times
Joined: Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel
Contact:

Re: Windows 2019, large REFS and deletes

Post by bretesq » Jul 29, 2019 5:01 pm

mkretzer wrote:
Jul 10, 2019 4:17 am
We just received new hardware and will switch to 1903 Server to test if that helps. I will keep this thread updated but expect results in 4-6 weeks (after first deletes).
Hey Mike,

Are you having any better luck with 1903? Hows the experience been thus far?

Thanks

mkretzer
Expert
Posts: 525
Liked: 111 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Jul 29, 2019 5:51 pm 1 person likes this post

Hello,

its Markus :-)
It gets interesting next week. Then we will have the first deletions.

bretesq
Service Provider
Posts: 16
Liked: 2 times
Joined: Jan 19, 2015 3:19 pm
Full Name: Bret Esquivel
Contact:

Re: Windows 2019, large REFS and deletes

Post by bretesq » Jul 29, 2019 5:53 pm

Sounds great. Look forward to hearing about it. Sorry about that. I don't know how I even got Mike :?

EvoGeek
Lurker
Posts: 1
Liked: never
Joined: Jun 26, 2019 5:55 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by EvoGeek » Aug 14, 2019 5:39 pm

Hi Markus:

How goes your system after the deletes?

Also, regarding version 1903. Since you mentioned being LTSC which is based off 1809, how did you go about getting 1903? Did you have to switch to Semi-Annual Channel (SAC)?

mkretzer
Expert
Posts: 525
Liked: 111 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 15, 2019 5:16 am 1 person likes this post

So far so good. Fírst deletes are through but those are rather small. But i am quite optimistic!

In 1-2 weeks the real big deletes are coming.

We completely reinstalled windows server core 1903.

YoMarK
Enthusiast
Posts: 49
Liked: 5 times
Joined: Jul 13, 2009 12:50 pm
Full Name: Mark
Location: The Netherlands
Contact:

Re: Windows 2019, large REFS and deletes

Post by YoMarK » Aug 15, 2019 3:22 pm

Subscribed to this topic.
Just installed three 60TB repo's(one 180TB scale out repo) based on Windows 2019 with ReFS(64k blocks), and am not really looking forward to ReFS issues again. :)

I'm not sure what would be the benefit of using Windows server 1903(for Veeam/RefS) compared to Windows 2016 given microsoft support cycles.

mkretzer
Expert
Posts: 525
Liked: 111 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Aug 15, 2019 5:15 pm 2 people like this post

According to MS developers the benefit is huge. They fixed one of the major issues in 1903 for deletes.

We were in direct contact with the REFS devs...

If the issue is still there we should have felt at least some issues by now. As I said I am quite optimistic for the first time in 2 years.

rhys.hammond
Veeam Vanguard
Posts: 43
Liked: 5 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond » Aug 18, 2019 11:59 pm

We are experiencing similar issues with ReFS + large deletions causing severe performance degradation on our win server 2019 servers (build 1809) repository servers.
Synthetic Full operations that were previously completed in under 10 hours are now taking 48+ hours or more. Also seeing REFS writes from jobs hang, then transfer data, hang again, after restore points are deleted.

We're currently on 1809 so we are now considering upgrading to 1903.

For those curious, the specs of our repository servers are as follows,
-Windows Server 2019 (version 1809)
-dual socket Intel Xeon-Gold 6130 (2.1Ghz 16 core)
-128GB DDR4
-2 x 480GB SSD for running Windows
- HPE Smart Array P408i-p SR Gen 10 Raid Controller with 2GB Cache
-26 x 8TB SATA 6Gbps 7.2K rpm 3.5”
The filesystem is ReFS with 64K cluster size

Regarding upgrading from 1809 to 1903,
-If you're on CB (SAC), it should be an in-place upgrade
-If you're on LTSC, to go to 1903 it will be a wipe / reload.

btw, our Veeam support case currently open for this issue is, 03699027.

I'm curious to understand if there is a 'quality update / monthly cumulative' update for 1809 to remediate this fix or does this fix rely on new functionality only available in the feature update (1903).
Are we required to reformat the ReFS volume for the fix to work?
I'm also very keen to hear more from mkretzer, if the ReFS deletion issue has well and truly been put to rest.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 23 guests