Discussions specific to the VMware vSphere hypervisor
tsightler
Veeam Software
Posts: 5195
Liked: 2077 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by tsightler » Jul 13, 2017 8:32 pm

There's an entire 30 page thread on issues with Veeam + ReFS in the main forum, we have lots of data on cases that work, and cases that cause issues, and are still collecting more data. I have probably talked to 100+ customers that are having no issues, vs a dozen or so that have. The ones that have experience issues have had many various problems, from anti-virus, to firmware drivers, to things addressed by the Microsoft hotfix. We haven't found resolutions for a few of these cases.

In some cases there are hotfixes from Microsoft that address some of these, but the backup copy/hang case is a little less understood, but in my experience it always occurs when the system has "free memory", but it is all in standby, i.e. for whatever reason Windows is unable to buffered memory to disk to truly free it. Unfortunately, unless I'm missing it, your screenshots of resource manager cut off the portion that breaks down the "free" memory into Standby vs Free.

Of course, there could also be issues that is not yet discovered, I was just trying to offer some suggestions based on what I have seen in the field with these clients. Do you have a support case?

kwinsor
Service Provider
Posts: 38
Liked: 2 times
Joined: Oct 17, 2014 1:26 pm
Full Name: Kent Winsor
Location: Toronto
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by kwinsor » Jul 14, 2017 2:05 pm

Thanks. We have a case as of yesterday. Veeam Support - Case # 02221686

jamerson
Expert
Posts: 285
Liked: 17 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by jamerson » Jul 19, 2017 9:03 pm

We have the same issue with quiet a lot of customer with Windows Server 2016 with REFS Disks.
actually haven't thought about the REFS allocation.
is there is a way to check which allocations is the hdd formatted with to narrow down this issue ?

mikegodwin
Enthusiast
Posts: 53
Liked: 1 time
Joined: Oct 12, 2012 12:28 am
Full Name: Mike Godwin
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by mikegodwin » Jul 20, 2017 4:21 am

Can you run this (assuming E: drive is your ReFS drive), and look for Bytes Per Cluster:

Code: Select all

fsutil fsinfo refsinfo e:

ferrus
Veeam ProPartner
Posts: 190
Liked: 23 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by ferrus » Jul 20, 2017 9:17 am

From the ReFS thread in the main forum, as I think it's probably relevant here:
It looks like Microsoft have released the official patch now.

Code: Select all

Addressed performance issues in ReFS when backing up many terabytes of data. 
Addressed issue where a stuck thread in ReFS might cause memory corruption.
https://support.microsoft.com/en-gb/help/4025334

jamerson
Expert
Posts: 285
Liked: 17 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by jamerson » Jul 23, 2017 11:27 am

mikegodwin wrote:Can you run this (assuming E: drive is your ReFS drive), and look for Bytes Per Cluster:

Code: Select all

fsutil fsinfo refsinfo e:
just checked one Storages and has the next :

C:\Windows\system32>fsutil fsinfo refsinfo f:
REFS Volume Serial Number : 0xd0ba8eb4ba8e971e
REFS Version : 3.1
Number Sectors : 0x00000003a37e0000
Total Clusters : 0x000000000746fc00
Free Clusters : 0x0000000004c05b8d
Total Reserved : 0x0000000000077c09
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Checksum Type: CHECKSUM_TYPE_NONE


as I believe the Storage is 4096 bytes .

and the off site storage

C:\Windows\system32>fsutil fsinfo refsinfo d:
REFS Volume Serial Number : 0x84ae3af2ae3adc7c
REFS Version : 3.1
Number Sectors : 0x00000001d10a0000
Total Clusters : 0x000000003a214000
Free Clusters : 0x000000000e86ad32
Total Reserved : 0x00000000003c0ba0
Bytes Per Sector : 512
Bytes Per Physical Sector : 512
Bytes Per Cluster : 4096
Checksum Type: CHECKSUM_TYPE_NONE

which one need to be reformatted to 64k?

davecla
Enthusiast
Posts: 25
Liked: 4 times
Joined: Feb 03, 2016 9:40 pm
Full Name: Dave Clarke
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by davecla » Aug 06, 2017 10:24 pm 1 person likes this post

To close off my experience here - After working with Veeam support for a few weeks and trying numerous options we have given up on ReFS and move our repo back to NTFS - everything working fine now.

ing:DT79
Novice
Posts: 4
Liked: never
Joined: Aug 09, 2016 7:24 am
Full Name: Ing. DAVIDE TONINI
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by ing:DT79 » Aug 24, 2017 3:20 pm

I have the same issue. No news from the community about this topic?

mkretzer
Expert
Posts: 401
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by mkretzer » Aug 25, 2017 7:47 am

We are also giving up on REFS. I think there is no hope that it will be fixed soon...

ing:DT79
Novice
Posts: 4
Liked: never
Joined: Aug 09, 2016 7:24 am
Full Name: Ing. DAVIDE TONINI
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by ing:DT79 » Aug 29, 2017 9:25 am

What's your experience mkretzer?

mkretzer
Expert
Posts: 401
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by mkretzer » Aug 29, 2017 11:57 am

How it all began you can read in the first post of the 4 k horror story.

We had many tries and basically are switching between our production backup repo and a temporary repo for 7 months now.

We had:

- Crashes with 4 k Block size (no longer happening with 64 k and 384 GB of RAM)
- Merges are ultra-fast the first week and then take longer and longer every week - a backup which took slightly over an hour now takes 10 hours after 5 weeks (still happening)
- While merging concurrent running backups take forever (somewhat better since the latest patches)
- After backups were deleted the filesystem gets even slower (somewhat better since the latest patches)
- Active fulls start extremly fast and then creep down to 1/6 performance the backend is capable of, stalling every few minutes (still happening)

Right now my theory is that integrity streams are to blame for the slow write speed. The problem i Veeam support told me there is no option to turn that off - which i find terrible as our backend storage already does silent data corruption protection and now it is done on two levels...

Markus

ing:DT79
Novice
Posts: 4
Liked: never
Joined: Aug 09, 2016 7:24 am
Full Name: Ing. DAVIDE TONINI
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by ing:DT79 » Aug 30, 2017 7:03 am

Very interesting experience!

In my case I have a DELL PowerEdge R730xd as Windows Server 2016 repository, with 10G nics: during the first Active Full I see the transfer rate growing up quickly, but after a few minutes slowing down completely to 0! The job is still running, so after a long time the transfer rate goes up again...the overall speed and time are very poor!!

There aren't any kind of logs from the operating system, so I opened a support ticket to Veeam.

I think that the problem could be at low-level inside the ReFS: without SSD caching (tiering) enabled is too slow to write data to physical disks.

jamerson
Expert
Posts: 285
Liked: 17 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by jamerson » Nov 14, 2017 1:13 pm

We are having the same issue with couple of our customer,
the solution was is go back to NTFS untill is this fixed in the next release.

tomkod
Lurker
Posts: 1
Liked: never
Joined: Feb 01, 2018 7:22 am
Full Name: Tomasz Dorocicz
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by tomkod » Feb 01, 2018 7:44 am

Hello @ll,

we are experiencing the same issue in our infrastructure.
In our case - there is Veeam Server running on Windows 2016 which has iSCSI Disk attached directly to the System and formatted as 4TB with ReFS 64K. This drive is set in Veeam as Repository for Copy Job to take advantage of Fast Clone feature etc.
At the first run everything was running smoothly, but after transfer about 1TB data it has started to freezing and performance drops dramatically. The job is still running but the transfer goes up for about minute than it freeze for about 5-10minutes. When its frozen than the iSCSI drive mapped as Drive Z:\ in System is unavailable.
I was changing the network configuration and iSCSI Target settings without knowing that it's the ReFS issue.
BTW. If the Copy Job is disabled, this issue not occurs. I can copy some large files directly to that drive without any problems. It starts to happening when the Copy Job is running.
For now the solution as I read is back to NTFS but it's realy

Gostev
Veeam Software
Posts: 22992
Liked: 2888 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Win2016 ReFs repository server freeze during Back-up cop

Post by Gostev » Feb 02, 2018 12:00 am

Microsoft is releasing the fix for this ReFS issue in the next cumulative update (there's the main 50 pages thread about this issue, this is a duplicate discussion).

Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests