Win2016 ReFs repository server freeze during Back-up copy

VMware specific discussions

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby tsightler » Thu Jul 13, 2017 8:32 pm

There's an entire 30 page thread on issues with Veeam + ReFS in the main forum, we have lots of data on cases that work, and cases that cause issues, and are still collecting more data. I have probably talked to 100+ customers that are having no issues, vs a dozen or so that have. The ones that have experience issues have had many various problems, from anti-virus, to firmware drivers, to things addressed by the Microsoft hotfix. We haven't found resolutions for a few of these cases.

In some cases there are hotfixes from Microsoft that address some of these, but the backup copy/hang case is a little less understood, but in my experience it always occurs when the system has "free memory", but it is all in standby, i.e. for whatever reason Windows is unable to buffered memory to disk to truly free it. Unfortunately, unless I'm missing it, your screenshots of resource manager cut off the portion that breaks down the "free" memory into Standby vs Free.

Of course, there could also be issues that is not yet discovered, I was just trying to offer some suggestions based on what I have seen in the field with these clients. Do you have a support case?
tsightler
Veeam Software
 
Posts: 4802
Liked: 1760 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby kwinsor » Fri Jul 14, 2017 2:05 pm

Thanks. We have a case as of yesterday. Veeam Support - Case # 02221686
kwinsor
Influencer
 
Posts: 14
Liked: 1 time
Joined: Fri Oct 17, 2014 1:26 pm
Location: Toronto
Full Name: Kent Winsor

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby jamerson » Wed Jul 19, 2017 9:03 pm

We have the same issue with quiet a lot of customer with Windows Server 2016 with REFS Disks.
actually haven't thought about the REFS allocation.
is there is a way to check which allocations is the hdd formatted with to narrow down this issue ?
jamerson
Expert
 
Posts: 257
Liked: 16 times
Joined: Wed May 01, 2013 9:54 pm
Full Name: Julien

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby mikegodwin » Thu Jul 20, 2017 4:21 am

Can you run this (assuming E: drive is your ReFS drive), and look for Bytes Per Cluster:
Code: Select all
fsutil fsinfo refsinfo e:
mikegodwin
Enthusiast
 
Posts: 52
Liked: 1 time
Joined: Fri Oct 12, 2012 12:28 am
Full Name: Mike Godwin

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby ferrus » Thu Jul 20, 2017 9:17 am

From the ReFS thread in the main forum, as I think it's probably relevant here:

It looks like Microsoft have released the official patch now.

Code: Select all
Addressed performance issues in ReFS when backing up many terabytes of data.
Addressed issue where a stuck thread in ReFS might cause memory corruption.


https://support.microsoft.com/en-gb/help/4025334
ferrus
Veeam ProPartner
 
Posts: 134
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby jamerson » Sun Jul 23, 2017 11:27 am

mikegodwin wrote:Can you run this (assuming E: drive is your ReFS drive), and look for Bytes Per Cluster:
Code: Select all
fsutil fsinfo refsinfo e:


just checked one Storages and has the next :

C:\Windows\system32>fsutil fsinfo refsinfo f:
REFS Volume Serial Number : 0xd0ba8eb4ba8e971e
REFS Version : 3.1
Number Sectors : 0x00000003a37e0000
Total Clusters : 0x000000000746fc00
Free Clusters : 0x0000000004c05b8d
Total Reserved : 0x0000000000077c09
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Checksum Type: CHECKSUM_TYPE_NONE


as I believe the Storage is 4096 bytes .

and the off site storage

C:\Windows\system32>fsutil fsinfo refsinfo d:
REFS Volume Serial Number : 0x84ae3af2ae3adc7c
REFS Version : 3.1
Number Sectors : 0x00000001d10a0000
Total Clusters : 0x000000003a214000
Free Clusters : 0x000000000e86ad32
Total Reserved : 0x00000000003c0ba0
Bytes Per Sector : 512
Bytes Per Physical Sector : 512
Bytes Per Cluster : 4096
Checksum Type: CHECKSUM_TYPE_NONE

which one need to be reformatted to 64k?
jamerson
Expert
 
Posts: 257
Liked: 16 times
Joined: Wed May 01, 2013 9:54 pm
Full Name: Julien

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby davecla » Sun Aug 06, 2017 10:24 pm 1 person likes this post

To close off my experience here - After working with Veeam support for a few weeks and trying numerous options we have given up on ReFS and move our repo back to NTFS - everything working fine now.
davecla
Influencer
 
Posts: 20
Liked: 2 times
Joined: Wed Feb 03, 2016 9:40 pm
Full Name: Dave Clarke

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby ing:DT79 » Thu Aug 24, 2017 3:20 pm

I have the same issue. No news from the community about this topic?
ing:DT79
Novice
 
Posts: 4
Liked: never
Joined: Tue Aug 09, 2016 7:24 am
Full Name: Ing. DAVIDE TONINI

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby mkretzer » Fri Aug 25, 2017 7:47 am

We are also giving up on REFS. I think there is no hope that it will be fixed soon...
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby ing:DT79 » Tue Aug 29, 2017 9:25 am

What's your experience mkretzer?
ing:DT79
Novice
 
Posts: 4
Liked: never
Joined: Tue Aug 09, 2016 7:24 am
Full Name: Ing. DAVIDE TONINI

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby mkretzer » Tue Aug 29, 2017 11:57 am

How it all began you can read in the first post of the 4 k horror story.

We had many tries and basically are switching between our production backup repo and a temporary repo for 7 months now.

We had:

- Crashes with 4 k Block size (no longer happening with 64 k and 384 GB of RAM)
- Merges are ultra-fast the first week and then take longer and longer every week - a backup which took slightly over an hour now takes 10 hours after 5 weeks (still happening)
- While merging concurrent running backups take forever (somewhat better since the latest patches)
- After backups were deleted the filesystem gets even slower (somewhat better since the latest patches)
- Active fulls start extremly fast and then creep down to 1/6 performance the backend is capable of, stalling every few minutes (still happening)

Right now my theory is that integrity streams are to blame for the slow write speed. The problem i Veeam support told me there is no option to turn that off - which i find terrible as our backend storage already does silent data corruption protection and now it is done on two levels...

Markus
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Win2016 ReFs repository server freeze during Back-up cop

Veeam Logoby ing:DT79 » Wed Aug 30, 2017 7:03 am

Very interesting experience!

In my case I have a DELL PowerEdge R730xd as Windows Server 2016 repository, with 10G nics: during the first Active Full I see the transfer rate growing up quickly, but after a few minutes slowing down completely to 0! The job is still running, so after a long time the transfer rate goes up again...the overall speed and time are very poor!!

There aren't any kind of logs from the operating system, so I opened a support ticket to Veeam.

I think that the problem could be at low-level inside the ReFS: without SSD caching (tiering) enabled is too slow to write data to physical disks.
ing:DT79
Novice
 
Posts: 4
Liked: never
Joined: Tue Aug 09, 2016 7:24 am
Full Name: Ing. DAVIDE TONINI

Previous

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 40 guests