Failed to flush file buffers

Availability for the Always-On Enterprise

Failed to flush file buffers

Veeam Logoby DavidS » Thu Jan 08, 2015 10:11 am

I recently encouter a strange error on Veeam fullbackups.

Code: Select all
07.01.2015 18:40:39 :: Error: Incorrect function.
Failed to flush file buffers. File: [E:\Backup\BCK_Name_ALL\BCK_Name_ALL2015-01-07T113550.vbk].


A fullbackup uses about 3-4TB in our environment, fullbackups worked before. I think this error occures since the upgrade to Veeam v8.

As soon as the Backupfile (vbk) reaches a size about 2 TB+, this error occurs and the session stops. If I do a Retry the performance drops to around 5MB/s and at some point the retry also fails with nothing written to the vbk.

What components I can exclude so far are the storage, the network and the server performance.

I suspect it has something to do with the repository, its an Windowsserver Repository, an iSCSI-Storage with a 17TB LUN behind that.

Now the blocksize we used in NTFS is 64k, maybe this is the issue? Would 8k be the better blocksize?

And what about REFS, would a Volume formatted with REFS maybe prevent this failure (although the REFS min. blocksize is alos 64k)

Thanks
David
DavidS
Veeam ProPartner
 
Posts: 11
Liked: 6 times
Joined: Mon Aug 18, 2014 12:00 pm
Location: Switzerland
Full Name: David

Re: Failed to flush file buffers

Veeam Logoby dellock6 » Thu Jan 08, 2015 11:12 am

Hi David,
this kind of errors are better to be investigated by support, so please open a ticket.

Regarding block size, default NTFS block size is ok, and if any, to gain better performances it should be increased rather than reduced. Veeam smaller blocks are 128KB (wan deduplication at 256kb plus compression adding around 2x reduction) so 64k or 128k is fine. Also, take care of the underlying storage, also those volume should use the same block size, to avoid write amplification. Alignment of the different layers is also another concern, but since you are talking about ReFS I suppose you are using Windows 2012, so alignment usually is not a problem since in modern windows edition (2008 and above) is done automatically.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5118
Liked: 1361 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

[solved] Re: Failed to flush file buffers

Veeam Logoby DavidS » Tue Feb 10, 2015 1:12 pm 2 people like this post

Hello Everyone

A little followup to my problem.
As Luca suggested I opened a support ticket.

The support couldn't really help us solving this problem so we invistigated some more hours by ourself.

We adjusted some iSCSI settings for optimization, finally this optimizations solved our problem:

TcpAckFrequenzy
Code: Select all
Edit all Adapters, used for the iSCSI connection with the Value "TcpAckFrequenzy = 1". If you use multipath, edit this settings for every adapter.

   Create this DWORD key with a value of 1 :
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\<Interface GUID>\TcpAckFrequency


Nagle
Code: Select all
Disabled Nagle for iSCSI

Create this DWORD key with a value of 1:
HKLM\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance Number>\Parameters\iSCSIDisableNagle


Last step -> Reboot Veeam Server

This solved our problem. We have 2 iSCSI connections, one over a 10GB Ethernet and the other with a simple 1GB Ethernet.
Though, we only encoutered this problems with the 10GB Ethernet. It seems Veeam, iSCSI and 10GB aren't the best friends

I hope this solutions may help someone with similar problems.

Regards
David
DavidS
Veeam ProPartner
 
Posts: 11
Liked: 6 times
Joined: Mon Aug 18, 2014 12:00 pm
Location: Switzerland
Full Name: David

Re: Failed to flush file buffers

Veeam Logoby spgsit5upport » Wed Nov 16, 2016 4:30 pm

No idea what it is about iSCSIDisableNagle, as such entry no longer is used for anything, it is only TcpAckFrequency to deactivate Nagle Algorithm on iSCSI interfaces

https://gallery.technet.microsoft.com/s ... m-66ca7608
spgsit5upport
Expert
 
Posts: 182
Liked: 14 times
Joined: Fri May 28, 2010 10:25 am
Full Name: Seb

Re: Failed to flush file buffers

Veeam Logoby spgsit5upport » Fri Nov 18, 2016 9:30 am

spgsit5upport
Expert
 
Posts: 182
Liked: 14 times
Joined: Fri May 28, 2010 10:25 am
Full Name: Seb


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: fika and 23 guests