Client error: An existing connection was forcibly closed ...

[unluckynoob]

Error: Client error: An existing connection was forcibly closed by remote host.
Error: Client error: ChannelError: ConnectionReset

hi all,

any ideas/pointers on how to troubleshoot this. it happens for one particularly large guest on hyper-v cluster, all other backups are fine.

it happens every time the backup runs, but at seemingly random points through the process from a couple of minutes to several hours.

target is over a WAN/vpn connection, however VPN is stable and has been ruled out.

backup job has been recreated from scratch and have also tried preseeding, still happens....

struggling to find any relevant errors in any server eventlogs that might indicate whats going on.

any help would be greatly appreciated

[Gostev]

Hi, according to your description this issue looks to be outside B&R, and rather pure networking problem. Perhaps with the help of some networking professional and via network traffic capture/analysis (I doubt you will be able to get assistance with that via the forum posts).

May be worth opening a case with our support anyway, as they have seen it all by now, and may have some suggestions for you where to start.

Thanks!

[cby]

Hi

This error, along with "Error: Client error: End of file Unable to retrieve next block transmission command." are typical of backup/replication of large VMs over a WAN that may be throttled or the bandwidth reduced for some reason (QoS etc.). The 'invisible' devices and infrastructure (e.g CWDM vs DWDM) in the network chain can additionally hamper/interrupt data transfer. Also don't exclude proxy/ESX host/SAN performance at the target end.

[unluckynoob]

thanks for all of the suggestions.

I did some further digging on the forums and found a couple of pointers to the 'use multiple tcp/ip streams' tick box on the backup proxy.

I've unticked this and all of the issues have gone away, not a sniff of a problem. there's also no noticeable difference to backup times so not really sure what the benefit was...

anyway, for me this is problem solved.

[cby]

Good find on the proxy. I'll bear that one in mind. Thanks.

[Gostev]

I've encountered both VPN servers and networking hardware that started to drop connections after reaching certain number of concurrent TCP/IP connection before, so this must have been the case here... strange though, as the default is just 5 concurrent connections, which is very low. This means other workloads can easily saturate the limit, and cause similar issues in future.

there's also no noticeable difference to backup times so not really sure what the benefit was

The benefit can usually be seen on high latency links, or when certain networking hardware throttles individual TCP/IP connections down, for example for QoS.