Error: Connection forcibly closed by remote host very often

[StephanF]

Hi,

we use Endpoint Backup 1.5 to backup about 20 machines to the repository of our Veeam B&R 9.5 backup server.
The problem we are seeing is that we get a lot of errors "Connection forcibly closed by remote host" on all the clients. Sometimes this error happens 20 or more times before a successful backup cycle occurs. This seems to happen on all endpoints. Server backup backup itself runs fine.

We already tried the registry key from KB1781 without success. Disabling the firewall on the backup server does not help either. There are no network or connectivity issues. Everything else runs fine. All machines are physical on the same LAN. Clients and backup server are in different subnets but these are routed to each other without any firewall or other blocks in between.

Any ideas?

[Dima P.]

Hi StephanF,

Kindly open a support case and let our team check your debug logs. Please do not forget to include the case ID in this thread. Thank you.

[StephanF]

Hi,

I had to install VEB on a Computer that is allowed to connect to the Internet first. Seeing the issue here as well.

Support case #02118660

[StephanF]

Hi all,

I just wanted to give a feedback that the issue is resolved. Though I found the final solution by myself, Veeam Support was extremely helpful to analyse the issue and point me to the right direction. A very big compliment for the professional work.

Finally the issue was not a software problem of Veeam itself. Deep analysis with wireshark showed that the server was sending RST packages on port 10001 which closed the connection to the clients. The root cause was unclear for a long time.
Updates of drivers, NIC firmware etc. did not help. After disabling more and more things we made it work by removing the virtual Hyper-V switch sitting on the network interface. Hyper-V role is enabled on the VBR server for having more restore and SureBackup options. Interestingly the issue first did not reappear after recreating the virtual switch. It needed a reboot of the server and then the issue was back. This made me mad.

Then I stumbled over some sites describing issues with Qlogic NICs and Virtual Machine Queues. Your find some information here, here and here. Those reports did not fit well because they reffer to VMs loosing connectivity and 1-gig NICs. Our NICs are 10-gig but from Qlogic. So I thought I give it a try. I disabled VMQ on all interfaces using PowerShell.

Code: Select all

Set-NetAdapterVmq -Name NIC1,NIC2 -Enabled $False

Since then the issue is gone and all endpoint backups work like a charm.

Hopefully this will help someone in the future.

Greetings
Stephan