WAN Replication failing

[pacmantravis]

We are trying to replicate VMs from one data center to another (vSphere ESXi hosts on both sides) and the replication job keeps failing after around 70-90GB of transfer.

The speeds and latency between the two data centers is actually pretty decent. Average throughput of around 20-30Mbs each way and latency is around 10-12ms. The connection is going through an IPSec VPN tunnel.

We have tried replicating different VMs using all replication methods and all of them have the same error:

Code: Select all

Client error: TCP error, code: [10054].An existing connection was forcibly closed by the remote host

I've opened a ticket with support, but haven't really gotten too far with them (they are working their butts off to try and help us out though).

I was just wondering if anyone here had the same issues and if so, what you did to fix it.

We have tried large FTP and large FastSCP and WinSCP transfers and the transfers work fine; it's just the replication jobs that aren't working.

[Vitaliy S.]

Hello Travis,

Have you tried setting up the replication job for one VM only, is it also failing? As it might be some kind of a connectivity issue I would also suggest trying replica seeding option, so you could transfer only changed blocks on the next run via your link betweet two datacenters.

And yes, please continue working with support.

Thank you!

[Gostev]

Travis, did you try to replicate the same VM locally (not across WAN) and see if the issue is gone. If this is the case, then this would indicate that the issue is somehow connected with your WAN/VPN link. It would be good test in all cases to locate where the issue sits.

[pacmantravis]

Hi All,

The replication job is set for just one VM at the moment. We have successfully replicated locally. so the issue does seem to be isolated the the transfer between the two hosts in the VPN link.

However, it's odd that will large file transfers for other protocols work, the replication process does not. Also, being that these are both ESXi hosts, there doesn't seem like much can be done on the hosts themselves to help fix this issue.

We may contact some WAN optimization vendors to try and see if their products can help out.

[Gostev]

Travis, have you seen this thread as it sounds exactly like your issue:
Problems with making replication work across a WAN/VPN?

[pacmantravis]

Yes, I have.

Unfortunately, most of the recommendations I have seen are for ESX servers only, not esxi. We have reduced the MTU settings on the Veeam server itself to 1400, however, still a no-go.

Ideally we would want to change the MTU settings on the ESXi hosts themselves, but that does not seem possible.

[dreddaway]

Check out this thread:
VEEAM with Riverbed

[joel0920]

[merged]

Replicating between two esxi 4.1 host over a 3Mbit wan link gives this error every time


2011-12-28 02:26:46 :: Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [120821].
Failed to replicate content of the disk [vddk://<vddkConnSpec><viConn name="192.168.0.8" authdPort="902" vicPort="443" /><vmxPath vmRef="48" datacenterRef="ha-datacenter" datacenterInventoryPath="ha-datacenter" snapshotRef="48-snapshot-617" datastoreName="xxxxxxxxxx" path="Windows Server 2003 Standard Edi/Windows Server 2003 S

The replication has been set up with Veeam 5 and worked (but slow) for several month

Now trying to speed it up with veeam 6
the job starts and is going 10 times faster but stops before it finnish, different amount every time.
at most i have processed 194 Gb out of 200 (transferred 14Gb in changes)

I have installed a clean Windows 7 VM at the source end to funktion as proxy
and tried with a 2003 backupserver and a 2008R2 SP1 veeam backup server at the destination side.
backupservers as VM:s in a separate ESX
i have a previously copied version in the destination datastore that i use as MAP replica
so just the changes need to be transfered over the wan

verified that i have 1500 MTU over the line both ways
and no blocking what so ever, all tcp , udp and icmp allowed.

help needed
/Jonny

[Vitaliy S.]

Looks like your WAN connection is pretty unstable, that is why your replication job stops randomly. Try using WAN accelerators (search these forums for Hyper-IP or Riverbed), should help.

[joel0920]

I dont know about others but mine Wan is stable and i have a Vcenter connection to the source (over the wan link) and that is never failing, even when the replication stops.

[foggy]

Johnny, please proceed investigating this with our technical support team. It's hard to troubleshoot such issues via forums and full logs are definitely required. Thanks.

[Gostev]

Connection does not drop, it is actively closed by one of the agents. Need to see the logs why.

[joel0920]

When the head is stupid.......

Im on my way of solving this, it seeams that it was the windows 7 (proxy) thats disconnect session
okey, in a standard windows 7 installation, what is default on? yes, sleep mode...
even if it,s installed in a VM server, it´s falling to sleep when it pleases..
and then it,, disconnects everything
but when i went in from Vcenter window and checked... yes it was there and kickning.
the startup from sleep is so quick that you dont see it..

So now it, s OFF and the replication is started, get back if this solved the problem..will take 10-15 hours to know for sure..

HAPPY NEW YEAR !

[Gostev]

Ouch

[joel0920]

Nope

that was NOT the issue
stopped with the same error

left the job running and it was disconnected 2-3 times (retries)
and then it suddenly was running OK ?
and this was during the holiday so no config changes...
and no Sleeping..

now im confused, now it has worked for 2-3 times during business hours

[Gostev]

Hi Jonny, this issue is environmental - some process actively terminates the network connection (OS shutdown, antivirus, etc). Thanks.

[mazcredi]

I am having the EXACT same problem as Joel, even the WAN link is the same speed. If I do a BACKUP over the WAN, it works fine over a 20-hour period but then a subsequent replication of the exact same VM fails with this connection issue. I know it is tempting to say, well the WAN is not stable, but frankly we have a host of other automated transfer jobs over the WAN that never fail due to connectivity issues. I am working with support on this but it has been a few weeks now and after trying a few different things, we still have not come to a resolution.

@Joel, I see your last post from earlier this month, were you ever able to get this straightened out?

[tsightler]

Do you have multiple replications running? Are you proxies used for other tasks?

[mazcredi]

No just a single one, both source and target proxies dedicated to this one job.

[tsightler]

Have you tried using Network mode on the target proxy?

[mazcredi]

That is the mode configured for the target.

[mysticjay]

Is there any final solution for this?
My (stopped) Ubuntu-VM fails to do subsequent replicas.
The first is Ok, when I start the job again (5 restorepoints) it fails with "Eine vorhandene Verbindung wurde vom Remotehost geschlossen. Failed to process [sendSignature]" (an existing connection was closed by remote host).

Mystic

[Vitaliy S.]

Matthias, try replicating this server locally, if it does work, then most likely there is some kind of environmental issue that actively terminates the network connection while your job is targeted to an offsite location.

[PK_GAA]

The An existing connection was forcibly closed by the remote host problem occured at our system too. It did not occur regularly but now and then, so i couldn't find out, why the connection got closed.
The wireless LAN connection to building where the replikation hardware is set up was online all the time. According to the lan monitor, the connection did not drop. Any ideas, what we could check, to find out what happened.

Some information about our system:
* 3 esxi hosts. 2 of them in HA-mode, the third is the backup host directly connected the the iscsi-backup-storage.
* on the production-system we run 10 VMs (including 1 storage-server, 1 sql-server, 1 vcenter client (which is also the veeam b&r server), 1 domain controller)
* Productive and backup hardware is set up in two buildings, connected via straight direct link wireless LAN (connection is monitored and 99.9% of time working)
* backup proxys are: the VBR server itself, the backup vcenter client, 1 VM at backup-side, 1 VM at prodictive-side, 1 physical machine. (would it be better to have more physical machines as proxys?)

[Gostev]

One idea I have - since you are using wireless LAN connection, it is possible that there was extended packet loss period due to interference. We are making some changes in Patch 1 for 6.1 (to be released in the next couple of weeks) that should make our engine more resistant to such packet loss.

[PK_GAA]

sounds promissing. we are looking forward to that.

[tietzjd]

Still seeing this error on 6.1. I set a proxy up at the remote site with 8 CPU cores and we have 100mb direct connect QMOE (Fiber Line) connecting the 2 sites that is dedicated to backups.

So far even with WAN optimization (With in Veeam ) and Proxy on each side I not achieving more than 4MB/s speed and I keep get the Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [157002].

I know I trying a rather large back up (500GB) but at 100mb direct connect from switch to switch and sup 1 MS latency I would not expect to lose the connection.

[dellock6]

Hi Joe,
please remember 100mbits line gives you a theoretical max speed of 12.5 MBs. Considering wan optimizations uses the highest compression rate, it could slow down replication right because it has to heavily compress data before sending it. So, regarding speed, you can also try and change to lan compression.
Line dropping is instead more worrying, can you try a simple copy/paste operation of a large file to check if you can reproduce the error, or if it's Veeam only?

Luca.

[Gostev]

Hi Joe, do you have patch 1 installed?

[tietzjd]

Gostev- Will add SP1 on next pass or failure. Luca- Will be running that test after this job, I rather run it till completion. Only worry is what will happen when normal backups kick off tonight. We will see.