Backup copy job across site to site keeps dropping

[mvalpreda]

Have an issue where backup copy job from our office to the owner's home is failing. Mixed environment of Hyper-V and VMWare. The message we keep seeing in the logs is 'Error: Source WAN accelerator error: Unstable connection: unable to transmit data.' and also seeing 'Error: Target WAN accelerator error: An existing connection was forcibly closed by the remote host'. Have also tried the job without the WAN accelerator with the same results....same with Windows Firewall on/off/disabled. There is no third-party A/V on the repository machines. The bottleneck will go between source WAN and network.

The backup copy job just stops. The 5 minute throughput meter will drop to nothing and the job will eventually time out. The server on the other side is up, accepting connections via RDP, ICMP, file shares, etc. There's a telnet-type connection that is going between the locations that does not drop - and it will drop for any little packet hiccup. After the job inevitably fails, the next job up will kick off and start transmitting data with no issue.

What is interesting is the 'smaller' VMs copy over (typically) just fine - these are <100Gb. 'Larger' VMs bomb out consistently - these are >1Tb.
What is also interesting is I don't have much issue with VMWare jobs - but these are smaller VMs.

The source and destination server are both fully patched Server 2019.

Support case: 05958490

[Mildur]

Hi Mark

Please keep working with your support engineer. It's not possible to analyze this behavior over the forum.
If you are unhappy about how the case goes, you can use our "escalate"-option:
https://www.veeam.com/kb2320

The backup copy job just stops. The 5 minute throughput meter will drop to nothing and the job will eventually time out.

I had that once with backup copy jobs to Cloud Connect. In the end we had to replace the firewall. The firewall was old.

from our office to the owner's home is failing

You should check with the ISP of your "owners's home" if they provide a stable network connection. I don't expect "home internet" for business services as stable enough. Or does your owner have a business internet service at home?

Best,
Fabian

[mvalpreda]

I posted on here to see if anyone had run into something similar before. Maybe there was some strange setting in Server 2019 that closes connections that are open too long or something else silly like that.

While it is coax, it is on their business class service.

[mvalpreda]

This case is still open. I have updated to v12 with no change. We have replaced firewalls, switches, computers, storage.....no change. Any other type of communication between the sites has no issue....just Veeam. I see transfers going over port 2500 between the source and destination repository. When it stops working, port 2500 is still open and listening on the remote side.

5/2/2023 3:47:13 PM :: Error: Unstable connection: unable to transmit data.
Failed to upload disk 'DC01-C.vhdx'
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: End of file
Unable to retrieve next block transmission command. Number of already processed blocks: [5699].
Failed to download disk 'DC01-C.vhdx'.

This worked fine on v9, v10, v11...and then somehow died with v11 and still doesn't work with v12. I'm having a hard time thinking it is something besides Veeam since any other type of traffic works fine...including manually copying Veeam VBK files between repositories.

[mvalpreda]

Just for kicks I did a straight Windows file copy of a 400GB VBK file from the source repository to the destination repository. It has been running for 3 hours and has not bombed. During that time, my Veeam backup copy job has stalled.

Veeam still shows this going for nearly 2 hours now.
5/2/2023 3:47:53 PM :: Hard disk 1 (500 GB) 2 GB read at 385 KB/s

This is more typical
5/2/2023 2:59:10 PM :: Hard disk 1 (500 GB) 3.1 GB read at 8 MB/s
That was <7 minutes to offload from the same backup copy job.

The throughput meter is not updating, it is showing 0 KB/s, and still shows red/green lines but not updating. How can I see what Veeam is doing or getting hung up on?

[karsten123]

Did you set av excludes according to kb1999 for Windows Defender or did you uninstall Defender?

[mvalpreda]

Defender is uninstalled.

Interesting is that I set up a new job and a new backup copy job. It bombed on the backup copy job in the same place.
Unable to retrieve next block transmission command. Number of already processed blocks: [2232].

I set up a new repository (Linux hardened XFS) and another new backup copy job. Backup copy job bombed in the same place.
Unable to retrieve next block transmission command. Number of already processed blocks: [2232].

Other backup copy jobs are running without incident. Just this one VM.

[BackupBytesTim]

I've seen the same error message before on different computers in our environment, though not recently. I'm curious if your support case has turned up anything since it's been an ongoing issue for a few months now.

[mvalpreda]

Set up a new repository for the remote side and same result. One VM refuses to backup copy no matter what. Tried it with WAN acceleration on and off....same result. Veeam just chokes on the backup copy. Did this with v11 and v12.

[BackupBytesTim]

When we had this same problem with regular backup jobs, not backup copy jobs, we ended up doing a new full backup. Based on the fact that it keeps happening with the same VM backup file, it sounds like a reasonable solution. Though it doesn't really explain what the problem is. In my experience most of the Veeam functionality is way over complicated, so it wouldn't surprise me that there's some metadata corruption or something similar that's causing it to fail because Veeam is processing the file in some way beyond just sending the file bytes from one repository to another.

I'll be watching, I can't help much, but I'm curious to see if you come to a solution.

[mvalpreda]

The original backup job with all of our 15 VMs is fine....including the one that will not backup copy.
Have set up brand new jobs for this one VM to the original repo and a new repo - the backup jobs are fine. The backup copy job bombs in nearly the same place no matter the source job.
Set up a new remote repo and bombs out at nearly the same place with each backup copy job source.

The only thing (that i know of) we have not done is anything with the database. I don't know if a backup/restore might be something, or moving from SQL to Postgres.