Fast Backup but slow replication

[Fwiler]

Two hyper-v hosts. Dell 730xd all ssd direct attached storage RAID 10. 10GB nics (not rdma)
Host1 running production vms. Host2 is exact same hardware housing backups.
Host2 runs Veeam as a vm
High availability is not the concern, but quick recovery is.

Backup jobs transfer at a good 800-900MB/s.
Replication avg about 50-100MB/s

Is replication that much slower? Should I just skip replication?
If it should be faster how to troubleshoot?
Load Source: 10% > Proxy 1% > Network 4% > Target 5% which is about the same for all vm's.
Host1 is underutilized on cpu, mem, and storage, so use on-host proxy. Not sure if using a 3rd server as a proxy would help any.

[Mike Resseler]

Dustin,

I doubt that a 3rd server as proxy will help. There is indeed a rather big difference in the transfer speed and that doesn't look normal. Any change the replication is using a different network?

[Fwiler]

Both servers are using dual port 10Gb nics, created as a team from Server Manager. Virtual Switch created on top of this.
Perhaps this is my problem? Although there is very very little traffic going on in our small environment.

[Mike Resseler]

Hey Dustin,

Sorry for my late reply (Have been travelling). It should not be a problem, but if you can, during non-production hours, it would not be bad if you could test this without the team.

[Fwiler]

No worries, I know travelling is a job in itself.
I've actually reinstalled the OS and hyper-v on both servers, upgrading to 2019. Storage is now using ReFS.
The vm's are now using a 4port 1Gb team. I've left the 10Gb nics with the same IP addresses that Veeam uses, but no longer in a team.

Replication is showing ~100MB/s read under ACTION screen in Veeam. Source is only 9%
Backup job is showing ~3-400MB/s read under ACTION screen. Source is 88%
I'm not sure if this source % and MB/s difference gives you any clue to what is happening. It's almost like Replication job is limiting the source % used, and maybe this is on purpose?

Or maybe because Veeam is running in a VM on the destination server. The data doesn't flow through Veeam does it? That would explain it, if that is true.

Otherwise I'm going to blame the Qlogic nics (broadcom). Iperf shows full bandwidth used at ~9Gb/s, but that doesn't mean everything. Maybe I'll move the veeam vm to the source server and see if that makes any difference.

[Fwiler]

This ended up being a network issue. For anyone reading, do not use qlogic (old broadcom) BCM57810 cards. These came with our Dell servers, but did not realize the issues these have had.
I eventually found a driver from May 2017 that made all the difference. 34.04 I believe.
Now I'm at a good 600-650MB/s for replication and bit higher for backup jobs.

It doesn't explain the difference between Replication jobs and Backup jobs, but I'm not going to try to figure out how the network was affecting Veeam at this point.