2019 Hyper-V Backup Performance (Case 03467937)

[dtwiley]

Hi,

I thought i'd post here to see if this would help alongside my support case.

We have just migrated some VMs from 2008 Hyper-V to 2019 Hyper-V. We ran full backups and got 42MB/s, sometimes slower. The old server would run at gigabit line speed without a problem. I decided to run a replica job instead to see if this would run quicker and it did. Results from backup and replica jobs below, they're clearly significantly different in terms of speed. This is just a test, we have much larger VMs to backup and I tried one of those and it was running at the same slow speed after an hour so i cancelled it.

All the hosts are on the same L2 network and the hyper-v and backup servers are on the same switch. Each has 4 NICs configured with LACP locally and port channels on the switch.

The network is the bottleneck for the replica job which is what we'd expect to see, however for the backup job its saying the source server is the problem but we don't know why. Are there known issues backing up from 2019 Hyper-V hosts?

Backup Job
18/03/2019 15:15:14 :: Hard disk 1 (60.0 GB) 12.8 GB read at 42 MB/s [CBT]
18/03/2019 15:20:47 :: Busy: Source 73% > Proxy 1% > Network 26% > Target 0%

Replication Job
18/03/2019 14:55:20 :: Hard disk 1 (60.0 GB) 12.8 GB read at 178 MB/s [CBT]
18/03/2019 14:57:05 :: Busy: Source 10% > Proxy 15% > Network 85% > Target 10%

Dave

[dtwiley]

We just recreated the environment in a lab in our office and got the same poor performance when backing up a 2019 host which had LACP configured.

We then disabled LACP and switched to a single NIC and the backup ran at full line speed.

We therefore believe this to be a Veeam/LACP specific issue, can this be tested please?

LACP - 2 NICS
19/03/2019 14:34:05 :: Busy: Source 74% > Proxy 4% > Network 25% > Target 0%
19/03/2019 14:30:55 :: Hard disk 1 (60.0 GB) 13.3 GB read at 78 MB/s [CBT]

LACP disabled - 1 NIC
19/03/2019 14:41:12 :: Busy: Source 75% > Proxy 6% > Network 25% > Target 0%
19/03/2019 14:39:17 :: Hard disk 1 (60.0 GB) 13.3 GB read at 136 MB/s [CBT]

[dtwiley]

FYI we've now carried out additional tests. If we remove 3 of the 4 LACP ports and leave it on just one but with LACP still configured, we get line speed without a problem. If we connect all 4 again, the performance drops again.

[nmdange]

One suggestion: try running your Hyper-V host with Switch Embedded Teaming (which runs in switch-independent mode, no LACP) and see what that does. Switch Embedded Teaming is new as of Server 2016 and is the preferred teaming configuration.

[dtwiley]

Thanks, i'll take a look. It looks a lot like VMWare's implementation of virtual switches and physical NIC handling which I think has always made more sense than Microsoft's.