Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
dtwiley
Enthusiast
Posts: 48
Liked: 3 times
Joined: Aug 26, 2014 1:33 pm
Full Name: Dave Twilley
Contact:

2019 Hyper-V Backup Performance (Case 03467937)

Post by dtwiley » Mar 19, 2019 12:10 pm

Hi,

I thought i'd post here to see if this would help alongside my support case.

We have just migrated some VMs from 2008 Hyper-V to 2019 Hyper-V. We ran full backups and got 42MB/s, sometimes slower. The old server would run at gigabit line speed without a problem. I decided to run a replica job instead to see if this would run quicker and it did. Results from backup and replica jobs below, they're clearly significantly different in terms of speed. This is just a test, we have much larger VMs to backup and I tried one of those and it was running at the same slow speed after an hour so i cancelled it.

All the hosts are on the same L2 network and the hyper-v and backup servers are on the same switch. Each has 4 NICs configured with LACP locally and port channels on the switch.

The network is the bottleneck for the replica job which is what we'd expect to see, however for the backup job its saying the source server is the problem but we don't know why. Are there known issues backing up from 2019 Hyper-V hosts?

Backup Job
18/03/2019 15:15:14 :: Hard disk 1 (60.0 GB) 12.8 GB read at 42 MB/s [CBT]
18/03/2019 15:20:47 :: Busy: Source 73% > Proxy 1% > Network 26% > Target 0%

Replication Job
18/03/2019 14:55:20 :: Hard disk 1 (60.0 GB) 12.8 GB read at 178 MB/s [CBT]
18/03/2019 14:57:05 :: Busy: Source 10% > Proxy 15% > Network 85% > Target 10%

Dave

dtwiley
Enthusiast
Posts: 48
Liked: 3 times
Joined: Aug 26, 2014 1:33 pm
Full Name: Dave Twilley
Contact:

Re: 2019 Hyper-V Backup Performance (Case 03467937)

Post by dtwiley » Mar 19, 2019 2:42 pm

We just recreated the environment in a lab in our office and got the same poor performance when backing up a 2019 host which had LACP configured.

We then disabled LACP and switched to a single NIC and the backup ran at full line speed.

We therefore believe this to be a Veeam/LACP specific issue, can this be tested please?

LACP - 2 NICS
19/03/2019 14:34:05 :: Busy: Source 74% > Proxy 4% > Network 25% > Target 0%
19/03/2019 14:30:55 :: Hard disk 1 (60.0 GB) 13.3 GB read at 78 MB/s [CBT]

LACP disabled - 1 NIC
19/03/2019 14:41:12 :: Busy: Source 75% > Proxy 6% > Network 25% > Target 0%
19/03/2019 14:39:17 :: Hard disk 1 (60.0 GB) 13.3 GB read at 136 MB/s [CBT]

dtwiley
Enthusiast
Posts: 48
Liked: 3 times
Joined: Aug 26, 2014 1:33 pm
Full Name: Dave Twilley
Contact:

Re: 2019 Hyper-V Backup Performance (Case 03467937)

Post by dtwiley » Mar 19, 2019 4:11 pm

FYI we've now carried out additional tests. If we remove 3 of the 4 LACP ports and leave it on just one but with LACP still configured, we get line speed without a problem. If we connect all 4 again, the performance drops again.

nmdange
Expert
Posts: 479
Liked: 120 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 2019 Hyper-V Backup Performance (Case 03467937)

Post by nmdange » Mar 19, 2019 6:09 pm

One suggestion: try running your Hyper-V host with Switch Embedded Teaming (which runs in switch-independent mode, no LACP) and see what that does. Switch Embedded Teaming is new as of Server 2016 and is the preferred teaming configuration.

dtwiley
Enthusiast
Posts: 48
Liked: 3 times
Joined: Aug 26, 2014 1:33 pm
Full Name: Dave Twilley
Contact:

Re: 2019 Hyper-V Backup Performance (Case 03467937)

Post by dtwiley » Mar 20, 2019 7:40 am

Thanks, i'll take a look. It looks a lot like VMWare's implementation of virtual switches and physical NIC handling which I think has always made more sense than Microsoft's.

Post Reply

Who is online

Users browsing this forum: cbodirca and 18 guests