Discussions specific to the Microsoft Hyper-V hypervisor
mkaec
Expert
Posts: 336
Liked: 78 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by mkaec »

If I remember correctly, Veeam B&R gets upset when it sees an old VM version like that on Windows Server 2016 or 2019. When I upgraded the 2012 R2 Hyper-V hosts to 2016, I had planned to leave the VM version at the old value for awhile to provide an easy way to go back if things didn't work out. But Veeam V&R complained about it and I changed the plan to upgrade the VM version immediately after the host upgrade.

Nick-SAC
Enthusiast
Posts: 49
Liked: 7 times
Joined: Oct 27, 2017 5:42 pm
Full Name: Nick
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Nick-SAC »

Thanks NMDANGE & Marc K,

I wasn’t really planning on trying to downgrade the existing/production VMs but I am toying with the idea of creating an older version Test VM just to see what happens... because I just don’t have enough things to waste my life on right now! :roll:

Nick

mgreenintellicomp
Lurker
Posts: 2
Liked: never
Joined: May 17, 2020 1:08 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by mgreenintellicomp »

Hey everyone

Can you confirm if your CSV file systems are running redirected or direct IO mode?

Execute the command Get-ClusterSharedVolumeState. I found out that we had setup ours as Refs which is not a good idea. We converted to ntfs and now our state is direct and most of our issues if not all are resolved. I can post more info with source links when I get to a computer.

pterpumpkin
Enthusiast
Posts: 25
Liked: 2 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

We are experiencing this too.

Virtual Host hardware: Lenovo x240 M5
Virtual Host OS: Windows Server 2019 Datacentre
Storage: Nimble CS3000

VM OS: Windows Server 2016 Standard
VM Size: ~2.5TB
VM File System: NTFS (no dedup)

One thing we noticed is that the RCT file for the large ~2.34TB disk is 500MB, which seems quite large?

The issue does not happen for us while backing up, but instead around the morning logon time (the server hosts user profiles).

We tested doing a Live Migration to another host and the issue seems to have disappeared, although it has only been one day since we did this.

giomoda
Lurker
Posts: 1
Liked: never
Joined: Jun 20, 2020 6:41 pm
Full Name: Giovani Moda
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by giomoda »

MatsL wrote:
Feb 07, 2020 8:51 am
Hi.
We have the same problem in 4 different clusters after Veeam Backup, two different i-SCSI SAN (Dell Compellent & Hitachi VSP G370).
Hosts: DELL R640 (latest patches)
OS: Windows Server 2019 (latest patches)
Storage: CSV, I-SCSI, Dell Compellent & Hitachi VSP G370
Backup: Veeam 9.5 u4

We have found a work around, if we move the VM that has problems (usually the SQL data disk, but no other disks are affected, weird) to another Host then we get normal performance again.
We have a script that checks the disk performance every two hours, and finds it a server (VM) with poor performance, then the script moves the server to another host in the cluster.
We suffer quite a lot from this.
/Mats
Hey Mats.

Would you care sharing that script? I have a setup very similar to yours and I'm suffering from constant performance issues, to a point where the VM is rebooted by the cluster. Maybe this can help me work around it until Microsoft decides to consider this an issue worth adressing.

Regards,

Giovani

ChristineAlexa
Novice
Posts: 3
Liked: never
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

I'm having the same issue. I can give some more details as well

- All servers experiencing it are upgrade installs from Windows Server 2016 GA (fully patche) to Windows Server 2019.
- It IMMEDIATELY happened after the upgrade.
- Disabling using RCT/CBT for all backup jobs on the volumes eliminates the issue.
- I've tried deleting/recreating volumes (and storage pools) for some of the affect machines, with no effect on the issue
- Setting the throttling to low values didn't seem to have an effect (not surprised since it looks like a FS bug with MS)
- Servers are
- 2 x Dell Poweredge R720 with H710p controllers
- 1 x Dell Poweredge R930 with H730p controllers (in 12x12 mode with dual cards)

Hope that helps narrow the issue

Nick-SAC
Enthusiast
Posts: 49
Liked: 7 times
Joined: Oct 27, 2017 5:42 pm
Full Name: Nick
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Nick-SAC »

Christine,

I just tried disabling CBT on all of the configured Jobs but I’m still getting the same I/O Delays as before; both during and not during the Backup Jobs, in fact most of them were not concurrent with running jobs.

Following your 2016-2019 upgrade; did you bring your VMs up to v9 or are they still on v8 ?

Thanks,
Nick

ChristineAlexa
Novice
Posts: 3
Liked: never
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

I'm on V10 already actually. And have been since V10s first patch. The upgrade was from 9.5b (whatever the latest patch a few months ago)

The Windows upgrade was 1-2 months after the Veeam upgrade

Also, these are *NOT* clustered (yet.... about to convert to S2D), so things like CSV aren't part of the equation either.

Nick-SAC
Enthusiast
Posts: 49
Liked: 7 times
Joined: Oct 27, 2017 5:42 pm
Full Name: Nick
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Nick-SAC »

Thanks Christine but I wasn’t questioning the Veeam VBR version (although that’s potentially helpful to know too).

I’d like to know what VM Machine/Configuration version you’re on (as seen in the Hyper-V Manager Console). Did you upgrade that from 8 to 9 when you did the Server 2016-2019 upgrade and if so, did you do so before or after you saw the I/O Delays?

Thanks again,
Nick

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests