Host-based backup of Microsoft Hyper-V VMs.
Dreadnought
Service Provider
Posts: 24
Liked: 2 times
Joined: May 03, 2017 3:36 pm
Full Name: Jerry Aherne
Contact:

Re: VM IO pause during backup

Post by Dreadnought »

Hi Mike, No problem, yes been through all these checks as well.
1. Yes VM is in the running state
2. Yes snapshot file location is the same location that the VMs VHD files reside
3. Yes all disks are basic disks, no dynamic disks on any of the VM's
4. Yes all disks are formated with NTFS

Hi Nmdange,
Yes we have tested using Hyper-V native Quiescence instead of Veeam Application Aware and get the same result.

as ive mentioned, we have also tested with the VM sitting on a dedicated standalone hyper-v host with the same result so i know the issue isnt cluster or storage related.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VM IO pause during backup

Post by Mike Resseler »

Hi Jerry,

I saw you are still working with support so keep doing that. I kind of am running out of ideas but wanted to give you my full list of things to check to see whether an online backup can be taken. I should have done that from the beginning, sorry about that

* The latest Integration Services must be installed and backup (volume snapshot) within those services must be running
* The guest OS inside the VM must support VSS
* Is the Snapshot File Location for the VM set to be the same volume in the host operating system as the VHD files for the VM?
* Are all volumes in the child VM basic disks and are there no dynamic disks?
* Are all disks in the child VM use a file system that supports snapshots (for example, NTFS, ReFS)?
* Following services must be running in the VM: COM+ Event System/ Distributed Transaction Coordinator/ Remote Procedure Call (RPC)/ System Event Notification/ Volume Shadow Copy. (Some are set to start automatically, others automatically (Delayed start). The COM+ System Application/ Microsoft Software Shadow Copy Provider services must at least be set to Manual
* Check inside the guest for event ID 13 with source vmicvss. If that exists, something is wrong
Dreadnought
Service Provider
Posts: 24
Liked: 2 times
Joined: May 03, 2017 3:36 pm
Full Name: Jerry Aherne
Contact:

Re: VM IO pause during backup

Post by Dreadnought »

Had a webex session with Anatoly from Veeam today. Thanks to him for confirming our thoughts on what the problem is.

New CSV created, single VM storage migrated to the csv, VM also the only VM running on cluster node, csv owned by said node.

Anatoly then ran a Shadow backup from the Hyper-V host, so not using Veeam at all.

Same pause happens and Event ID 12 logged in the system log inside the guest VM.

So the conclusion is this is a Microsoft VSS issue and it needs to be logged with Microsoft. Ive also looked at other Hyper-V VM's that run on a customer site so completly isolated from our environment and the same events occur on their VM's when the backup runs, the difference being they run one backup at night when no one is accessing the VM's so they have not seen the issue. Also plenty of posts on the internet of other people having the same issue when using other vendors backup software that uses VSS, not found any fix though that resolves the problem.

So we are now looking to upgrade our clusters to 2016 server as i have little to no faith in Microsoft fixing this issue. I will log it with them none the less but dont hold your breath.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VM IO pause during backup

Post by Mike Resseler »

Jerry,

Please let us know the outcome of the support call with Microsoft. I am convinced you will help a lot of people here also.

And thanks for doing this

Mike
Dreadnought
Service Provider
Posts: 24
Liked: 2 times
Joined: May 03, 2017 3:36 pm
Full Name: Jerry Aherne
Contact:

Re: VM IO pause during backup

Post by Dreadnought »

Hi All,

Support case has now been escalated within Microsoft support, they did suggest that this may be by design in which case i expect they wont do anything to try and resolve. Microsoft have performed the same tests on our environment that Anatoly performed so they know the issue is VSS, they did try to pass it off as its an issue with the VSS writers and various other excuses.

I have managed to reduce the pause to the VM's by doing the following:
1. make sure all VM's in a job reside on the same node.
2. make sure the CSV the VM's in the backup job reside on is owned by the Node the VM's reside on.

The two changes above have reduced the pause to a couple of seconds but in the long term this would just not be manageable once we on board the other 320 VM's. Ive also tried reducing the size of the CSV's which made no difference at all.

Ive also tested backing up a VM using the Veeam agent for Windows and this backups the VM without causing any pause at all, but again this would not be a manageable solution going forwards.

So we are still stuck at a position where we are not able to move our customer base over to use Veeam and still have a minimum commit per month on the Veeam licensing.

I'm going to try getting the software from Dell as a trial that will allow us to perform backups using Off-Host proxys to see if that resolves the issue, again not ideal as this will introduce another 30k cost which we hadn't factored in.

very very frustrating.
Also found this post from a guy called Jason that seems to have experienced the exact same issues we are having. microsoft-hyper-v-f25/hourly-backups-pe ... 30967.html

cheers
Jerry
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: VM IO pause during backup

Post by nmdange »

If the pause is related to the CSV on the host, then moving to Hyper-V 2016 should solve the issue as backing up VMs in Hyper-V 2016 no longer requires snapshotting the host volume.
Dreadnought
Service Provider
Posts: 24
Liked: 2 times
Joined: May 03, 2017 3:36 pm
Full Name: Jerry Aherne
Contact:

Re: VM IO pause during backup

Post by Dreadnought »

this isnt the case, we have tested it with a server 2016 cluster and yes we know the mechanism has changed but you still see pases in the VM.

This test was completed using completly isolated hardware as well.

We have also run these tests against standalone Hyper-V hosts with direct attached storage so nothing to do with clustering or CSV's etc and the issue is still present.

The other issue with moving to a 2016 based cluster is the changes to SPLA licensing which we use. We are currently signed up for a further two years to the per socket licensing model for 2012R2, to move to server 2016 would force us straight into per core pricing which would see around an 14k per month uplift to our licenising costs. Yes we know this is coming at some point down the road but not viable at the moment without a significant increase in costs to our customers. Well done Microsoft on that one.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: VM IO pause during backup

Post by nmdange »

Hmmm if it still happens in 2016 and on standalone hosts, then it would seem likely it's VSS within the VM itself causing the pauses and not anything on the host side. Have you tested disabling the "Backup (Volume Checkpoint)" setting under the VM integration services? This would run a crash-consistent backup so the in-guest VSS would not be involved at all. If the pause goes away then there is likely nothing you can do about it because the application within the VM simply needs that much time to flush all IO for an application-consistent backup.
kcm_aaron
Lurker
Posts: 2
Liked: never
Joined: Mar 27, 2017 4:19 pm
Full Name: Aaron
Contact:

Re: VM IO pause during backup

Post by kcm_aaron »

Just wanted to follow up on this - better late than never, right?

We ended up disabling ODX, but I don't think we were using it anyway, so we continued to see "crashed" servers afterwards. We are currently in the process of migrating to Veeam, about halfway there, and we are not seeing the issue with any of the Veeam backup or replication jobs, but we did have to switch to the native Microsoft CSV VSS writer. It seems like it takes maybe 15-20% longer for the backup jobs to run using this VSS writer, but well worth it for us to not have servers randomly crash due to the I/O lockup.
pterpumpkin
Enthusiast
Posts: 36
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: VM IO pause during backup

Post by pterpumpkin »

Sorry for dragging up an old post, but hopefully this will be helpful to someone in the future.

We had NIGHTMARES with this issue and lots of angry clients. We opened multiple Veeam and MS cases and basically got told "that's just how it is on 2012R2". Using our SAN vendors VSS provider prevent the VM from pausing on completion of the backup, but not the start. Using our SAN vendors VSS provider was also saw OTHER VM's on the same CSV crash, even if they weren't being backed up.

Basically... our only solution was to move to 2016 as it doesn't snapshot the CSV, just the individual VM. So far 2016 has been working well for us :)
ptoro
Influencer
Posts: 21
Liked: 5 times
Joined: Jul 24, 2015 4:41 pm
Contact:

Re: VM IO pause during backup

Post by ptoro »

Backing up pterpumpkin here, we also has IO pause issues on the CSVs UNTIL we:

1. upgraded the hyperv servers to 2016
2. upgraded the vm's config to version 8 (to allow for RCT)

Once we did that, VM backups we FAST and CSVs we happy.

Backend is SMB storage is still Windows 2012R2 SOFS
Post Reply

Who is online

Users browsing this forum: reiso22 and 17 guests