VM IO pause during backup

Hyper-V specific discussions

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Tue May 23, 2017 7:03 am

Hi Mike, No problem, yes been through all these checks as well.
1. Yes VM is in the running state
2. Yes snapshot file location is the same location that the VMs VHD files reside
3. Yes all disks are basic disks, no dynamic disks on any of the VM's
4. Yes all disks are formated with NTFS

Hi Nmdange,
Yes we have tested using Hyper-V native Quiescence instead of Veeam Application Aware and get the same result.

as ive mentioned, we have also tested with the VM sitting on a dedicated standalone hyper-v host with the same result so i know the issue isnt cluster or storage related.
Dreadnought
Service Provider
 
Posts: 24
Liked: 1 time
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Tue May 23, 2017 8:42 am

Hi Jerry,

I saw you are still working with support so keep doing that. I kind of am running out of ideas but wanted to give you my full list of things to check to see whether an online backup can be taken. I should have done that from the beginning, sorry about that

* The latest Integration Services must be installed and backup (volume snapshot) within those services must be running
* The guest OS inside the VM must support VSS
* Is the Snapshot File Location for the VM set to be the same volume in the host operating system as the VHD files for the VM?
* Are all volumes in the child VM basic disks and are there no dynamic disks?
* Are all disks in the child VM use a file system that supports snapshots (for example, NTFS, ReFS)?
* Following services must be running in the VM: COM+ Event System/ Distributed Transaction Coordinator/ Remote Procedure Call (RPC)/ System Event Notification/ Volume Shadow Copy. (Some are set to start automatically, others automatically (Delayed start). The COM+ System Application/ Microsoft Software Shadow Copy Provider services must at least be set to Manual
* Check inside the guest for event ID 13 with source vmicvss. If that exists, something is wrong
Mike Resseler
Veeam Software
 
Posts: 3150
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Wed May 24, 2017 4:04 pm

Had a webex session with Anatoly from Veeam today. Thanks to him for confirming our thoughts on what the problem is.

New CSV created, single VM storage migrated to the csv, VM also the only VM running on cluster node, csv owned by said node.

Anatoly then ran a Shadow backup from the Hyper-V host, so not using Veeam at all.

Same pause happens and Event ID 12 logged in the system log inside the guest VM.

So the conclusion is this is a Microsoft VSS issue and it needs to be logged with Microsoft. Ive also looked at other Hyper-V VM's that run on a customer site so completly isolated from our environment and the same events occur on their VM's when the backup runs, the difference being they run one backup at night when no one is accessing the VM's so they have not seen the issue. Also plenty of posts on the internet of other people having the same issue when using other vendors backup software that uses VSS, not found any fix though that resolves the problem.

So we are now looking to upgrade our clusters to 2016 server as i have little to no faith in Microsoft fixing this issue. I will log it with them none the less but dont hold your breath.
Dreadnought
Service Provider
 
Posts: 24
Liked: 1 time
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Thu May 25, 2017 4:33 am

Jerry,

Please let us know the outcome of the support call with Microsoft. I am convinced you will help a lot of people here also.

And thanks for doing this

Mike
Mike Resseler
Veeam Software
 
Posts: 3150
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Thu Jun 08, 2017 8:48 am

Hi All,

Support case has now been escalated within Microsoft support, they did suggest that this may be by design in which case i expect they wont do anything to try and resolve. Microsoft have performed the same tests on our environment that Anatoly performed so they know the issue is VSS, they did try to pass it off as its an issue with the VSS writers and various other excuses.

I have managed to reduce the pause to the VM's by doing the following:
1. make sure all VM's in a job reside on the same node.
2. make sure the CSV the VM's in the backup job reside on is owned by the Node the VM's reside on.

The two changes above have reduced the pause to a couple of seconds but in the long term this would just not be manageable once we on board the other 320 VM's. Ive also tried reducing the size of the CSV's which made no difference at all.

Ive also tested backing up a VM using the Veeam agent for Windows and this backups the VM without causing any pause at all, but again this would not be a manageable solution going forwards.

So we are still stuck at a position where we are not able to move our customer base over to use Veeam and still have a minimum commit per month on the Veeam licensing.

I'm going to try getting the software from Dell as a trial that will allow us to perform backups using Off-Host proxys to see if that resolves the issue, again not ideal as this will introduce another 30k cost which we hadn't factored in.

very very frustrating.
Also found this post from a guy called Jason that seems to have experienced the exact same issues we are having. microsoft-hyper-v-f25/hourly-backups-performance-t30967.html

cheers
Jerry
Dreadnought
Service Provider
 
Posts: 24
Liked: 1 time
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby nmdange » Thu Jun 08, 2017 7:05 pm

If the pause is related to the CSV on the host, then moving to Hyper-V 2016 should solve the issue as backing up VMs in Hyper-V 2016 no longer requires snapshotting the host volume.
nmdange
Expert
 
Posts: 191
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Fri Jun 09, 2017 7:39 am

this isnt the case, we have tested it with a server 2016 cluster and yes we know the mechanism has changed but you still see pases in the VM.

This test was completed using completly isolated hardware as well.

We have also run these tests against standalone Hyper-V hosts with direct attached storage so nothing to do with clustering or CSV's etc and the issue is still present.

The other issue with moving to a 2016 based cluster is the changes to SPLA licensing which we use. We are currently signed up for a further two years to the per socket licensing model for 2012R2, to move to server 2016 would force us straight into per core pricing which would see around an 14k per month uplift to our licenising costs. Yes we know this is coming at some point down the road but not viable at the moment without a significant increase in costs to our customers. Well done Microsoft on that one.
Dreadnought
Service Provider
 
Posts: 24
Liked: 1 time
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby nmdange » Fri Jun 09, 2017 2:48 pm

Hmmm if it still happens in 2016 and on standalone hosts, then it would seem likely it's VSS within the VM itself causing the pauses and not anything on the host side. Have you tested disabling the "Backup (Volume Checkpoint)" setting under the VM integration services? This would run a crash-consistent backup so the in-guest VSS would not be involved at all. If the pause goes away then there is likely nothing you can do about it because the application within the VM simply needs that much time to flush all IO for an application-consistent backup.
nmdange
Expert
 
Posts: 191
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: VM IO pause during backup

Veeam Logoby kcm_aaron » Sat Aug 12, 2017 3:28 am

Just wanted to follow up on this - better late than never, right?

We ended up disabling ODX, but I don't think we were using it anyway, so we continued to see "crashed" servers afterwards. We are currently in the process of migrating to Veeam, about halfway there, and we are not seeing the issue with any of the Veeam backup or replication jobs, but we did have to switch to the native Microsoft CSV VSS writer. It seems like it takes maybe 15-20% longer for the backup jobs to run using this VSS writer, but well worth it for us to not have servers randomly crash due to the I/O lockup.
kcm_aaron
Lurker
 
Posts: 2
Liked: never
Joined: Mon Mar 27, 2017 4:19 pm
Full Name: Aaron

Previous

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: nmdange and 5 guests