VM IO pause during backup

Hyper-V specific discussions

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Tue Mar 28, 2017 6:55 am

Aaron,

You make a good point here... ODX. ODX was (sorry is) a good idea and could be very useful and then it got really badly implemented because it is a combination of Microsoft and the storage/ hardware vendor that needs to work together (probably the reason of the fingerpointing...)

In the past many told us that disabling ODX indeed helped and solved the issues. So a good next test to run.

Are you also having other offloading techniques running/ activated? Those could also be an issue

PS: How was the performance for the software VSS provider? If that is acceptable, why not consider using that method instead of staying with the hardware VSS provider that is giving you issues?
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby nismoau » Fri Mar 31, 2017 3:04 am

kcm_aaron wrote:Next, I will be disabling ODX on my Hyper-V hosts to see if that has any impact


We're also experiencing just about the exact same issue as you describe. Let us know how disabling ODX goes - I'm very interested! :)

We have also been able to isolate VM lockups during backups to happen only when the SAN Hardware VSS provider is used. Changing to using the Microsoft CSV VSS provider does not (seem to) exhibit the VM lockup issue - as I read it, very similar to what you're seeing.

Cheers!
Justin
nismoau
Novice
 
Posts: 7
Liked: never
Joined: Wed Jul 06, 2016 1:29 am

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Tue May 09, 2017 12:47 pm

Hi,

Were suffering the same issue as well.

We have a 16 node Hyper-V cluster, all nodes are 2012R2, each node is dual 16 core CPU's with 384GB ram, resources on the cluster are not an issue, lots of performance monitoring and testing has been conducted to rule this out, ODX has always been disabled as its dangerously unreliable, we are using Compellent SANs, IO latency is very low, we dont have any performance issues with almost 400 VM's running on the cluster.

Weve recently introduced Veeam to replace StorageCraft which had no issues with VSS.

At the moment we have only migrated our own servers to Veeam but we are seeing the VM's hang when VSS snapshots take place and at the end of the backup, also the VM's can hang when they arent backing up but others are that are on the same CSV, most noticeable on Exchange and anything with a Database.

We are using on host proxy's.

Currently going through every node to see if we can find anything but nothing so far, all patched, all NIC offload settings disabled, ODX disabled, Compellent Firmware up to date.

This has put our entire migration project to Veeam on hold until we can resolve the problem, it would seem that in this instance Veeam just doesn't work.

I'll post here if we find anything that resolves the problem.
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Tue May 09, 2017 5:51 pm

Hi Jerry,

Did you already created a support case?

Mike
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Wed May 10, 2017 3:28 pm

Hi Mike,

We havent yet as we are going through everything on our side first to make sure there isnt anything out of place that might be causing the issue. This has highlighted a lot of issues with windows updates on the cluster nodes where they are at different levels of patching. Unfortunatley due to the way Microsoft manage their rollups we arent able to bring them all up to the same level as they are so we are having to rebuild all the nodes, bit of a ball ache but sorts that problem out at least.

We now have some of the nodes back in the cluster fully patched, ODX off etc etc as i previously mentioned, our compellents are fully patched as is our switch fabric.

Ive created a new CSV and storage migrated all our own VMs to the CSV (17 in total that run in a single backup job), moved the csv to one of the fully patched nodes and live migrated all our VM's to that same node. Backup job has then been started, initial testing looked promising, didnt see any hangs in Outlook as Exchange is very suseptable to the pauses. The backup job has been left running and we have seen some pauses although not as bad as before, this could be because of the reduction in running jobs or the fact that ive moved our VM's to an isolated CSV.

Until we have all the nodes back in the cluster i dont think we can say one way or the other if we have found the issue. Ill post results here once we do have them all back in and i can conduct further testing, if at that point the issue is still present then i'll raise a support case.
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Thu May 11, 2017 1:32 pm

Hi Mike,

We have completed further testing.

Nodes fully updated, VM's now using the latest version of Integration Services, all VM's reside on same cluster node and same CSV, CSV is owned by the node the VM's reside on.
Pause during snapshotting is still present, doesnt appear to be as bad as it was but its still not going to be possible for us to move any of our customers to the platform as it stands. I'm going to log a support ticket today with Veeam.

If anyone has any ideas on how to resolve this issue i'm all ears.

thanks
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Thu May 11, 2017 1:40 pm

Jerry,

Thanks for the updates. Please log the support case here and the result after investigation with support

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Thu May 11, 2017 2:24 pm

Hi Mike,

Support case raised, log files submitted with case.

Case #02153463 was created.

thanks
Jerry
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Mon May 15, 2017 10:44 am

Thanks Jerry,

Keep us informed about the outcome please!

Cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Mon May 15, 2017 11:11 am

will do.

I have a question regarding the GIP server that may or may not be contributing to the problem. Should a dedicated server be used as the GIP or is it ok to use an existing server that performs other roles.
The reason i ask is that we initially tested with a dedicated GIP in the customer network, monitoring showed that usage on the dedicated GIP was barely noticable when backup jobs were running and the GIP was performing injection of the runtimes so we decided to use an existing server within the customer network to perform the role of GIP (reduces costs etc).

This then means that the server performing the GIP role is also included in that customers backup job so its performing snapshots of itself when the backup runs as well as dealing with runtime injection for all the other servers in the job and dealing with preperation of application aware processing. Not been able to find much info regarding best practice of the GIP server and what should and shouldnt be used.

Should we be using dedicated servers to perform the GIP role, that arent included in the backup jobs?
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Mon May 15, 2017 11:16 am

Actually. No. It shouldn't be a dedicated machine. In fact, in my setup my hosts are GIP proxies also.

The only thing is what will happen when the VM is snapshotted who is at the same time the GIP proxy. I am not sure what happens then as I don't have that specific setup. So ask that to support (And I would like to know the answer also :-))
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Mon May 15, 2017 3:51 pm

well just to rule it out ive built a dedicated GIP on the customer network.

Hasnt made the slightest difference. I had the backups running this morning without any pause (6 seconds) in the VM. Now i'm back to the 6 second pause again. The VM is a busy exchange server admitedly but its

Ive moved all the Veeam servers to the same node that the VM's to be backed up reside on so they use the Virtual switch rather than breaking out onto the switch infrastructure to rule out any type of network latency (network has been checked and there are no performance issues or bottlenecks). All VM's reside on the same CSV and the same cluster node, the CSV is also owned by the node. Perfmon shows no performance issues on the cluster node, the CSV shows very low read and write latency so i know i dont have a performance issue anywhere.

All the VSS writers checkout ok on the VM, Shadowsstorage has plenty of space, everything works as it should do with no errors anywhere yet i get a 6 second pause inside the VM when it thaws the snapshot. If i dnot do an application aware backup and use the Crash consistent backup it works fine, no pause and looking at the event log it shows everything happening the same as if i was useing an application aware, snapshoting freezes and thaws all in the space of 1 second and the logs all get truncated.

frustrating to say the least.
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Dreadnought » Mon May 22, 2017 4:10 pm

Hi,

So further testing done. to rule out an issue with the Cluster nodes and underlying compellent storage ive migrated our exchange server to a standalone host, the host has been added to the Veeam backup infrastructure. New backup job created for our exchange server that is on the standalone host with direct attached storage. Host is dual CPU 24 cores, 192 GB RAM, 24 x 146GB 15k disks so its by no means slow and the only VM is the exchange server.

Start the new backup job, snapshot process starts, pauses experienced in outlook, there are always 2 pauses as well. So the exact same behaviour on a standalone host.

Next test, our veeam platform sits behind a Virtual PFSense firewall which segregates it from our cloud platform so to rule the Pf Sense out of the equation ive multi-honed the B&R and Repo servers so they now sit in the cloud platform and can talk directly between the two without touching the virtual firewall. Again backup tested and the pause is experienced exactly as before.

In addition to the above i have also changed settings within the B&R server as per Veeams support request, so Storage Latency control enabled and set to 10ms on both settings, Max concurrent snapshots set to 1 on all volumes, Max tasks on the On Host Proxy's set to the default of 4.
Again ran the backup with all of the above settings reduced as well as the B&R and the Repo in their new configuration as well as the old configuration with no change what so ever, we still get the pause.

According to this article https://www.veeam.com/kb1896 The cause of the pause can be caused by Hyper-V 2012 R2 using a saved state backup if Online backup is not available, The saved state backup process causes the system up-time counter in Hyper-V manager to reset, which we are seeing, why we are seeing it i have not the foggiest idea or how to find out why its doing it and how to stop it doing it.

completely out of ideas now with this one. Anyone have any good suggestions as to where to look to try and resolve this will be appreciated
Dreadnought
Service Provider
 
Posts: 24
Liked: 2 times
Joined: Wed May 03, 2017 3:36 pm
Full Name: Jerry Aherne

Re: VM IO pause during backup

Veeam Logoby Mike Resseler » Mon May 22, 2017 5:40 pm

Just to be sure and on the safe side (if you already looked at it with our support team, I apologize for the double question...)

1. Is the child VM must in the running state. (I am sure it is yes but this is part of my standard research :-))
2. Is the Snapshot File Location for the VM set to be the same volume in the host operating system as the VHD files for the VM?
3. Are all volumes in the child VM basic disks and are there no dynamic disks?
4. Are all disks in the child VM use a file system that supports snapshots (for example, NTFS, ReFS)?
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VM IO pause during backup

Veeam Logoby nmdange » Mon May 22, 2017 6:21 pm

Have you tested using Hyper-V Native Quiescence instead of Veeam Application-Aware processing?
nmdange
Expert
 
Posts: 194
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 5 guests