Host-based backup of Microsoft Hyper-V VMs.
Post Reply
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

High latency with backup on CSV

Post by jdud »

Hello,

We have a problem with backup and only backup on a CSV storage Windows 2012R2 up to date with last KB.
On backup with Veeam we have high latency up to 5 seconds and sometimes 15 seconds !
Perfmon show latency only on IO Redirect. So it seems there is no problem with storage.
And IO redirect show high latency also with only one node !
I dont understand why there is IO Redirect when we have only one node UP. Why CSV dont make direct IO ?

Do you know issue ?

Thank a lot ;)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. »

Do you use software or hardware VSS provider? Also not sure I fully understand when you say that only one node is running. Can you please clarify this?
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud »

Hello,

We use software VSS provider. With our ZFS san we dont have vss hardware provider.
I was thinking that if we have only one node UP it was not possible to have IO redirect. But even only one node up there is IO redirect with high latency.
Today i use Perfmon and the "Direct IO Failure redirection" counter is not incremented. So this is the proof that the latency cause is between CSV and HyperV.

Also Get-StorageReliabilityCounter show latency up to 50ms.

Thank you for you help Vitaliy.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. »

What node is the current owner of the CSV volume in your setup? You can check that via Failover Cluster settings.
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud »

Node1 is the owner of the CSV.
When node1 and node2 are up we have high latency on redirected IO.
When node1 is alone (node2 is halted) we have same high latency on redirected IO.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. »

That's weird. Did you ever have direct IO mode working? Do you see this event on the Hyper-V host by any chance - Event ID = 5125?
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud »

Yes direct IO is working very good.
We have about 30 vm working with very good performance.
The redirected IO happens only while backup is running.

I have sometimes 5120 and 5217 but no 5125. Also there is 10172 and 16010.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. »

I meant to say if ever had a direct IO working during backup job. Assuming that all below is correct, then please contact our technical support team to review your setup:

1. You're using Software VSS provider
2. You don't have Event ID = 5125 on the Hyper-V node acting as the owner of CSV
3. When one host is shut down (not owner of the CSV) and you start to backup VMs residing on the CSV owner host and still observe redirected IO all the time

After reviewing your setup our team should be able to understand what is causing this behavior of the Hyper-V host. Since we do not control this process (redirecting traffic), it would be beneficial for all community members to know the the results of your investigation.
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud »

Thank you Vitally, we have opening a case on Veeam support and we will post issue in this thread (If we find it !).
ptoro
Influencer
Posts: 21
Liked: 5 times
Joined: Jul 24, 2015 4:41 pm
Contact:

Re: High latency with backup on CSV

Post by ptoro »

Was anything found?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: High latency with backup on CSV

Post by veremin »

The OP has never come back to update the topic with the found resolution, nor has he posted the ticket ID so we can track the case.

Thus, it's recommended to open your own ticket.

Thanks.
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud »

Hi all,

No solution for this problem without using hardware provider.
We have this problem with 4 SAN (10Gpbs, infiniband and FC) : ZFS san, P2000, msa2000 and HP 3PAR
This happens only with CSV on Windows 2012 r2 cluster. And the IO stats in Windows shows latency ONLY on IO redirect, but the CSV status never change to "io redirect".

This is probably a CSV bug.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: High latency with backup on CSV

Post by Shestakov »

Hi Julien,
Thanks for updating. Could you provide your support case number for us to get the details? Thanks!
RWeiss
Enthusiast
Posts: 33
Liked: 2 times
Joined: Nov 13, 2014 10:29 am
Contact:

Re: High latency with backup on CSV

Post by RWeiss »

We do seem to have a similar issue where we get huge response times on all VMs in the CSV during the backup. With huge response times I mean they sometimes freeze up for 10 seconds. I do not see any performance stress on the SAN (reports <<10ms on the disks), not on the network (2x 10Gbit Team, SAN connected through 2x 10Gbit MPIO), no CPU usage.

The hugest spikes in latency I always see in
\Device\Harddisk\Volume6\System Volume Information\... which correlates to the CSV that hosts the VMs.

As soon as I perform any action in the VMs (log in, open internet explorer, log out is all that's needed to cause this).

Case ID is 02053685.

Could someone let me know whether I can check if the IOs are redirected as well in our case? It does not show up in Failover Cluster Manager - which metric should I check in perfmon?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: High latency with backup on CSV

Post by Andreas Neufert »

Another customer had an issue with ODX and the Storage.

Deactivation of the SSD trim helped in that case. Please check with Microsoft Support if this is the case.

The solution was the following reg key:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: High latency with backup on CSV

Post by Andreas Neufert »

Another customer had an issue with ODX and the Storage.

Deaktivation of the SSD trim helped in that case. Please check with Microsoft Support if this is the case.

The solution was the following reg key:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests