Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

High latency with backup on CSV

Post by jdud » Dec 17, 2015 11:26 am

Hello,

We have a problem with backup and only backup on a CSV storage Windows 2012R2 up to date with last KB.
On backup with Veeam we have high latency up to 5 seconds and sometimes 15 seconds !
Perfmon show latency only on IO Redirect. So it seems there is no problem with storage.
And IO redirect show high latency also with only one node !
I dont understand why there is IO Redirect when we have only one node UP. Why CSV dont make direct IO ?

Do you know issue ?

Thank a lot ;)

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. » Dec 17, 2015 4:10 pm

Do you use software or hardware VSS provider? Also not sure I fully understand when you say that only one node is running. Can you please clarify this?

jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud » Dec 18, 2015 1:06 pm

Hello,

We use software VSS provider. With our ZFS san we dont have vss hardware provider.
I was thinking that if we have only one node UP it was not possible to have IO redirect. But even only one node up there is IO redirect with high latency.
Today i use Perfmon and the "Direct IO Failure redirection" counter is not incremented. So this is the proof that the latency cause is between CSV and HyperV.

Also Get-StorageReliabilityCounter show latency up to 50ms.

Thank you for you help Vitaliy.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. » Dec 18, 2015 1:15 pm

What node is the current owner of the CSV volume in your setup? You can check that via Failover Cluster settings.

jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud » Dec 18, 2015 1:22 pm

Node1 is the owner of the CSV.
When node1 and node2 are up we have high latency on redirected IO.
When node1 is alone (node2 is halted) we have same high latency on redirected IO.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. » Dec 18, 2015 2:31 pm

That's weird. Did you ever have direct IO mode working? Do you see this event on the Hyper-V host by any chance - Event ID = 5125?

jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud » Dec 18, 2015 3:16 pm

Yes direct IO is working very good.
We have about 30 vm working with very good performance.
The redirected IO happens only while backup is running.

I have sometimes 5120 and 5217 but no 5125. Also there is 10172 and 16010.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High latency with backup on CSV

Post by Vitaliy S. » Dec 18, 2015 3:35 pm

I meant to say if ever had a direct IO working during backup job. Assuming that all below is correct, then please contact our technical support team to review your setup:

1. You're using Software VSS provider
2. You don't have Event ID = 5125 on the Hyper-V node acting as the owner of CSV
3. When one host is shut down (not owner of the CSV) and you start to backup VMs residing on the CSV owner host and still observe redirected IO all the time

After reviewing your setup our team should be able to understand what is causing this behavior of the Hyper-V host. Since we do not control this process (redirecting traffic), it would be beneficial for all community members to know the the results of your investigation.

jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud » Dec 18, 2015 4:10 pm

Thank you Vitally, we have opening a case on Veeam support and we will post issue in this thread (If we find it !).

ptoro
Influencer
Posts: 21
Liked: 5 times
Joined: Jul 24, 2015 4:41 pm
Contact:

Re: High latency with backup on CSV

Post by ptoro » Mar 19, 2016 8:07 am

Was anything found?

veremin
Product Manager
Posts: 16884
Liked: 1433 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: High latency with backup on CSV

Post by veremin » Mar 21, 2016 11:36 am

The OP has never come back to update the topic with the found resolution, nor has he posted the ticket ID so we can track the case.

Thus, it's recommended to open your own ticket.

Thanks.

jdud
Novice
Posts: 6
Liked: never
Joined: Dec 17, 2015 11:15 am
Full Name: Julien Durand
Contact:

Re: High latency with backup on CSV

Post by jdud » Jun 08, 2016 1:15 pm

Hi all,

No solution for this problem without using hardware provider.
We have this problem with 4 SAN (10Gpbs, infiniband and FC) : ZFS san, P2000, msa2000 and HP 3PAR
This happens only with CSV on Windows 2012 r2 cluster. And the IO stats in Windows shows latency ONLY on IO redirect, but the CSV status never change to "io redirect".

This is probably a CSV bug.

Shestakov
Veeam Software
Posts: 6974
Liked: 711 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: High latency with backup on CSV

Post by Shestakov » Jun 12, 2016 4:10 pm

Hi Julien,
Thanks for updating. Could you provide your support case number for us to get the details? Thanks!

RWeiss
Influencer
Posts: 21
Liked: 1 time
Joined: Nov 13, 2014 10:29 am
Contact:

Re: High latency with backup on CSV

Post by RWeiss » Jan 31, 2017 9:30 am

We do seem to have a similar issue where we get huge response times on all VMs in the CSV during the backup. With huge response times I mean they sometimes freeze up for 10 seconds. I do not see any performance stress on the SAN (reports <<10ms on the disks), not on the network (2x 10Gbit Team, SAN connected through 2x 10Gbit MPIO), no CPU usage.

The hugest spikes in latency I always see in
\Device\Harddisk\Volume6\System Volume Information\... which correlates to the CSV that hosts the VMs.

As soon as I perform any action in the VMs (log in, open internet explorer, log out is all that's needed to cause this).

Case ID is 02053685.

Could someone let me know whether I can check if the IOs are redirected as well in our case? It does not show up in Failover Cluster Manager - which metric should I check in perfmon?

Andreas Neufert
Veeam Software
Posts: 3813
Liked: 684 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: High latency with backup on CSV

Post by Andreas Neufert » May 24, 2018 8:07 am

Another customer had an issue with ODX and the Storage.

Deactivation of the SSD trim helped in that case. Please check with Microsoft Support if this is the case.

The solution was the following reg key:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1

Andreas Neufert
Veeam Software
Posts: 3813
Liked: 684 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: High latency with backup on CSV

Post by Andreas Neufert » May 24, 2018 8:07 am

Another customer had an issue with ODX and the Storage.

Deaktivation of the SSD trim helped in that case. Please check with Microsoft Support if this is the case.

The solution was the following reg key:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests