High latency with backup on CSV

Hyper-V specific discussions

High latency with backup on CSV

Veeam Logoby jdud » Thu Dec 17, 2015 11:26 am

Hello,

We have a problem with backup and only backup on a CSV storage Windows 2012R2 up to date with last KB.
On backup with Veeam we have high latency up to 5 seconds and sometimes 15 seconds !
Perfmon show latency only on IO Redirect. So it seems there is no problem with storage.
And IO redirect show high latency also with only one node !
I dont understand why there is IO Redirect when we have only one node UP. Why CSV dont make direct IO ?

Do you know issue ?

Thank a lot ;)
jdud
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 17, 2015 11:15 am
Full Name: Julien Durand

Re: High latency with backup on CSV

Veeam Logoby Vitaliy S. » Thu Dec 17, 2015 4:10 pm

Do you use software or hardware VSS provider? Also not sure I fully understand when you say that only one node is running. Can you please clarify this?
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: High latency with backup on CSV

Veeam Logoby jdud » Fri Dec 18, 2015 1:06 pm

Hello,

We use software VSS provider. With our ZFS san we dont have vss hardware provider.
I was thinking that if we have only one node UP it was not possible to have IO redirect. But even only one node up there is IO redirect with high latency.
Today i use Perfmon and the "Direct IO Failure redirection" counter is not incremented. So this is the proof that the latency cause is between CSV and HyperV.

Also Get-StorageReliabilityCounter show latency up to 50ms.

Thank you for you help Vitaliy.
jdud
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 17, 2015 11:15 am
Full Name: Julien Durand

Re: High latency with backup on CSV

Veeam Logoby Vitaliy S. » Fri Dec 18, 2015 1:15 pm

What node is the current owner of the CSV volume in your setup? You can check that via Failover Cluster settings.
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: High latency with backup on CSV

Veeam Logoby jdud » Fri Dec 18, 2015 1:22 pm

Node1 is the owner of the CSV.
When node1 and node2 are up we have high latency on redirected IO.
When node1 is alone (node2 is halted) we have same high latency on redirected IO.
jdud
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 17, 2015 11:15 am
Full Name: Julien Durand

Re: High latency with backup on CSV

Veeam Logoby Vitaliy S. » Fri Dec 18, 2015 2:31 pm

That's weird. Did you ever have direct IO mode working? Do you see this event on the Hyper-V host by any chance - Event ID = 5125?
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: High latency with backup on CSV

Veeam Logoby jdud » Fri Dec 18, 2015 3:16 pm

Yes direct IO is working very good.
We have about 30 vm working with very good performance.
The redirected IO happens only while backup is running.

I have sometimes 5120 and 5217 but no 5125. Also there is 10172 and 16010.
jdud
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 17, 2015 11:15 am
Full Name: Julien Durand

Re: High latency with backup on CSV

Veeam Logoby Vitaliy S. » Fri Dec 18, 2015 3:35 pm

I meant to say if ever had a direct IO working during backup job. Assuming that all below is correct, then please contact our technical support team to review your setup:

1. You're using Software VSS provider
2. You don't have Event ID = 5125 on the Hyper-V node acting as the owner of CSV
3. When one host is shut down (not owner of the CSV) and you start to backup VMs residing on the CSV owner host and still observe redirected IO all the time

After reviewing your setup our team should be able to understand what is causing this behavior of the Hyper-V host. Since we do not control this process (redirecting traffic), it would be beneficial for all community members to know the the results of your investigation.
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: High latency with backup on CSV

Veeam Logoby jdud » Fri Dec 18, 2015 4:10 pm

Thank you Vitally, we have opening a case on Veeam support and we will post issue in this thread (If we find it !).
jdud
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 17, 2015 11:15 am
Full Name: Julien Durand

Re: High latency with backup on CSV

Veeam Logoby ptoro » Sat Mar 19, 2016 8:07 am

Was anything found?
ptoro
Influencer
 
Posts: 12
Liked: 5 times
Joined: Fri Jul 24, 2015 4:41 pm

Re: High latency with backup on CSV

Veeam Logoby v.Eremin » Mon Mar 21, 2016 11:36 am

The OP has never come back to update the topic with the found resolution, nor has he posted the ticket ID so we can track the case.

Thus, it's recommended to open your own ticket.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13558
Liked: 1006 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: High latency with backup on CSV

Veeam Logoby jdud » Wed Jun 08, 2016 1:15 pm

Hi all,

No solution for this problem without using hardware provider.
We have this problem with 4 SAN (10Gpbs, infiniband and FC) : ZFS san, P2000, msa2000 and HP 3PAR
This happens only with CSV on Windows 2012 r2 cluster. And the IO stats in Windows shows latency ONLY on IO redirect, but the CSV status never change to "io redirect".

This is probably a CSV bug.
jdud
Novice
 
Posts: 6
Liked: never
Joined: Thu Dec 17, 2015 11:15 am
Full Name: Julien Durand

Re: High latency with backup on CSV

Veeam Logoby Shestakov » Sun Jun 12, 2016 4:10 pm

Hi Julien,
Thanks for updating. Could you provide your support case number for us to get the details? Thanks!
Shestakov
Veeam Software
 
Posts: 5027
Liked: 419 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: High latency with backup on CSV

Veeam Logoby RWeiss » Tue Jan 31, 2017 9:30 am

We do seem to have a similar issue where we get huge response times on all VMs in the CSV during the backup. With huge response times I mean they sometimes freeze up for 10 seconds. I do not see any performance stress on the SAN (reports <<10ms on the disks), not on the network (2x 10Gbit Team, SAN connected through 2x 10Gbit MPIO), no CPU usage.

The hugest spikes in latency I always see in
\Device\Harddisk\Volume6\System Volume Information\... which correlates to the CSV that hosts the VMs.

As soon as I perform any action in the VMs (log in, open internet explorer, log out is all that's needed to cause this).

Case ID is 02053685.

Could someone let me know whether I can check if the IOs are redirected as well in our case? It does not show up in Failover Cluster Manager - which metric should I check in perfmon?
RWeiss
Influencer
 
Posts: 19
Liked: 1 time
Joined: Thu Nov 13, 2014 10:29 am


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: Yahoo [Bot] and 10 guests