V7 - VMware Issue - False alerts

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Re: V7 - VMware Issue - False alerts

Veeam Logoby hafesta » Fri Oct 11, 2013 1:38 pm

We got the same problem.

ESXi 5.1.0, 1157734
vShere 5.1.0 Build 1123961
hafesta
Novice
 
Posts: 3
Liked: never
Joined: Mon Nov 12, 2012 10:02 am

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Fri Oct 11, 2013 1:39 pm

Have you contacted VMware support team on this issue? What was their response?
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby odge » Fri Oct 11, 2013 2:52 pm

HI

Can I ask the question, why would it be different in version 7 to version 6.5? (If it was in fact a VMWare issue)

Regards
Matthew
odge
Veeam ProPartner
 
Posts: 59
Liked: 3 times
Joined: Thu Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Fri Oct 11, 2013 2:59 pm

Hi Matthew, please see the reason for this in earlier posts of this topic. Also you can check that by yourself via vSphere Client as one of the OPs did.

Anyway, currently we are working on a workaround for this VMware issue, this should be included in patch 1. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby richardb092 » Tue Oct 29, 2013 10:15 am

Hi All,

Getting the same issue only after upgrading both Veeam B&R to v7 with the latest patch, and Veeam One from 6.5 to V7.

It has happened twice now. I will be raising a ticket tomorrow morning.

Veeam ONE Monitor

Server: BAS-APPS-V02

Object:BAS-MAIL-V01
Object Type:Virtual Machine
Parent Object:Resources
Status:Error
Alarm:VM total disk latency
Time:29/10/2013 7:12:25 PM
Details:"Datastore Highest Latency" (24617045 ms) is above a defined threshold (75 ms)

It does appear to be a false alert from what I can tell. It looks like it is happening when the backup is over and is merging the snap-shot back into the vmdk file.

Richard
richardb092
Novice
 
Posts: 5
Liked: never
Joined: Tue Oct 29, 2013 10:10 am
Full Name: Richard Bance

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Tue Oct 29, 2013 11:04 am

Richard, can you please check vSphere Client performance graphs to see if you see these spikes there or not?
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby richardb092 » Tue Oct 29, 2013 8:54 pm

Hi Vitaly,

CPU - nothing spectacular --looked normal
Disk - 542kBps usage and 10ms latency was the highest recorded during the incident.
Network went from 80kBps at 7:12PM to 258kBps then back down again at 7:27PM (when the ok from Veeam One was received).

This is the performace of the GUEST.

Any other things??

Richard
richardb092
Novice
 
Posts: 5
Liked: never
Joined: Tue Oct 29, 2013 10:10 am
Full Name: Richard Bance

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Tue Oct 29, 2013 8:59 pm

So this counter (vSphere: VM Disk Highest Latency) shows nothing like that at the time when the issue occurred?
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby richardb092 » Tue Oct 29, 2013 10:55 pm

That is correct from what I am seeing...The performance from the HOST confirms this too...

The Remove Snap-shot function kicked in at 7:10:44PM. It looks like it completed at 7:11:24PM. This all looks good.

There is nothing in vSphere that seems to be a red flag....


Job details below

VMware Backup job: BAS-MAIL-V01-NEW
Created by BAS\xxxxxxx at 20/09/2013 8:43:56 PM.
Success
1 of 1 VMs processed
Tuesday, 29 October 2013 6:30:17 PM
Success1Start time6:30:17 PMTotal size550.0 GBBackup size317.7 GB
Warning0End time7:12:07 PMData read26.5 GBDedupe1.2x
Error0Duration0:41:50Transferred14.2 GBCompression1.4x
Details
NameStatusStart timeEnd timeSizeReadTransferredDurationDetails
BAS-MAIL-V01Success6:30:50 PM7:11:55 PM550.0 GB26.5 GB14.2 GB0:41:04

Veeam Backup & Replication 7.0.0.715
richardb092
Novice
 
Posts: 5
Liked: never
Joined: Tue Oct 29, 2013 10:10 am
Full Name: Richard Bance

Re: V7 - VMware Issue - False alerts

Veeam Logoby v.Eremin » Wed Oct 30, 2013 8:56 am

Hi, Richard,

Actually, Veeam One collects information from vCenter Server/ESXi hosts via VMware API. In other words, we rely mostly on what is informed to us by VMware. Previous users affected by the said behavior confirmed that those spikes were also shown in vSphere Client.

If you’re completely sure, that it is only VeeamOne that keeps informing you about enormous latency numbers, then, open a ticket with our support team and let them verify your setup/environment.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: V7 - VMware Issue - False alerts

Veeam Logoby guillermo.lozano » Wed Oct 30, 2013 4:58 pm

Hello Everyone, I have the same issue and I’ve been in contact with VMware Support after treating this under Veeam Case # 00438976.

Veeam Support told me that if this values are reported under vSphere Client that this should be treated with VMware Support.

I've Opened VMware Support Request 13376817109 & VMware support states that this is an issue with Veeam.

I have this alert turned off and this issue only occurs when I enable this alert on Veeam ONE.

If the alarm remains disabled I do not see any spike with 43588595 ms of latency.

Veeam says that Veeam ONE only reports what the ESX server displays.
But the ESX server displays this when the Veeam ONE Alarm is enabled.

Everyone is right or everyone is wrong??

Thanks.
Regards,
Guillermo.-
guillermo.lozano
Enthusiast
 
Posts: 69
Liked: 10 times
Joined: Mon Feb 04, 2013 4:35 pm
Full Name: Guillermo Lozano

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Wed Oct 30, 2013 9:06 pm

Hello Guillermo,

guillermo.lozano wrote:I have this alert turned off and this issue only occurs when I enable this alert on Veeam ONE.

If the alarm remains disabled I do not see any spike with 43588595 ms of latency.

Veeam ONE only requires read-only access to retrieve performance data, so I'm not quite sure how Veeam ONE can be the reason for this latency spike.

Can you please tell me what does vSphere Client show when you enable this alarm and see its fired in Veeam ONE? Does it show 43588595 ms of latency?
guillermo.lozano wrote:If the alarm remains disabled I do not see any spike with 43588595 ms of latency.

Disabling the alarm does not stop the process of monitoring this metrics. If you disable it, it is just disabled in Veeam ONE alarms engine, so ESXi hosts should not be affected by this change.

guillermo.lozano wrote:Veeam says that Veeam ONE only reports what the ESX server displays.
But the ESX server displays this when the Veeam ONE Alarm is enabled.

Will you be able to reproduce this behavior with enabling/disabling alarm and show it to our support team? In this case I will be able to chase our dev team and ask to clarify how it is possible.

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby guillermo.lozano » Thu Oct 31, 2013 1:42 pm

Hi VItaly,

When I first migrated to Veeam ONE v7, that alarm started monitoring these spikes the vSphere client confirmed the spikes on the stated VM's.

When i disabled the alarm, the spikes never showed up again on the VM's and vSphere Client never showed that again. I've even checked the esxi host form the console.

As soon as I enable the alarm, the latency spikes with an abnormal latency value appear again:

Image

Image

Image

Each spike corresponds to the alarm, as soon as the alarm is disabled the latency spikes dissapear.

I hope this is more clear.

I'll be happy to show it on realtime via webex with the dev's-

Thanks.
Regards,
Guillermo.-
guillermo.lozano
Enthusiast
 
Posts: 69
Liked: 10 times
Joined: Mon Feb 04, 2013 4:35 pm
Full Name: Guillermo Lozano

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Thu Oct 31, 2013 1:52 pm

Guillermo, thanks a lot for the clarification! I have passed this information to our support and QC teams. If you could confirm this behavior over the WebEx session that would be highly appreciated.
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby guillermo.lozano » Thu Oct 31, 2013 2:06 pm

Sure no problem, just contact me, the details are in the support case i mentioned earlier.

Best Regards,
Guillermo.-
guillermo.lozano
Enthusiast
 
Posts: 69
Liked: 10 times
Joined: Mon Feb 04, 2013 4:35 pm
Full Name: Guillermo Lozano

PreviousNext

Return to Veeam ONE



Who is online

Users browsing this forum: No registered users and 4 guests