Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. » Sep 11, 2013 1:41 pm

Indeed, have you have a chance to show these spikes to VMware support team? What was their response?

TomBlue
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann
Contact:

Re: V7 - False alerts

Post by TomBlue » Sep 11, 2013 1:48 pm

Hi KOMBillM!

With this value your're leader in then ranking :-)

But as far as I can see this is not a Veeam One issue because this values are reported in vShere also. Do you have more than basic maintenance to open a support case with VMware?

Cheers,
Thomas

cmiller@hipco.com
Novice
Posts: 3
Liked: 1 time
Joined: Apr 25, 2013 9:26 pm
Full Name: Charles Miller
Contact:

Re: V7 - False alerts

Post by cmiller@hipco.com » Sep 12, 2013 7:58 pm

Same problem here. Case #0444378

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. » Sep 12, 2013 8:21 pm

Charles, what does "Total Disk Latency" performance counter show in vSphere Client for the same time period?

pufferdude
Expert
Posts: 145
Liked: 5 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: V7 - VMware False alerts

Post by pufferdude » Sep 16, 2013 4:49 pm

Just jumping in that I too am seeing these crazy Total Disk Latency alerts. I'm saddened to come here and find out that's it's a problem on the vSphere side, because the likelihood of THAT being fixed readily is much less likely than if it were an error on Veeam's side :-(

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware False alerts

Post by Vitaliy S. » Sep 16, 2013 6:52 pm

Jim, I have discussed this issue with our dev team and currently we are working on possible workarounds for this behavior. Most likely it will be included in one of the upcoming updates of Veeam ONE.

KOMBillM
Novice
Posts: 5
Liked: never
Joined: Jan 03, 2012 9:14 pm
Full Name: Bill MacEachern
Contact:

Re: V7 - VMware False alerts

Post by KOMBillM » Sep 18, 2013 12:45 pm

Unfortunately, I do not have a support contract to lean on.

VanUden
Lurker
Posts: 2
Liked: 2 times
Joined: Mar 06, 2012 7:49 am
Contact:

Re: V7 - VMware False alerts

Post by VanUden » Sep 18, 2013 1:22 pm

Same problem here...
Upgrated an hour ago to v7 (no fresh install) and receiving these errors.
I will monitor for a day and create a support ticket if the problem exists.
I will also follow this thread, hopefully for a solution...

veremin
Product Manager
Posts: 16901
Liked: 1438 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: V7 - VMware False alerts

Post by veremin » Sep 18, 2013 1:32 pm

As mentioned above, the problem seems to be related to VMware which keeps reporting enormous latency numbers. However, we’re currently investigating this case and trying to figure out potential workarounds for it.

Also, it might be worth opening a ticket with VMware support, so that, they will have more information regarding this problem and will be able to address it soon.

Thanks.

odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

[MERGED] : strange (unrealistic) vm disk latency warnings/er

Post by odge » Sep 27, 2013 12:07 pm

Hi

Since upgrading VeeamOne, we are getting VM disk latency warnings as follows... which obviously is not correct (Thats like an 8 hour offset?)

Details: "Datastore Highest Latency" (28893416 ms) is above a defined threshold (75 ms)

We are seeing it on multiple VMs, for short periods, then its gone. (In the last week, 4 of the VM's have it repeated, 84,29,4,2 times respectively)

VanUden
Lurker
Posts: 2
Liked: 2 times
Joined: Mar 06, 2012 7:49 am
Contact:

Re: V7 - VMware False alerts

Post by VanUden » Sep 27, 2013 1:15 pm 2 people like this post

My problem seems to be disappeared...
I was running vShpere 5.1.0 update 1b with ESXi hosts on 5.0.0.
Then Veeam One v6.5.x was upgraded to v7.x and the problem started.

Yesterday I've upgrade the ESXi hosts from 5.0.0 to 5.1.0 build 1065491 and the problem is gone...
Don't know why but maybe this helps someone ;)

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware False alerts

Post by Vitaliy S. » Sep 27, 2013 1:19 pm

Thanks for letting us now, maybe VMware has fixed it in their latest build.

pufferdude
Expert
Posts: 145
Liked: 5 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: V7 - VMware False alerts

Post by pufferdude » Sep 27, 2013 2:29 pm

VanUden wrote:My problem seems to be disappeared...
Yesterday I've upgrade the ESXi hosts from 5.0.0 to 5.1.0 build 1065491 and the problem is gone...
Don't know why but maybe this helps someone ;)
Alas, we were already on ESXi 5.1.0 build 1157734 when we upgraded to Veeam 7 and started seeing the false latency alerts. So maybe it was "fixed" in your slightly-older version of ESXi but it's definitely there in our slightly-newer one :-(

pendragoncrw
Enthusiast
Posts: 38
Liked: 3 times
Joined: Jun 14, 2010 3:06 am
Full Name: C White
Contact:

Re: V7 - VMware False alerts

Post by pendragoncrw » Oct 01, 2013 12:18 am

I've also experienced this since upgrading to Veeam v7. We have a mix of 5.0u1, 5.1 u1a, and 5.1u1b hosts.

Chris

simon617
Influencer
Posts: 15
Liked: never
Joined: Jul 28, 2011 4:24 pm
Contact:

Re: V7 - VMware False alerts

Post by simon617 » Oct 04, 2013 2:55 pm

We also having issues with this after we upgrade to v7. Look forward to the fix.

hafesta
Novice
Posts: 3
Liked: never
Joined: Nov 12, 2012 10:02 am
Contact:

Re: V7 - VMware Issue - False alerts

Post by hafesta » Oct 11, 2013 1:38 pm

We got the same problem.

ESXi 5.1.0, 1157734
vShere 5.1.0 Build 1123961

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Oct 11, 2013 1:39 pm

Have you contacted VMware support team on this issue? What was their response?

odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

Re: V7 - VMware Issue - False alerts

Post by odge » Oct 11, 2013 2:52 pm

HI

Can I ask the question, why would it be different in version 7 to version 6.5? (If it was in fact a VMWare issue)

Regards
Matthew

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Oct 11, 2013 2:59 pm

Hi Matthew, please see the reason for this in earlier posts of this topic. Also you can check that by yourself via vSphere Client as one of the OPs did.

Anyway, currently we are working on a workaround for this VMware issue, this should be included in patch 1. Thanks!

richardb092
Novice
Posts: 5
Liked: never
Joined: Oct 29, 2013 10:10 am
Full Name: Richard Bance
Contact:

Re: V7 - VMware Issue - False alerts

Post by richardb092 » Oct 29, 2013 10:15 am

Hi All,

Getting the same issue only after upgrading both Veeam B&R to v7 with the latest patch, and Veeam One from 6.5 to V7.

It has happened twice now. I will be raising a ticket tomorrow morning.

Veeam ONE Monitor

Server: BAS-APPS-V02

Object: BAS-MAIL-V01
Object Type: Virtual Machine
Parent Object: Resources
Status: Error
Alarm: VM total disk latency
Time: 29/10/2013 7:12:25 PM
Details: "Datastore Highest Latency" (24617045 ms) is above a defined threshold (75 ms)

It does appear to be a false alert from what I can tell. It looks like it is happening when the backup is over and is merging the snap-shot back into the vmdk file.

Richard

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Oct 29, 2013 11:04 am

Richard, can you please check vSphere Client performance graphs to see if you see these spikes there or not?

richardb092
Novice
Posts: 5
Liked: never
Joined: Oct 29, 2013 10:10 am
Full Name: Richard Bance
Contact:

Re: V7 - VMware Issue - False alerts

Post by richardb092 » Oct 29, 2013 8:54 pm

Hi Vitaly,

CPU - nothing spectacular --looked normal
Disk - 542kBps usage and 10ms latency was the highest recorded during the incident.
Network went from 80kBps at 7:12PM to 258kBps then back down again at 7:27PM (when the ok from Veeam One was received).

This is the performace of the GUEST.

Any other things??

Richard

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Oct 29, 2013 8:59 pm

So this counter (vSphere: VM Disk Highest Latency) shows nothing like that at the time when the issue occurred?

richardb092
Novice
Posts: 5
Liked: never
Joined: Oct 29, 2013 10:10 am
Full Name: Richard Bance
Contact:

Re: V7 - VMware Issue - False alerts

Post by richardb092 » Oct 29, 2013 10:55 pm

That is correct from what I am seeing...The performance from the HOST confirms this too...

The Remove Snap-shot function kicked in at 7:10:44PM. It looks like it completed at 7:11:24PM. This all looks good.

There is nothing in vSphere that seems to be a red flag....


Job details below

VMware Backup job: BAS-MAIL-V01-NEW
Created by BAS\xxxxxxx at 20/09/2013 8:43:56 PM.
Success
1 of 1 VMs processed
Tuesday, 29 October 2013 6:30:17 PM
Success 1 Start time 6:30:17 PM Total size 550.0 GB Backup size 317.7 GB
Warning 0 End time 7:12:07 PM Data read 26.5 GB Dedupe 1.2x
Error 0 Duration 0:41:50 Transferred 14.2 GB Compression 1.4x
Details
Name Status Start time End time Size Read Transferred Duration Details
BAS-MAIL-V01 Success 6:30:50 PM 7:11:55 PM 550.0 GB 26.5 GB 14.2 GB 0:41:04

Veeam Backup & Replication 7.0.0.715

veremin
Product Manager
Posts: 16901
Liked: 1438 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: V7 - VMware Issue - False alerts

Post by veremin » Oct 30, 2013 8:56 am

Hi, Richard,

Actually, Veeam One collects information from vCenter Server/ESXi hosts via VMware API. In other words, we rely mostly on what is informed to us by VMware. Previous users affected by the said behavior confirmed that those spikes were also shown in vSphere Client.

If you’re completely sure, that it is only VeeamOne that keeps informing you about enormous latency numbers, then, open a ticket with our support team and let them verify your setup/environment.

Thanks.

guillermo.lozano
Enthusiast
Posts: 69
Liked: 10 times
Joined: Feb 04, 2013 4:35 pm
Full Name: Guillermo Lozano
Contact:

Re: V7 - VMware Issue - False alerts

Post by guillermo.lozano » Oct 30, 2013 4:58 pm

Hello Everyone, I have the same issue and I’ve been in contact with VMware Support after treating this under Veeam Case # 00438976.

Veeam Support told me that if this values are reported under vSphere Client that this should be treated with VMware Support.

I've Opened VMware Support Request 13376817109 & VMware support states that this is an issue with Veeam.

I have this alert turned off and this issue only occurs when I enable this alert on Veeam ONE.

If the alarm remains disabled I do not see any spike with 43588595 ms of latency.

Veeam says that Veeam ONE only reports what the ESX server displays.
But the ESX server displays this when the Veeam ONE Alarm is enabled.

Everyone is right or everyone is wrong??

Thanks.
Regards,
Guillermo.-

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Oct 30, 2013 9:06 pm

Hello Guillermo,
guillermo.lozano wrote:I have this alert turned off and this issue only occurs when I enable this alert on Veeam ONE.

If the alarm remains disabled I do not see any spike with 43588595 ms of latency.
Veeam ONE only requires read-only access to retrieve performance data, so I'm not quite sure how Veeam ONE can be the reason for this latency spike.

Can you please tell me what does vSphere Client show when you enable this alarm and see its fired in Veeam ONE? Does it show 43588595 ms of latency?
guillermo.lozano wrote:If the alarm remains disabled I do not see any spike with 43588595 ms of latency.
Disabling the alarm does not stop the process of monitoring this metrics. If you disable it, it is just disabled in Veeam ONE alarms engine, so ESXi hosts should not be affected by this change.
guillermo.lozano wrote:Veeam says that Veeam ONE only reports what the ESX server displays.
But the ESX server displays this when the Veeam ONE Alarm is enabled.
Will you be able to reproduce this behavior with enabling/disabling alarm and show it to our support team? In this case I will be able to chase our dev team and ask to clarify how it is possible.

Thank you!

guillermo.lozano
Enthusiast
Posts: 69
Liked: 10 times
Joined: Feb 04, 2013 4:35 pm
Full Name: Guillermo Lozano
Contact:

Re: V7 - VMware Issue - False alerts

Post by guillermo.lozano » Oct 31, 2013 1:42 pm

Hi VItaly,

When I first migrated to Veeam ONE v7, that alarm started monitoring these spikes the vSphere client confirmed the spikes on the stated VM's.

When i disabled the alarm, the spikes never showed up again on the VM's and vSphere Client never showed that again. I've even checked the esxi host form the console.

As soon as I enable the alarm, the latency spikes with an abnormal latency value appear again:

Image

Image

Image

Each spike corresponds to the alarm, as soon as the alarm is disabled the latency spikes dissapear.

I hope this is more clear.

I'll be happy to show it on realtime via webex with the dev's-

Thanks.
Regards,
Guillermo.-

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Oct 31, 2013 1:52 pm

Guillermo, thanks a lot for the clarification! I have passed this information to our support and QC teams. If you could confirm this behavior over the WebEx session that would be highly appreciated.

guillermo.lozano
Enthusiast
Posts: 69
Liked: 10 times
Joined: Feb 04, 2013 4:35 pm
Full Name: Guillermo Lozano
Contact:

Re: V7 - VMware Issue - False alerts

Post by guillermo.lozano » Oct 31, 2013 2:06 pm

Sure no problem, just contact me, the details are in the support case i mentioned earlier.

Best Regards,
Guillermo.-

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest