Monitoring and reporting for Veeam Data Platform
AlainRussell
Enthusiast
Posts: 49
Liked: 1 time
Joined: Aug 27, 2011 12:04 am
Full Name: Alain Russell
Contact:

V7 - VMware Issue - False alerts

Post by AlainRussell »

Since upgrading to v7 I've been getting some false alerts as below, I can't see a corresponding warning in Vcenter so not sure where to look to fix? Any ideas?

"Datastore Highest Latency" (28251344 ms) is above a defined threshold (75 ms)"
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Hi Alain,

What values are displayed on the performance graphs in the vSphere Client for this metric? Can you please open a support case with our technical team and let them have a look at your debug logs?

Thanks!
AlainRussell
Enthusiast
Posts: 49
Liked: 1 time
Joined: Aug 27, 2011 12:04 am
Full Name: Alain Russell
Contact:

Re: V7 - False alerts

Post by AlainRussell »

There are no alerts showing (screenshot attached), strange that the false alert coincides with the Backup job completing. I'm travelling at the moment but will open a support case when I get back - not a major issue currently.

http://utility.blackpepper.co.nz/alain/VCenter.png
http://utility.blackpepper.co.nz/alain/Alert.png
http://utility.blackpepper.co.nz/alain/Datastore.png

Datastore latency shows a max of 19ms read, 16ms max over the last 2 hours.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Ok, let me check with our QC team if it is a known issue or not.
emsaus
Novice
Posts: 3
Liked: never
Joined: Aug 14, 2012 2:08 am
Full Name: Ben Howell
Contact:

Re: V7 - False alerts

Post by emsaus »

We are also seeing the same thing. Upgraded to V7 yesterday the next backup that ran generated the same alarm. "Datastore Highest Latency (3869081 ms) is above a defined threshold (75 ms)". It lasted for about 20 mins then cleared however the backup took ~60mins to run. No mention of any issues in vcenter server.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?
Jadawyn
Lurker
Posts: 1
Liked: never
Joined: Aug 31, 2012 8:41 am
Full Name: Luca Bedogni
Contact:

Re: V7 - False alerts

Post by Jadawyn »

Same problem here. It only happens when Backup job completes. No warning from vcenter.
emsaus
Novice
Posts: 3
Liked: never
Joined: Aug 14, 2012 2:08 am
Full Name: Ben Howell
Contact:

Re: V7 - False alerts

Post by emsaus »

Vitaliy S. wrote:Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?
No problems, case id: 00432028

Thanks.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: V7 - False alerts

Post by rfn »

I'm having this problem as well... I haven't created a case yet.
TomBlue
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann
Contact:

Re: V7 - False alerts

Post by TomBlue »

We exoperience the same issue. I opened a case: 00434907
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: V7 - False alerts

Post by rfn »

I now has a case as well: Case # 00434916
Orven
Service Provider
Posts: 2
Liked: 2 times
Joined: Aug 27, 2013 12:29 pm
Full Name: Olaf Reitz
Contact:

Re: V7 - False alerts

Post by Orven »

Same issue since Update to v7 case # 00435011
TomBlue
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann
Contact:

Re: V7 - False alerts

Post by TomBlue »

TomBlue wrote:We exoperience the same issue. I opened a case: 00434907
UPDATE:

We investigated our case a little deeper, thanks to Veeam Support. We recognized that the spikes we experience in Veeam One are reported as well in vSphere performance monitor. Here is one example where you can see this spikes:

http://sdrv.ms/1ad6cMV (Veeam One: Datastore Highest Latency)
http://sdrv.ms/1ad6kMx (vSphere: VM Disk Highest Latency)
http://sdrv.ms/1ad6raR (vSphere: Datastore Highest Latency)

I cannot find any jobs that can cause these extrem spikes. If there were real latencies like this, I guess we would experience performance issues in any application. As well as the "Datastore Highest Latency" shows negative numbers, strange isn't it? I just found out that around the same time as the high latency is occuring, the machine in question has high CPU load, but not sure yet if this causes the high latency or if hight latency is the reason for CPU load.

We do not have any solution for this yet, but I'm curious if anyone has the same experience and maybe have a solution on this. Of course I agree to Veeam support that this is a problem in vSphere, not in Veeam One, though these error started with update of Veem One + Veeam B&R to version 7.0.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Thomas,

Yes, our R&D team has confirmed that the same spikes are observed in the vSphere Client as well, thanks for the update.

Btw, do you experience these issues at the particular backup job operation? For example, during snapshot commit operation or similar? What processing mode do you use for your backup jobs? HotAdd?

Thanks!
TomBlue
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann
Contact:

Re: V7 - False alerts

Post by TomBlue »

Hi Vitaly!
No, in my case there is no relation to the backup jobs. I do backups only once a night and spikes I can see about 1 - 3 times per hour on this specific VM. On two other VMs I get the error only 1 or 2 times a day. There is also no connection to any jobs on these servers. I still do not have any idea.

Using network mode for backup. Veeam B&R and Veeam One is installed on a physical machine.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Hmm...at what times do you usually get these spikes? Are there any sort of maintenance tasks scheduled at the time when spikes occur?
TomBlue
Influencer
Posts: 15
Liked: 3 times
Joined: Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann
Contact:

Re: V7 - False alerts

Post by TomBlue »

Spikes are occuring unregular. I cannot find any relation to jobs running on this or other servers. Here's a screenshot of todays metrics: http://sdrv.ms/18p5TZR. Because this is 1 day period values are a little "foggy" - you do not see spikes that are only half of others. I counted the errors of last 24 hours: There are approx. 25 and no recognisable pattern.

[Update]: There have been some occurences last minutes with maximum values of 127131031 ms , that's 35 hours. That could not be any real latency. At the same time I had very low DAVG/cms values esxtop.

Anyway, thanks for your help, I will try to get any VMware support - or simply ignore this issue, because there is no constraint in "real life" :?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: V7 - False alerts

Post by foggy »

I'm trying to reproduce this issue in my labs, but I haven't been successful so far. If VMware support nails down this issue, I would appreciate if could come back and update this topic with their findings.
GL@MO
Enthusiast
Posts: 25
Liked: 2 times
Joined: Nov 02, 2012 9:04 pm
Location: Vreden, Germany
Contact:

Re: V7 - False alerts

Post by GL@MO »

Experiencing the same "problem"
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Hello Guido,

Did you have a chance to check the reason for this behavior with VMware support team? Still wondering if there is any pattern in this "weird" behavior.

Thank you!
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: V7 - False alerts

Post by mcwill »

We are also seeing this, from what we've observed...
  • Alerts started after upgrading to 7.0 from 6.5. Now receiving approx. 3 alerts per day from 50 VMs
  • Latency also appears in VCenter performance current graph, but not in daily or any other rollup. May be filtered as part of roll-up process?
  • Latency alert always relates to a associated backup snapshot creation/removal
Backups are processed by hotadd and we have parallel processing enabled.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Hi Iain,

1. You didn't see that in 6.5 cause we have added a couple of new predefined alarms to v7.
2. Yes, VMware rollups real-time performance data, so these spikes are not visible in the daily or other reporting periods.
3. I would recommend to involve VMware support team in the troubleshooting process, as these extremely high values for latency shouldn't be true.

Thank you!
JROleon
Lurker
Posts: 1
Liked: never
Joined: Oct 30, 2012 1:07 pm
Full Name: Jeroen Roegiers
Contact:

Re: V7 - False alerts

Post by JROleon »

Hi,
I'm also experiencing the same problem. No case made. I'll keep an eye on this topic.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Jeroen, I would also suggest contacting VMware support on this, the more info they have the better.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

[MERGED] VM total disk latency

Post by JosueM »

Good day everyone,

I just installed veeam 7 no changes made on default config except the email notification, suddenly I started to receive a lot of emails for random VMs with VM total disk latency alert then in minutes the alert goes back to green, this happens more or les each 5 to 10 minutes for multiple vms.

I have noticed no alert is triggered on vcenter itself and no user is complaining about performance. Not sure how risky this message could be.

I attached a copy of the message
Image
image upload
GL@MO
Enthusiast
Posts: 25
Liked: 2 times
Joined: Nov 02, 2012 9:04 pm
Location: Vreden, Germany
Contact:

Re: V7 - False alerts

Post by GL@MO »

hmmmmm latency notifications are gone...
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: VM total disk latency

Post by rfn »

This is happening here as well. I've been told by support (Case # 00434916) that the cause is that Veeam One 6 didn't know about the counter "Total Disk Latency" which is supposed to be a fairly new counter. Veeam One 7 sees this counter and correctly flags it as an error when vCenter reports these high numbers.

Veeam support suggests that I contact VMware but I don't have a support agreement - only maintenance - so I might just disable this alarm since it looks to be a bug. It's happening when snapshots are being committed during backups.
Vitaliy S.
VP, Product Management
Posts: 27026
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - False alerts

Post by Vitaliy S. »

Hi Josue.

Rene is correct, if you do not have any issues you may temporary disable this alarm and open a support case with VMware for further troubleshooting.

Thanks!
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: V7 - False alerts

Post by JosueM »

Ok, thanks for your help. Is just the message at first time seems like critic but after reading on this thread it seems more like a warning.
KOMBillM
Novice
Posts: 5
Liked: never
Joined: Jan 03, 2012 9:14 pm
Full Name: Bill MacEachern
Contact:

Re: V7 - False alerts

Post by KOMBillM »

We are also seeing this. Every night I get an alarm for one specific VM:

"Datastore Highest Latency" (3353953467950 ms) is above a defined threshold (75 ms)

If you do the math I believe it works out to 106 years, which is absurd.
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests