-
- Enthusiast
- Posts: 49
- Liked: 1 time
- Joined: Aug 27, 2011 12:04 am
- Full Name: Alain Russell
- Contact:
V7 - VMware Issue - False alerts
Since upgrading to v7 I've been getting some false alerts as below, I can't see a corresponding warning in Vcenter so not sure where to look to fix? Any ideas?
"Datastore Highest Latency" (28251344 ms) is above a defined threshold (75 ms)"
"Datastore Highest Latency" (28251344 ms) is above a defined threshold (75 ms)"
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Hi Alain,
What values are displayed on the performance graphs in the vSphere Client for this metric? Can you please open a support case with our technical team and let them have a look at your debug logs?
Thanks!
What values are displayed on the performance graphs in the vSphere Client for this metric? Can you please open a support case with our technical team and let them have a look at your debug logs?
Thanks!
-
- Enthusiast
- Posts: 49
- Liked: 1 time
- Joined: Aug 27, 2011 12:04 am
- Full Name: Alain Russell
- Contact:
Re: V7 - False alerts
There are no alerts showing (screenshot attached), strange that the false alert coincides with the Backup job completing. I'm travelling at the moment but will open a support case when I get back - not a major issue currently.
http://utility.blackpepper.co.nz/alain/VCenter.png
http://utility.blackpepper.co.nz/alain/Alert.png
http://utility.blackpepper.co.nz/alain/Datastore.png
Datastore latency shows a max of 19ms read, 16ms max over the last 2 hours.
http://utility.blackpepper.co.nz/alain/VCenter.png
http://utility.blackpepper.co.nz/alain/Alert.png
http://utility.blackpepper.co.nz/alain/Datastore.png
Datastore latency shows a max of 19ms read, 16ms max over the last 2 hours.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Ok, let me check with our QC team if it is a known issue or not.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Aug 14, 2012 2:08 am
- Full Name: Ben Howell
- Contact:
Re: V7 - False alerts
We are also seeing the same thing. Upgraded to V7 yesterday the next backup that ran generated the same alarm. "Datastore Highest Latency (3869081 ms) is above a defined threshold (75 ms)". It lasted for about 20 mins then cleared however the backup took ~60mins to run. No mention of any issues in vcenter server.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Aug 31, 2012 8:41 am
- Full Name: Luca Bedogni
- Contact:
Re: V7 - False alerts
Same problem here. It only happens when Backup job completes. No warning from vcenter.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Aug 14, 2012 2:08 am
- Full Name: Ben Howell
- Contact:
Re: V7 - False alerts
No problems, case id: 00432028Vitaliy S. wrote:Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?
Thanks.
-
- Expert
- Posts: 141
- Liked: 5 times
- Joined: Jan 27, 2010 9:43 am
- Full Name: René Frej Nielsen
- Contact:
Re: V7 - False alerts
I'm having this problem as well... I haven't created a case yet.
-
- Influencer
- Posts: 15
- Liked: 3 times
- Joined: Mar 23, 2012 8:19 am
- Full Name: Thomas Blühmann
- Contact:
Re: V7 - False alerts
We exoperience the same issue. I opened a case: 00434907
-
- Expert
- Posts: 141
- Liked: 5 times
- Joined: Jan 27, 2010 9:43 am
- Full Name: René Frej Nielsen
- Contact:
Re: V7 - False alerts
I now has a case as well: Case # 00434916
-
- Service Provider
- Posts: 2
- Liked: 2 times
- Joined: Aug 27, 2013 12:29 pm
- Full Name: Olaf Reitz
- Contact:
Re: V7 - False alerts
Same issue since Update to v7 case # 00435011
-
- Influencer
- Posts: 15
- Liked: 3 times
- Joined: Mar 23, 2012 8:19 am
- Full Name: Thomas Blühmann
- Contact:
Re: V7 - False alerts
UPDATE:TomBlue wrote:We exoperience the same issue. I opened a case: 00434907
We investigated our case a little deeper, thanks to Veeam Support. We recognized that the spikes we experience in Veeam One are reported as well in vSphere performance monitor. Here is one example where you can see this spikes:
http://sdrv.ms/1ad6cMV (Veeam One: Datastore Highest Latency)
http://sdrv.ms/1ad6kMx (vSphere: VM Disk Highest Latency)
http://sdrv.ms/1ad6raR (vSphere: Datastore Highest Latency)
I cannot find any jobs that can cause these extrem spikes. If there were real latencies like this, I guess we would experience performance issues in any application. As well as the "Datastore Highest Latency" shows negative numbers, strange isn't it? I just found out that around the same time as the high latency is occuring, the machine in question has high CPU load, but not sure yet if this causes the high latency or if hight latency is the reason for CPU load.
We do not have any solution for this yet, but I'm curious if anyone has the same experience and maybe have a solution on this. Of course I agree to Veeam support that this is a problem in vSphere, not in Veeam One, though these error started with update of Veem One + Veeam B&R to version 7.0.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Thomas,
Yes, our R&D team has confirmed that the same spikes are observed in the vSphere Client as well, thanks for the update.
Btw, do you experience these issues at the particular backup job operation? For example, during snapshot commit operation or similar? What processing mode do you use for your backup jobs? HotAdd?
Thanks!
Yes, our R&D team has confirmed that the same spikes are observed in the vSphere Client as well, thanks for the update.
Btw, do you experience these issues at the particular backup job operation? For example, during snapshot commit operation or similar? What processing mode do you use for your backup jobs? HotAdd?
Thanks!
-
- Influencer
- Posts: 15
- Liked: 3 times
- Joined: Mar 23, 2012 8:19 am
- Full Name: Thomas Blühmann
- Contact:
Re: V7 - False alerts
Hi Vitaly!
No, in my case there is no relation to the backup jobs. I do backups only once a night and spikes I can see about 1 - 3 times per hour on this specific VM. On two other VMs I get the error only 1 or 2 times a day. There is also no connection to any jobs on these servers. I still do not have any idea.
Using network mode for backup. Veeam B&R and Veeam One is installed on a physical machine.
No, in my case there is no relation to the backup jobs. I do backups only once a night and spikes I can see about 1 - 3 times per hour on this specific VM. On two other VMs I get the error only 1 or 2 times a day. There is also no connection to any jobs on these servers. I still do not have any idea.
Using network mode for backup. Veeam B&R and Veeam One is installed on a physical machine.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Hmm...at what times do you usually get these spikes? Are there any sort of maintenance tasks scheduled at the time when spikes occur?
-
- Influencer
- Posts: 15
- Liked: 3 times
- Joined: Mar 23, 2012 8:19 am
- Full Name: Thomas Blühmann
- Contact:
Re: V7 - False alerts
Spikes are occuring unregular. I cannot find any relation to jobs running on this or other servers. Here's a screenshot of todays metrics: http://sdrv.ms/18p5TZR. Because this is 1 day period values are a little "foggy" - you do not see spikes that are only half of others. I counted the errors of last 24 hours: There are approx. 25 and no recognisable pattern.
[Update]: There have been some occurences last minutes with maximum values of 127131031 ms , that's 35 hours. That could not be any real latency. At the same time I had very low DAVG/cms values esxtop.
Anyway, thanks for your help, I will try to get any VMware support - or simply ignore this issue, because there is no constraint in "real life"
[Update]: There have been some occurences last minutes with maximum values of 127131031 ms , that's 35 hours. That could not be any real latency. At the same time I had very low DAVG/cms values esxtop.
Anyway, thanks for your help, I will try to get any VMware support - or simply ignore this issue, because there is no constraint in "real life"
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: V7 - False alerts
I'm trying to reproduce this issue in my labs, but I haven't been successful so far. If VMware support nails down this issue, I would appreciate if could come back and update this topic with their findings.
-
- Enthusiast
- Posts: 25
- Liked: 2 times
- Joined: Nov 02, 2012 9:04 pm
- Location: Vreden, Germany
- Contact:
Re: V7 - False alerts
Experiencing the same "problem"
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Hello Guido,
Did you have a chance to check the reason for this behavior with VMware support team? Still wondering if there is any pattern in this "weird" behavior.
Thank you!
Did you have a chance to check the reason for this behavior with VMware support team? Still wondering if there is any pattern in this "weird" behavior.
Thank you!
-
- Enthusiast
- Posts: 64
- Liked: 10 times
- Joined: Jan 16, 2010 9:47 am
- Full Name: Iain McWilliams
- Contact:
Re: V7 - False alerts
We are also seeing this, from what we've observed...
- Alerts started after upgrading to 7.0 from 6.5. Now receiving approx. 3 alerts per day from 50 VMs
- Latency also appears in VCenter performance current graph, but not in daily or any other rollup. May be filtered as part of roll-up process?
- Latency alert always relates to a associated backup snapshot creation/removal
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Hi Iain,
1. You didn't see that in 6.5 cause we have added a couple of new predefined alarms to v7.
2. Yes, VMware rollups real-time performance data, so these spikes are not visible in the daily or other reporting periods.
3. I would recommend to involve VMware support team in the troubleshooting process, as these extremely high values for latency shouldn't be true.
Thank you!
1. You didn't see that in 6.5 cause we have added a couple of new predefined alarms to v7.
2. Yes, VMware rollups real-time performance data, so these spikes are not visible in the daily or other reporting periods.
3. I would recommend to involve VMware support team in the troubleshooting process, as these extremely high values for latency shouldn't be true.
Thank you!
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Oct 30, 2012 1:07 pm
- Full Name: Jeroen Roegiers
- Contact:
Re: V7 - False alerts
Hi,
I'm also experiencing the same problem. No case made. I'll keep an eye on this topic.
I'm also experiencing the same problem. No case made. I'll keep an eye on this topic.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Jeroen, I would also suggest contacting VMware support on this, the more info they have the better.
-
- Expert
- Posts: 187
- Liked: 12 times
- Joined: Sep 01, 2012 2:53 pm
- Full Name: Josue Maldonado
- Contact:
[MERGED] VM total disk latency
Good day everyone,
I just installed veeam 7 no changes made on default config except the email notification, suddenly I started to receive a lot of emails for random VMs with VM total disk latency alert then in minutes the alert goes back to green, this happens more or les each 5 to 10 minutes for multiple vms.
I have noticed no alert is triggered on vcenter itself and no user is complaining about performance. Not sure how risky this message could be.
I attached a copy of the message
image upload
I just installed veeam 7 no changes made on default config except the email notification, suddenly I started to receive a lot of emails for random VMs with VM total disk latency alert then in minutes the alert goes back to green, this happens more or les each 5 to 10 minutes for multiple vms.
I have noticed no alert is triggered on vcenter itself and no user is complaining about performance. Not sure how risky this message could be.
I attached a copy of the message
image upload
-
- Enthusiast
- Posts: 25
- Liked: 2 times
- Joined: Nov 02, 2012 9:04 pm
- Location: Vreden, Germany
- Contact:
Re: V7 - False alerts
hmmmmm latency notifications are gone...
-
- Expert
- Posts: 141
- Liked: 5 times
- Joined: Jan 27, 2010 9:43 am
- Full Name: René Frej Nielsen
- Contact:
Re: VM total disk latency
This is happening here as well. I've been told by support (Case # 00434916) that the cause is that Veeam One 6 didn't know about the counter "Total Disk Latency" which is supposed to be a fairly new counter. Veeam One 7 sees this counter and correctly flags it as an error when vCenter reports these high numbers.
Veeam support suggests that I contact VMware but I don't have a support agreement - only maintenance - so I might just disable this alarm since it looks to be a bug. It's happening when snapshots are being committed during backups.
Veeam support suggests that I contact VMware but I don't have a support agreement - only maintenance - so I might just disable this alarm since it looks to be a bug. It's happening when snapshots are being committed during backups.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: V7 - False alerts
Hi Josue.
Rene is correct, if you do not have any issues you may temporary disable this alarm and open a support case with VMware for further troubleshooting.
Thanks!
Rene is correct, if you do not have any issues you may temporary disable this alarm and open a support case with VMware for further troubleshooting.
Thanks!
-
- Expert
- Posts: 187
- Liked: 12 times
- Joined: Sep 01, 2012 2:53 pm
- Full Name: Josue Maldonado
- Contact:
Re: V7 - False alerts
Ok, thanks for your help. Is just the message at first time seems like critic but after reading on this thread it seems more like a warning.
-
- Novice
- Posts: 5
- Liked: never
- Joined: Jan 03, 2012 9:14 pm
- Full Name: Bill MacEachern
- Contact:
Re: V7 - False alerts
We are also seeing this. Every night I get an alarm for one specific VM:
"Datastore Highest Latency" (3353953467950 ms) is above a defined threshold (75 ms)
If you do the math I believe it works out to 106 years, which is absurd.
"Datastore Highest Latency" (3353953467950 ms) is above a defined threshold (75 ms)
If you do the math I believe it works out to 106 years, which is absurd.
Who is online
Users browsing this forum: No registered users and 13 guests