V7 - VMware Issue - False alerts

[AlainRussell]

Since upgrading to v7 I've been getting some false alerts as below, I can't see a corresponding warning in Vcenter so not sure where to look to fix? Any ideas?

"Datastore Highest Latency" (28251344 ms) is above a defined threshold (75 ms)"

[Vitaliy S.]

Hi Alain,

What values are displayed on the performance graphs in the vSphere Client for this metric? Can you please open a support case with our technical team and let them have a look at your debug logs?

Thanks!

[AlainRussell]

There are no alerts showing (screenshot attached), strange that the false alert coincides with the Backup job completing. I'm travelling at the moment but will open a support case when I get back - not a major issue currently.

http://utility.blackpepper.co.nz/alain/VCenter.png
http://utility.blackpepper.co.nz/alain/Alert.png
http://utility.blackpepper.co.nz/alain/Datastore.png

Datastore latency shows a max of 19ms read, 16ms max over the last 2 hours.

[Vitaliy S.]

Ok, let me check with our QC team if it is a known issue or not.

[emsaus]

We are also seeing the same thing. Upgraded to V7 yesterday the next backup that ran generated the same alarm. "Datastore Highest Latency (3869081 ms) is above a defined threshold (75 ms)". It lasted for about 20 mins then cleared however the backup took ~60mins to run. No mention of any issues in vcenter server.

[Vitaliy S.]

Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?

[Jadawyn]

Same problem here. It only happens when Backup job completes. No warning from vcenter.

[emsaus]

Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?

No problems, case id: 00432028

Thanks.

[rfn]

I'm having this problem as well... I haven't created a case yet.

[TomBlue]

We exoperience the same issue. I opened a case: 00434907

[rfn]

I now has a case as well: Case # 00434916

[Orven]

Same issue since Update to v7 case # 00435011

[TomBlue]

We exoperience the same issue. I opened a case: 00434907

UPDATE:

We investigated our case a little deeper, thanks to Veeam Support. We recognized that the spikes we experience in Veeam One are reported as well in vSphere performance monitor. Here is one example where you can see this spikes:

http://sdrv.ms/1ad6cMV (Veeam One: Datastore Highest Latency)
http://sdrv.ms/1ad6kMx (vSphere: VM Disk Highest Latency)
http://sdrv.ms/1ad6raR (vSphere: Datastore Highest Latency)

I cannot find any jobs that can cause these extrem spikes. If there were real latencies like this, I guess we would experience performance issues in any application. As well as the "Datastore Highest Latency" shows negative numbers, strange isn't it? I just found out that around the same time as the high latency is occuring, the machine in question has high CPU load, but not sure yet if this causes the high latency or if hight latency is the reason for CPU load.

We do not have any solution for this yet, but I'm curious if anyone has the same experience and maybe have a solution on this. Of course I agree to Veeam support that this is a problem in vSphere, not in Veeam One, though these error started with update of Veem One + Veeam B&R to version 7.0.

[Vitaliy S.]

Thomas,

Yes, our R&D team has confirmed that the same spikes are observed in the vSphere Client as well, thanks for the update.

Btw, do you experience these issues at the particular backup job operation? For example, during snapshot commit operation or similar? What processing mode do you use for your backup jobs? HotAdd?

Thanks!

[TomBlue]

Hi Vitaly!
No, in my case there is no relation to the backup jobs. I do backups only once a night and spikes I can see about 1 - 3 times per hour on this specific VM. On two other VMs I get the error only 1 or 2 times a day. There is also no connection to any jobs on these servers. I still do not have any idea.

Using network mode for backup. Veeam B&R and Veeam One is installed on a physical machine.

[Vitaliy S.]

Hmm...at what times do you usually get these spikes? Are there any sort of maintenance tasks scheduled at the time when spikes occur?

[TomBlue]

Spikes are occuring unregular. I cannot find any relation to jobs running on this or other servers. Here's a screenshot of todays metrics: http://sdrv.ms/18p5TZR. Because this is 1 day period values are a little "foggy" - you do not see spikes that are only half of others. I counted the errors of last 24 hours: There are approx. 25 and no recognisable pattern.

[Update]: There have been some occurences last minutes with maximum values of 127131031 ms , that's 35 hours. That could not be any real latency. At the same time I had very low DAVG/cms values esxtop.

Anyway, thanks for your help, I will try to get any VMware support - or simply ignore this issue, because there is no constraint in "real life"

[foggy]

I'm trying to reproduce this issue in my labs, but I haven't been successful so far. If VMware support nails down this issue, I would appreciate if could come back and update this topic with their findings.

[GL@MO]

Experiencing the same "problem"

[Vitaliy S.]

Hello Guido,

Did you have a chance to check the reason for this behavior with VMware support team? Still wondering if there is any pattern in this "weird" behavior.

Thank you!

[mcwill]

We are also seeing this, from what we've observed...

• Alerts started after upgrading to 7.0 from 6.5. Now receiving approx. 3 alerts per day from 50 VMs
• Latency also appears in VCenter performance current graph, but not in daily or any other rollup. May be filtered as part of roll-up process?
• Latency alert always relates to a associated backup snapshot creation/removal

Backups are processed by hotadd and we have parallel processing enabled.

[Vitaliy S.]

Hi Iain,

1. You didn't see that in 6.5 cause we have added a couple of new predefined alarms to v7.
2. Yes, VMware rollups real-time performance data, so these spikes are not visible in the daily or other reporting periods.
3. I would recommend to involve VMware support team in the troubleshooting process, as these extremely high values for latency shouldn't be true.

Thank you!

[JROleon]

Hi,
I'm also experiencing the same problem. No case made. I'll keep an eye on this topic.

[Vitaliy S.]

Jeroen, I would also suggest contacting VMware support on this, the more info they have the better.

[JosueM]

Good day everyone,

I just installed veeam 7 no changes made on default config except the email notification, suddenly I started to receive a lot of emails for random VMs with VM total disk latency alert then in minutes the alert goes back to green, this happens more or les each 5 to 10 minutes for multiple vms.

I have noticed no alert is triggered on vcenter itself and no user is complaining about performance. Not sure how risky this message could be.

I attached a copy of the message

image upload

[GL@MO]

hmmmmm latency notifications are gone...

[rfn]

This is happening here as well. I've been told by support (Case # 00434916) that the cause is that Veeam One 6 didn't know about the counter "Total Disk Latency" which is supposed to be a fairly new counter. Veeam One 7 sees this counter and correctly flags it as an error when vCenter reports these high numbers.

Veeam support suggests that I contact VMware but I don't have a support agreement - only maintenance - so I might just disable this alarm since it looks to be a bug. It's happening when snapshots are being committed during backups.

[Vitaliy S.]

Hi Josue.

Rene is correct, if you do not have any issues you may temporary disable this alarm and open a support case with VMware for further troubleshooting.

Thanks!

[JosueM]

Ok, thanks for your help. Is just the message at first time seems like critic but after reading on this thread it seems more like a warning.

[KOMBillM]

We are also seeing this. Every night I get an alarm for one specific VM:

"Datastore Highest Latency" (3353953467950 ms) is above a defined threshold (75 ms)

If you do the math I believe it works out to 106 years, which is absurd.