V7 - VMware Issue - False alerts

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

V7 - VMware Issue - False alerts

Veeam Logoby AlainRussell » Sun Aug 18, 2013 8:32 pm

Since upgrading to v7 I've been getting some false alerts as below, I can't see a corresponding warning in Vcenter so not sure where to look to fix? Any ideas?

"Datastore Highest Latency" (28251344 ms) is above a defined threshold (75 ms)"
AlainRussell
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Sat Aug 27, 2011 12:04 am
Full Name: Alain Russell

Re: V7 - False alerts

Veeam Logoby Vitaliy S. » Sun Aug 18, 2013 8:41 pm

Hi Alain,

What values are displayed on the performance graphs in the vSphere Client for this metric? Can you please open a support case with our technical team and let them have a look at your debug logs?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - False alerts

Veeam Logoby AlainRussell » Mon Aug 19, 2013 1:27 am

There are no alerts showing (screenshot attached), strange that the false alert coincides with the Backup job completing. I'm travelling at the moment but will open a support case when I get back - not a major issue currently.

http://utility.blackpepper.co.nz/alain/VCenter.png
http://utility.blackpepper.co.nz/alain/Alert.png
http://utility.blackpepper.co.nz/alain/Datastore.png

Datastore latency shows a max of 19ms read, 16ms max over the last 2 hours.
AlainRussell
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Sat Aug 27, 2011 12:04 am
Full Name: Alain Russell

Re: V7 - False alerts

Veeam Logoby Vitaliy S. » Mon Aug 19, 2013 9:15 am

Ok, let me check with our QC team if it is a known issue or not.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - False alerts

Veeam Logoby emsaus » Mon Aug 19, 2013 11:00 pm

We are also seeing the same thing. Upgraded to V7 yesterday the next backup that ran generated the same alarm. "Datastore Highest Latency (3869081 ms) is above a defined threshold (75 ms)". It lasted for about 20 mins then cleared however the backup took ~60mins to run. No mention of any issues in vcenter server.
emsaus
Novice
 
Posts: 3
Liked: never
Joined: Tue Aug 14, 2012 2:08 am
Full Name: Ben Howell

Re: V7 - False alerts

Veeam Logoby Vitaliy S. » Tue Aug 20, 2013 8:02 am

Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - False alerts

Veeam Logoby Jadawyn » Tue Aug 20, 2013 10:50 am

Same problem here. It only happens when Backup job completes. No warning from vcenter.
Jadawyn
Lurker
 
Posts: 1
Liked: never
Joined: Fri Aug 31, 2012 8:41 am
Full Name: Luca Bedogni

Re: V7 - False alerts

Veeam Logoby emsaus » Wed Aug 21, 2013 5:26 am

Vitaliy S. wrote:Ben, can you also open a support case with our technical team and post your ticket number, so I could discuss it with our dev team?


No problems, case id: 00432028

Thanks.
emsaus
Novice
 
Posts: 3
Liked: never
Joined: Tue Aug 14, 2012 2:08 am
Full Name: Ben Howell

Re: V7 - False alerts

Veeam Logoby rfn » Sun Aug 25, 2013 8:30 pm

I'm having this problem as well... I haven't created a case yet.
rfn
Expert
 
Posts: 131
Liked: 4 times
Joined: Wed Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen

Re: V7 - False alerts

Veeam Logoby TomBlue » Tue Aug 27, 2013 8:50 am

We exoperience the same issue. I opened a case: 00434907
TomBlue
Influencer
 
Posts: 15
Liked: 3 times
Joined: Fri Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann

Re: V7 - False alerts

Veeam Logoby rfn » Tue Aug 27, 2013 9:11 am

I now has a case as well: Case # 00434916
rfn
Expert
 
Posts: 131
Liked: 4 times
Joined: Wed Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen

Re: V7 - False alerts

Veeam Logoby Orven » Tue Aug 27, 2013 12:31 pm

Same issue since Update to v7 case # 00435011
Orven
Lurker
 
Posts: 1
Liked: never
Joined: Tue Aug 27, 2013 12:29 pm
Full Name: Olaf Reitz

Re: V7 - False alerts

Veeam Logoby TomBlue » Mon Sep 02, 2013 7:26 am

TomBlue wrote:We exoperience the same issue. I opened a case: 00434907


UPDATE:

We investigated our case a little deeper, thanks to Veeam Support. We recognized that the spikes we experience in Veeam One are reported as well in vSphere performance monitor. Here is one example where you can see this spikes:

http://sdrv.ms/1ad6cMV (Veeam One: Datastore Highest Latency)
http://sdrv.ms/1ad6kMx (vSphere: VM Disk Highest Latency)
http://sdrv.ms/1ad6raR (vSphere: Datastore Highest Latency)

I cannot find any jobs that can cause these extrem spikes. If there were real latencies like this, I guess we would experience performance issues in any application. As well as the "Datastore Highest Latency" shows negative numbers, strange isn't it? I just found out that around the same time as the high latency is occuring, the machine in question has high CPU load, but not sure yet if this causes the high latency or if hight latency is the reason for CPU load.

We do not have any solution for this yet, but I'm curious if anyone has the same experience and maybe have a solution on this. Of course I agree to Veeam support that this is a problem in vSphere, not in Veeam One, though these error started with update of Veem One + Veeam B&R to version 7.0.
TomBlue
Influencer
 
Posts: 15
Liked: 3 times
Joined: Fri Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann

Re: V7 - False alerts

Veeam Logoby Vitaliy S. » Mon Sep 02, 2013 11:25 am

Thomas,

Yes, our R&D team has confirmed that the same spikes are observed in the vSphere Client as well, thanks for the update.

Btw, do you experience these issues at the particular backup job operation? For example, during snapshot commit operation or similar? What processing mode do you use for your backup jobs? HotAdd?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - False alerts

Veeam Logoby TomBlue » Tue Sep 03, 2013 6:06 pm

Hi Vitaly!
No, in my case there is no relation to the backup jobs. I do backups only once a night and spikes I can see about 1 - 3 times per hour on this specific VM. On two other VMs I get the error only 1 or 2 times a day. There is also no connection to any jobs on these servers. I still do not have any idea.

Using network mode for backup. Veeam B&R and Veeam One is installed on a physical machine.
TomBlue
Influencer
 
Posts: 15
Liked: 3 times
Joined: Fri Mar 23, 2012 8:19 am
Full Name: Thomas Blühmann

Next

Return to Veeam ONE



Who is online

Users browsing this forum: No registered users and 4 guests