V7 - VMware Issue - False alerts

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Re: V7 - VMware Issue - False alerts

Veeam Logoby richardb092 » Sun Nov 03, 2013 9:18 am

Hi
I would of thought so, but am absolutely certain there is no corresponding spike in vsphere...
Thanks Richard

v.Eremin wrote:Hi, Richard,

Actually, Veeam One collects information from vCenter Server/ESXi hosts via VMware API. In other words, we rely mostly on what is informed to us by VMware. Previous users affected by the said behavior confirmed that those spikes were also shown in vSphere Client.

If you’re completely sure, that it is only VeeamOne that keeps informing you about enormous latency numbers, then, open a ticket with our support team and let them verify your setup/environment.

Thanks.
richardb092
Novice
 
Posts: 5
Liked: never
Joined: Tue Oct 29, 2013 10:10 am
Full Name: Richard Bance

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Sun Nov 03, 2013 9:30 am

Hi Richard, in this case please open a support case with our team to identify the reason of these spikes. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby jeffmetcalf » Sun Nov 03, 2013 3:57 pm

See post below:
jeffmetcalf
Influencer
 
Posts: 16
Liked: never
Joined: Thu Jan 19, 2012 9:03 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby jeffmetcalf » Sun Nov 03, 2013 4:00 pm

I am also experiencing these. If I may interject another observation, they only occur with systems backed up via a Veeam B&R server that was upgraded from 6.5 to 7. I do not see these occurring on systems that are backed up via a B&R server where 6.5 was removed, and v7 put in cleanly. (VMWare 5.1)
jeffmetcalf
Influencer
 
Posts: 16
Liked: never
Joined: Thu Jan 19, 2012 9:03 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Sun Nov 03, 2013 4:53 pm

Jeff, do you use parallel VM and virtual disk processing?
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby jeffmetcalf » Sun Nov 03, 2013 11:48 pm

Yes, across three proxies. There is no pattern to a certain proxy being assigned when this occurs. All three are VM WinSrv 2008r2 standard configured exactly the same hardware wise.
jeffmetcalf
Influencer
 
Posts: 16
Liked: never
Joined: Thu Jan 19, 2012 9:03 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Mon Nov 04, 2013 1:12 pm

Thanks for the clarification. I believe this issue is somehow related to the load on the datastore produced by the snapshot creation/commit operations. Since VMs can be processed in parallel the load on the datastore is now higher, and I guess this might be one of the reasons for such behavior.

On top of that, our dev team keeps researching this issue and here is what we have recently found out about possible spikes of performance metrics:
VMware KB wrote:If the sampled metric consists of a non-homogeneous set of data, metrics may be skewed in unexpected ways.

Here is a full KB article for further reading > http://kb.vmware.com/selfservice/micros ... Id=2001676
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby jeffmetcalf » Mon Nov 04, 2013 6:56 pm

In our case, it is not the same datastore. I see this on 2 of our larger VMs, 1 a SQL server, the other an Exchange server. Both are on separate Dell PowerVault MD32xx, in RAID 5 stripes across 5 drives in the Virtual disk array.

I do agree that is happens during the snapshot release, but it doesn't happen every night. My job settings are Reverse Incremental, active Full on the first Sat of the month, safe removal of snapshots over 100 MB is turned on, 4 restore points. VSS and guest indexing is on.
jeffmetcalf
Influencer
 
Posts: 16
Liked: never
Joined: Thu Jan 19, 2012 9:03 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby ddayton » Tue Nov 05, 2013 4:14 pm

We also have this issue and I had opened a case with Veeam support. However I was simply told this is VMware bug and should contact my VMware support and the Veeam case was closed.

So for now I am simply ignoring the alert and watching the VMware forum and patches for a fix.
ddayton
Enthusiast
 
Posts: 31
Liked: 10 times
Joined: Mon Nov 12, 2012 6:40 pm
Full Name: Don Dayton

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Tue Nov 05, 2013 4:18 pm

Hi Don,

These spikes are indeed generated by VMware, Veeam cannot generate performance metrics, since we only require read only account for monitoring.

Can you please navigate to Virtual Disks tab and check if you have any spikes there? Also do you run any backup job at the time when spikes are generated in your VI?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby jeffmetcalf » Tue Nov 05, 2013 4:22 pm

Eh...I think it's a case of he said/she said. I didn't get these with B&R 6.5/Veeam One 6.5. And I don't get them on servers backed up from a clean install of B&R v7. Just the server upgraded from 6.5 to 7.
My ESXi hosts have not been patched since last fall and are all on the same version.
jeffmetcalf
Influencer
 
Posts: 16
Liked: never
Joined: Thu Jan 19, 2012 9:03 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Tue Nov 05, 2013 4:26 pm

jeffmetcalf wrote:Just the server upgraded from 6.5 to 7.

You're referring to upgrade of Veeam B&R, right? Unfortunately, we were unable to reproduce this behavior internally, so any information would help. Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby jeffmetcalf » Thu Nov 07, 2013 12:36 pm

It happened again last night to one of my servers. I have submitted both the B&R and One logs to support, case # 00473947.
jeffmetcalf
Influencer
 
Posts: 16
Liked: never
Joined: Thu Jan 19, 2012 9:03 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby stevenrodenburg1 » Fri Nov 08, 2013 6:36 am

I have this issue too. I upgraded VeeamOne from 6.5 to 7 last night. This morning, found two unbelievable emails saying:

Object:Veeam-Proxy01
Object Type:Virtual Machine
Parent Object:Normal
Status:Error
Alarm:VM total disk latency
Time:08.11.2013 03:03:55
Details:"Datastore Highest Latency" (43441964 ms) is above a defined threshold (75 ms)


Object:Veeam-Proxy02
Object Type:Virtual Machine
Parent Object:Normal
Status:Error
Alarm:VM total disk latency
Time:08.11.2013 02:52:35
Details:"Datastore Highest Latency" (15116940 ms) is above a defined threshold (75 ms)


I mean, really? 43 MILLION Milliseconds latency?

The times correspond with the backup-window.
Looked in vCenter ofcource, the highest latency was 55ms for Veeam-Proxy01 and 26ms for Veeam-Proxy02.

Been running B&R v7 (Patch 1) with VeeamOne on v6.5. Never ever happened in VO 6.5 and come on, 43 and 15 million ms ????
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Fri Nov 08, 2013 11:51 am

Guys,

Just a quick update - we have managed to reproduce this issue internally without using Veeam ONE and Veeam B&R, so this is definitely a VMware issue. Our QC team has been emulating backup job behavior by creating and committing snapshots and during one of the commit operations we have caught the Latency spike for VM Virtual Disk and Highest Latency counter for 26-28 millions ms.

We will open a support case with VMware and inform them about the observed behavior, meanwhile our dev team will think about possible workaround for this issue.

P.S. if you see the same behavior, feel free contacting VMware support team as well. I will post our support ticket number here, so you could use it as a reference when talking to VMware technical team.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

PreviousNext

Return to Veeam ONE



Who is online

Users browsing this forum: No registered users and 3 guests