Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
Post Reply
richardb092
Novice
Posts: 5
Liked: never
Joined: Oct 29, 2013 10:10 am
Full Name: Richard Bance
Contact:

Re: V7 - VMware Issue - False alerts

Post by richardb092 » Nov 03, 2013 9:18 am

Hi
I would of thought so, but am absolutely certain there is no corresponding spike in vsphere...
Thanks Richard
v.Eremin wrote:Hi, Richard,

Actually, Veeam One collects information from vCenter Server/ESXi hosts via VMware API. In other words, we rely mostly on what is informed to us by VMware. Previous users affected by the said behavior confirmed that those spikes were also shown in vSphere Client.

If you’re completely sure, that it is only VeeamOne that keeps informing you about enormous latency numbers, then, open a ticket with our support team and let them verify your setup/environment.

Thanks.

Vitaliy S.
Product Manager
Posts: 22127
Liked: 1380 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 03, 2013 9:30 am

Hi Richard, in this case please open a support case with our team to identify the reason of these spikes. Thanks!

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 03, 2013 3:57 pm

See post below:

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 03, 2013 4:00 pm

I am also experiencing these. If I may interject another observation, they only occur with systems backed up via a Veeam B&R server that was upgraded from 6.5 to 7. I do not see these occurring on systems that are backed up via a B&R server where 6.5 was removed, and v7 put in cleanly. (VMWare 5.1)

Vitaliy S.
Product Manager
Posts: 22127
Liked: 1380 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 03, 2013 4:53 pm

Jeff, do you use parallel VM and virtual disk processing?

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 03, 2013 11:48 pm

Yes, across three proxies. There is no pattern to a certain proxy being assigned when this occurs. All three are VM WinSrv 2008r2 standard configured exactly the same hardware wise.

Vitaliy S.
Product Manager
Posts: 22127
Liked: 1380 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 04, 2013 1:12 pm

Thanks for the clarification. I believe this issue is somehow related to the load on the datastore produced by the snapshot creation/commit operations. Since VMs can be processed in parallel the load on the datastore is now higher, and I guess this might be one of the reasons for such behavior.

On top of that, our dev team keeps researching this issue and here is what we have recently found out about possible spikes of performance metrics:
VMware KB wrote:If the sampled metric consists of a non-homogeneous set of data, metrics may be skewed in unexpected ways.
Here is a full KB article for further reading > http://kb.vmware.com/selfservice/micros ... Id=2001676

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 04, 2013 6:56 pm

In our case, it is not the same datastore. I see this on 2 of our larger VMs, 1 a SQL server, the other an Exchange server. Both are on separate Dell PowerVault MD32xx, in RAID 5 stripes across 5 drives in the Virtual disk array.

I do agree that is happens during the snapshot release, but it doesn't happen every night. My job settings are Reverse Incremental, active Full on the first Sat of the month, safe removal of snapshots over 100 MB is turned on, 4 restore points. VSS and guest indexing is on.

ddayton
Enthusiast
Posts: 34
Liked: 12 times
Joined: Nov 12, 2012 6:40 pm
Full Name: Don Dayton
Contact:

Re: V7 - VMware Issue - False alerts

Post by ddayton » Nov 05, 2013 4:14 pm

We also have this issue and I had opened a case with Veeam support. However I was simply told this is VMware bug and should contact my VMware support and the Veeam case was closed.

So for now I am simply ignoring the alert and watching the VMware forum and patches for a fix.

Vitaliy S.
Product Manager
Posts: 22127
Liked: 1380 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 05, 2013 4:18 pm

Hi Don,

These spikes are indeed generated by VMware, Veeam cannot generate performance metrics, since we only require read only account for monitoring.

Can you please navigate to Virtual Disks tab and check if you have any spikes there? Also do you run any backup job at the time when spikes are generated in your VI?

Thanks!

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 05, 2013 4:22 pm

Eh...I think it's a case of he said/she said. I didn't get these with B&R 6.5/Veeam One 6.5. And I don't get them on servers backed up from a clean install of B&R v7. Just the server upgraded from 6.5 to 7.
My ESXi hosts have not been patched since last fall and are all on the same version.

Vitaliy S.
Product Manager
Posts: 22127
Liked: 1380 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 05, 2013 4:26 pm

jeffmetcalf wrote:Just the server upgraded from 6.5 to 7.
You're referring to upgrade of Veeam B&R, right? Unfortunately, we were unable to reproduce this behavior internally, so any information would help. Thank you!

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 07, 2013 12:36 pm

It happened again last night to one of my servers. I have submitted both the B&R and One logs to support, case # 00473947.

stevenrodenburg1
Expert
Posts: 125
Liked: 19 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V7 - VMware Issue - False alerts

Post by stevenrodenburg1 » Nov 08, 2013 6:36 am

I have this issue too. I upgraded VeeamOne from 6.5 to 7 last night. This morning, found two unbelievable emails saying:

Object: Veeam-Proxy01
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 08.11.2013 03:03:55
Details: "Datastore Highest Latency" (43441964 ms) is above a defined threshold (75 ms)


Object: Veeam-Proxy02
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 08.11.2013 02:52:35
Details: "Datastore Highest Latency" (15116940 ms) is above a defined threshold (75 ms)


I mean, really? 43 MILLION Milliseconds latency?

The times correspond with the backup-window.
Looked in vCenter ofcource, the highest latency was 55ms for Veeam-Proxy01 and 26ms for Veeam-Proxy02.

Been running B&R v7 (Patch 1) with VeeamOne on v6.5. Never ever happened in VO 6.5 and come on, 43 and 15 million ms ????

Vitaliy S.
Product Manager
Posts: 22127
Liked: 1380 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 08, 2013 11:51 am

Guys,

Just a quick update - we have managed to reproduce this issue internally without using Veeam ONE and Veeam B&R, so this is definitely a VMware issue. Our QC team has been emulating backup job behavior by creating and committing snapshots and during one of the commit operations we have caught the Latency spike for VM Virtual Disk and Highest Latency counter for 26-28 millions ms.

We will open a support case with VMware and inform them about the observed behavior, meanwhile our dev team will think about possible workaround for this issue.

P.S. if you see the same behavior, feel free contacting VMware support team as well. I will post our support ticket number here, so you could use it as a reference when talking to VMware technical team.

Thanks!

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 3 guests