Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
richardb092
Novice
Posts: 5
Liked: never
Joined: Oct 29, 2013 10:10 am
Full Name: Richard Bance
Contact:

Re: V7 - VMware Issue - False alerts

Post by richardb092 » Nov 03, 2013 9:18 am

Hi
I would of thought so, but am absolutely certain there is no corresponding spike in vsphere...
Thanks Richard
v.Eremin wrote:Hi, Richard,

Actually, Veeam One collects information from vCenter Server/ESXi hosts via VMware API. In other words, we rely mostly on what is informed to us by VMware. Previous users affected by the said behavior confirmed that those spikes were also shown in vSphere Client.

If you’re completely sure, that it is only VeeamOne that keeps informing you about enormous latency numbers, then, open a ticket with our support team and let them verify your setup/environment.

Thanks.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 03, 2013 9:30 am

Hi Richard, in this case please open a support case with our team to identify the reason of these spikes. Thanks!

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 03, 2013 3:57 pm

See post below:

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 03, 2013 4:00 pm

I am also experiencing these. If I may interject another observation, they only occur with systems backed up via a Veeam B&R server that was upgraded from 6.5 to 7. I do not see these occurring on systems that are backed up via a B&R server where 6.5 was removed, and v7 put in cleanly. (VMWare 5.1)

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 03, 2013 4:53 pm

Jeff, do you use parallel VM and virtual disk processing?

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 03, 2013 11:48 pm

Yes, across three proxies. There is no pattern to a certain proxy being assigned when this occurs. All three are VM WinSrv 2008r2 standard configured exactly the same hardware wise.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 04, 2013 1:12 pm

Thanks for the clarification. I believe this issue is somehow related to the load on the datastore produced by the snapshot creation/commit operations. Since VMs can be processed in parallel the load on the datastore is now higher, and I guess this might be one of the reasons for such behavior.

On top of that, our dev team keeps researching this issue and here is what we have recently found out about possible spikes of performance metrics:
VMware KB wrote:If the sampled metric consists of a non-homogeneous set of data, metrics may be skewed in unexpected ways.
Here is a full KB article for further reading > http://kb.vmware.com/selfservice/micros ... Id=2001676

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 04, 2013 6:56 pm

In our case, it is not the same datastore. I see this on 2 of our larger VMs, 1 a SQL server, the other an Exchange server. Both are on separate Dell PowerVault MD32xx, in RAID 5 stripes across 5 drives in the Virtual disk array.

I do agree that is happens during the snapshot release, but it doesn't happen every night. My job settings are Reverse Incremental, active Full on the first Sat of the month, safe removal of snapshots over 100 MB is turned on, 4 restore points. VSS and guest indexing is on.

ddayton
Enthusiast
Posts: 34
Liked: 12 times
Joined: Nov 12, 2012 6:40 pm
Full Name: Don Dayton
Contact:

Re: V7 - VMware Issue - False alerts

Post by ddayton » Nov 05, 2013 4:14 pm

We also have this issue and I had opened a case with Veeam support. However I was simply told this is VMware bug and should contact my VMware support and the Veeam case was closed.

So for now I am simply ignoring the alert and watching the VMware forum and patches for a fix.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 05, 2013 4:18 pm

Hi Don,

These spikes are indeed generated by VMware, Veeam cannot generate performance metrics, since we only require read only account for monitoring.

Can you please navigate to Virtual Disks tab and check if you have any spikes there? Also do you run any backup job at the time when spikes are generated in your VI?

Thanks!

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 05, 2013 4:22 pm

Eh...I think it's a case of he said/she said. I didn't get these with B&R 6.5/Veeam One 6.5. And I don't get them on servers backed up from a clean install of B&R v7. Just the server upgraded from 6.5 to 7.
My ESXi hosts have not been patched since last fall and are all on the same version.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 05, 2013 4:26 pm

jeffmetcalf wrote:Just the server upgraded from 6.5 to 7.
You're referring to upgrade of Veeam B&R, right? Unfortunately, we were unable to reproduce this behavior internally, so any information would help. Thank you!

jeffmetcalf
Influencer
Posts: 16
Liked: never
Joined: Jan 19, 2012 9:03 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by jeffmetcalf » Nov 07, 2013 12:36 pm

It happened again last night to one of my servers. I have submitted both the B&R and One logs to support, case # 00473947.

stevenrodenburg1
Expert
Posts: 125
Liked: 19 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V7 - VMware Issue - False alerts

Post by stevenrodenburg1 » Nov 08, 2013 6:36 am

I have this issue too. I upgraded VeeamOne from 6.5 to 7 last night. This morning, found two unbelievable emails saying:

Object: Veeam-Proxy01
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 08.11.2013 03:03:55
Details: "Datastore Highest Latency" (43441964 ms) is above a defined threshold (75 ms)


Object: Veeam-Proxy02
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 08.11.2013 02:52:35
Details: "Datastore Highest Latency" (15116940 ms) is above a defined threshold (75 ms)


I mean, really? 43 MILLION Milliseconds latency?

The times correspond with the backup-window.
Looked in vCenter ofcource, the highest latency was 55ms for Veeam-Proxy01 and 26ms for Veeam-Proxy02.

Been running B&R v7 (Patch 1) with VeeamOne on v6.5. Never ever happened in VO 6.5 and come on, 43 and 15 million ms ????

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 08, 2013 11:51 am

Guys,

Just a quick update - we have managed to reproduce this issue internally without using Veeam ONE and Veeam B&R, so this is definitely a VMware issue. Our QC team has been emulating backup job behavior by creating and committing snapshots and during one of the commit operations we have caught the Latency spike for VM Virtual Disk and Highest Latency counter for 26-28 millions ms.

We will open a support case with VMware and inform them about the observed behavior, meanwhile our dev team will think about possible workaround for this issue.

P.S. if you see the same behavior, feel free contacting VMware support team as well. I will post our support ticket number here, so you could use it as a reference when talking to VMware technical team.

Thanks!

stevenrodenburg1
Expert
Posts: 125
Liked: 19 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V7 - VMware Issue - False alerts

Post by stevenrodenburg1 » Nov 09, 2013 10:31 pm

Had another one. Ironically, this time the VM in question is the VeeamOne VM itself.

Object: Veeam-One
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 09.11.2013 23:23:30
Details: "Datastore Highest Latency" (14941280 ms) is above a defined threshold (75 ms)

14 millions milliseconds is indeed a bit higher that 75 so the alert is valid ;-)

And again, nothing to see in vCenter.

There were no backups running or anything. The database sits on a SQL server on another spindle-group.

I hope it get's fixed soon. These emails are starting to become annoying and they never occurred in VeeamOne 6.5 and the vSphere version did not change.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 10, 2013 7:26 am

Steven,

You didn't see this alarm in Veeam ONE v6.5, because we didn't monitor this metric - Datastore Highest Latency before. Can you please tell me what tabs in vSphere Client did you check?

As to possible reasons of this weird behavior, I believe snapshots is not the only factor that can cause it.

Thanks!

stevenrodenburg1
Expert
Posts: 125
Liked: 19 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: V7 - VMware Issue - False alerts

Post by stevenrodenburg1 » Nov 10, 2013 9:14 am

Hmmm, there was no backup going on. But this VM, the VeeamOne VM, did indeed have a snapshot (as a means of fallback in case the upgrade to v7 had a problem).
This has been the only time i got that wierd error outside the backup window. Then again, it was the only VM to have a "regular, user initiated" snapshot. All other snapshots are taken by Veeam B&R v7 and the times of these alerts more or less correspond with those. Of the many backup jobs that are running, only a very few trigger such an alert.

In vCenter, i look at the performance counters of the VM -> advanced -> datastore.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 10, 2013 9:25 am

Take a look at Virtual Disk tab, this should record all the spikes occurred. Once I have information from VMware support team on these spikes I will let everyone know.

athome
Service Provider
Posts: 57
Liked: 1 time
Joined: Jul 16, 2010 9:26 am
Full Name: Anders Thome
Contact:

Re: V7 - VMware Issue - False alerts

Post by athome » Nov 13, 2013 10:14 pm

I see the same issue.

vSphere 5.1U1B with latest patches.
Veeam B&R 7 clean install.
Veeam ONE 7 upgraded from 6.5

A

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 14, 2013 6:49 am

Anders, have you been able to open a support case with VMware on this, as I have advised in the previous posts?

athome
Service Provider
Posts: 57
Liked: 1 time
Joined: Jul 16, 2010 9:26 am
Full Name: Anders Thome
Contact:

Re: V7 - VMware Issue - False alerts

Post by athome » Nov 14, 2013 7:18 am

Vitaliy S. wrote:Anders, have you been able to open a support case with VMware on this, as I have advised in the previous posts?
I have, but I didnt see any casenumber from VMwarethat I could refer to.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 14, 2013 11:50 am

Here is a VMware case number you can use as a reference - 13402946911

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 24, 2013 9:07 pm 1 person likes this post

Just another update - VMware has acknowledged this behavior. I'm not sure they are going to fix it soon, but we have come up with a workaround that should address this issue in the upcoming Veeam ONE v7 R2 update.

Here is a Veeam KB article for more info > http://www.veeam.com/kb1836

athome
Service Provider
Posts: 57
Liked: 1 time
Joined: Jul 16, 2010 9:26 am
Full Name: Anders Thome
Contact:

Re: V7 - VMware Issue - False alerts

Post by athome » Nov 24, 2013 9:55 pm

Thanks for update Vitaliy!

Do you have an eta of Veeam ONE v7 R2?

Is there a workaround for it until then? (There were someone earlier in the post that stated that issue only was seen when alert was enabled in Veeam ONE.)

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 25, 2013 5:56 am

These earlier statements were not accurate and during WebEx sessions with our engineer we couldn't confirm it. Veeam ONE v7 R2 is going to be available very soon. As a workaround for now you may want to suppress this alarm during heavy datastore I/O operations.

Mazzotta
Veeam Software
Posts: 107
Liked: 12 times
Joined: Dec 14, 2010 9:26 am
Full Name: Gianluca Mazzotta
Contact:

Re: V7 - VMware Issue - False alerts

Post by Mazzotta » Nov 25, 2013 1:48 pm

Vitaliy S. wrote:Here is a VMware case number you can use as a reference - 13402946911
Any news on VMware side about this?

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 25, 2013 1:52 pm

No news, they have created a KB I have posted above and that's all I know from our support team.

hauckm
Novice
Posts: 4
Liked: never
Joined: Dec 21, 2012 2:44 pm
Contact:

Re: V7 - VMware Issue - False alerts

Post by hauckm » Nov 27, 2013 5:05 pm

I just now got off the phone with VMware support for a ticket we opened for this issue.

We opened the ticket a week or so ago and at that time gave VMware the Veeam ticket as a reference.

I was told today by VMware support the Veeam ticket has been closed and this is not a VMware issue. I was also told if I wanted to pursue the open ticket we would have to do so without reference to the now closed Veeam ticket. In other words we would have to pursue this as a latency issue with our storage.

Now this is the information I received from one guy, so it may or may not be VMware's stance on this issue. I am just passing along what I was told.

I will be disabling this alarm until R2 is available.

Vitaliy S.
Product Manager
Posts: 22557
Liked: 1480 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: V7 - VMware Issue - False alerts

Post by Vitaliy S. » Nov 28, 2013 7:50 am

Hauckm,

Thank you for the update. I think I can understand the point VMware engineer was heading to, they also observe these high values in the vSphere Client and other monitoring tools, but I'm not sure it is correct to blame a storage vendor for that.

Anyway, I have just talked to our QC team and the RC build of Veeam ONE v7 R2 looks stable, so the release is almost there.

Thank you!

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests