V7 - VMware Issue - False alerts

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Re: V7 - VMware Issue - False alerts

Veeam Logoby stevenrodenburg1 » Sat Nov 09, 2013 10:31 pm

Had another one. Ironically, this time the VM in question is the VeeamOne VM itself.

Object: Veeam-One
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 09.11.2013 23:23:30
Details: "Datastore Highest Latency" (14941280 ms) is above a defined threshold (75 ms)

14 millions milliseconds is indeed a bit higher that 75 so the alert is valid ;-)

And again, nothing to see in vCenter.

There were no backups running or anything. The database sits on a SQL server on another spindle-group.

I hope it get's fixed soon. These emails are starting to become annoying and they never occurred in VeeamOne 6.5 and the vSphere version did not change.
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Sun Nov 10, 2013 7:26 am

Steven,

You didn't see this alarm in Veeam ONE v6.5, because we didn't monitor this metric - Datastore Highest Latency before. Can you please tell me what tabs in vSphere Client did you check?

As to possible reasons of this weird behavior, I believe snapshots is not the only factor that can cause it.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby stevenrodenburg1 » Sun Nov 10, 2013 9:14 am

Hmmm, there was no backup going on. But this VM, the VeeamOne VM, did indeed have a snapshot (as a means of fallback in case the upgrade to v7 had a problem).
This has been the only time i got that wierd error outside the backup window. Then again, it was the only VM to have a "regular, user initiated" snapshot. All other snapshots are taken by Veeam B&R v7 and the times of these alerts more or less correspond with those. Of the many backup jobs that are running, only a very few trigger such an alert.

In vCenter, i look at the performance counters of the VM -> advanced -> datastore.
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Sun Nov 10, 2013 9:25 am

Take a look at Virtual Disk tab, this should record all the spikes occurred. Once I have information from VMware support team on these spikes I will let everyone know.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby athome » Wed Nov 13, 2013 10:14 pm

I see the same issue.

vSphere 5.1U1B with latest patches.
Veeam B&R 7 clean install.
Veeam ONE 7 upgraded from 6.5

A
athome
Enthusiast
 
Posts: 56
Liked: 1 time
Joined: Fri Jul 16, 2010 9:26 am
Full Name: Anders Thome

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Thu Nov 14, 2013 6:49 am

Anders, have you been able to open a support case with VMware on this, as I have advised in the previous posts?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby athome » Thu Nov 14, 2013 7:18 am

Vitaliy S. wrote:Anders, have you been able to open a support case with VMware on this, as I have advised in the previous posts?


I have, but I didnt see any casenumber from VMwarethat I could refer to.
athome
Enthusiast
 
Posts: 56
Liked: 1 time
Joined: Fri Jul 16, 2010 9:26 am
Full Name: Anders Thome

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Thu Nov 14, 2013 11:50 am

Here is a VMware case number you can use as a reference - 13402946911
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Sun Nov 24, 2013 9:07 pm 1 person likes this post

Just another update - VMware has acknowledged this behavior. I'm not sure they are going to fix it soon, but we have come up with a workaround that should address this issue in the upcoming Veeam ONE v7 R2 update.

Here is a Veeam KB article for more info > http://www.veeam.com/kb1836
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby athome » Sun Nov 24, 2013 9:55 pm

Thanks for update Vitaliy!

Do you have an eta of Veeam ONE v7 R2?

Is there a workaround for it until then? (There were someone earlier in the post that stated that issue only was seen when alert was enabled in Veeam ONE.)
athome
Enthusiast
 
Posts: 56
Liked: 1 time
Joined: Fri Jul 16, 2010 9:26 am
Full Name: Anders Thome

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Mon Nov 25, 2013 5:56 am

These earlier statements were not accurate and during WebEx sessions with our engineer we couldn't confirm it. Veeam ONE v7 R2 is going to be available very soon. As a workaround for now you may want to suppress this alarm during heavy datastore I/O operations.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby Mazzotta » Mon Nov 25, 2013 1:48 pm

Vitaliy S. wrote:Here is a VMware case number you can use as a reference - 13402946911


Any news on VMware side about this?
Mazzotta
Veeam Software
 
Posts: 106
Liked: 12 times
Joined: Tue Dec 14, 2010 9:26 am
Full Name: Gianluca Mazzotta

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Mon Nov 25, 2013 1:52 pm

No news, they have created a KB I have posted above and that's all I know from our support team.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: V7 - VMware Issue - False alerts

Veeam Logoby hauckm » Wed Nov 27, 2013 5:05 pm

I just now got off the phone with VMware support for a ticket we opened for this issue.

We opened the ticket a week or so ago and at that time gave VMware the Veeam ticket as a reference.

I was told today by VMware support the Veeam ticket has been closed and this is not a VMware issue. I was also told if I wanted to pursue the open ticket we would have to do so without reference to the now closed Veeam ticket. In other words we would have to pursue this as a latency issue with our storage.

Now this is the information I received from one guy, so it may or may not be VMware's stance on this issue. I am just passing along what I was told.

I will be disabling this alarm until R2 is available.
hauckm
Novice
 
Posts: 4
Liked: never
Joined: Fri Dec 21, 2012 2:44 pm

Re: V7 - VMware Issue - False alerts

Veeam Logoby Vitaliy S. » Thu Nov 28, 2013 7:50 am

Hauckm,

Thank you for the update. I think I can understand the point VMware engineer was heading to, they also observe these high values in the vSphere Client and other monitoring tools, but I'm not sure it is correct to blame a storage vendor for that.

Anyway, I have just talked to our QC team and the RC build of Veeam ONE v7 R2 looks stable, so the release is almost there.

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

PreviousNext

Return to Veeam ONE



Who is online

Users browsing this forum: No registered users and 2 guests