V7 - VMware Issue - False alerts

[richardb092]

Hi
I would of thought so, but am absolutely certain there is no corresponding spike in vsphere...
Thanks Richard

Hi, Richard,

Actually, Veeam One collects information from vCenter Server/ESXi hosts via VMware API. In other words, we rely mostly on what is informed to us by VMware. Previous users affected by the said behavior confirmed that those spikes were also shown in vSphere Client.

If you’re completely sure, that it is only VeeamOne that keeps informing you about enormous latency numbers, then, open a ticket with our support team and let them verify your setup/environment.

Thanks.

[Vitaliy S.]

Hi Richard, in this case please open a support case with our team to identify the reason of these spikes. Thanks!

[jeffmetcalf]

See post below:

[jeffmetcalf]

I am also experiencing these. If I may interject another observation, they only occur with systems backed up via a Veeam B&R server that was upgraded from 6.5 to 7. I do not see these occurring on systems that are backed up via a B&R server where 6.5 was removed, and v7 put in cleanly. (VMWare 5.1)

[Vitaliy S.]

Jeff, do you use parallel VM and virtual disk processing?

[jeffmetcalf]

Yes, across three proxies. There is no pattern to a certain proxy being assigned when this occurs. All three are VM WinSrv 2008r2 standard configured exactly the same hardware wise.

[Vitaliy S.]

Thanks for the clarification. I believe this issue is somehow related to the load on the datastore produced by the snapshot creation/commit operations. Since VMs can be processed in parallel the load on the datastore is now higher, and I guess this might be one of the reasons for such behavior.

On top of that, our dev team keeps researching this issue and here is what we have recently found out about possible spikes of performance metrics:

If the sampled metric consists of a non-homogeneous set of data, metrics may be skewed in unexpected ways.

Here is a full KB article for further reading > http://kb.vmware.com/selfservice/micros ... Id=2001676

[jeffmetcalf]

In our case, it is not the same datastore. I see this on 2 of our larger VMs, 1 a SQL server, the other an Exchange server. Both are on separate Dell PowerVault MD32xx, in RAID 5 stripes across 5 drives in the Virtual disk array.

I do agree that is happens during the snapshot release, but it doesn't happen every night. My job settings are Reverse Incremental, active Full on the first Sat of the month, safe removal of snapshots over 100 MB is turned on, 4 restore points. VSS and guest indexing is on.

[ddayton]

We also have this issue and I had opened a case with Veeam support. However I was simply told this is VMware bug and should contact my VMware support and the Veeam case was closed.

So for now I am simply ignoring the alert and watching the VMware forum and patches for a fix.

[Vitaliy S.]

Hi Don,

These spikes are indeed generated by VMware, Veeam cannot generate performance metrics, since we only require read only account for monitoring.

Can you please navigate to Virtual Disks tab and check if you have any spikes there? Also do you run any backup job at the time when spikes are generated in your VI?

Thanks!

[jeffmetcalf]

Eh...I think it's a case of he said/she said. I didn't get these with B&R 6.5/Veeam One 6.5. And I don't get them on servers backed up from a clean install of B&R v7. Just the server upgraded from 6.5 to 7.
My ESXi hosts have not been patched since last fall and are all on the same version.

[Vitaliy S.]

Just the server upgraded from 6.5 to 7.

You're referring to upgrade of Veeam B&R, right? Unfortunately, we were unable to reproduce this behavior internally, so any information would help. Thank you!

[jeffmetcalf]

It happened again last night to one of my servers. I have submitted both the B&R and One logs to support, case # 00473947.

[stevenrodenburg1]

I have this issue too. I upgraded VeeamOne from 6.5 to 7 last night. This morning, found two unbelievable emails saying:

Object: Veeam-Proxy01
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 08.11.2013 03:03:55
Details: "Datastore Highest Latency" (43441964 ms) is above a defined threshold (75 ms)


Object: Veeam-Proxy02
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 08.11.2013 02:52:35
Details: "Datastore Highest Latency" (15116940 ms) is above a defined threshold (75 ms)


I mean, really? 43 MILLION Milliseconds latency?

The times correspond with the backup-window.
Looked in vCenter ofcource, the highest latency was 55ms for Veeam-Proxy01 and 26ms for Veeam-Proxy02.

Been running B&R v7 (Patch 1) with VeeamOne on v6.5. Never ever happened in VO 6.5 and come on, 43 and 15 million ms ????

[Vitaliy S.]

Guys,

Just a quick update - we have managed to reproduce this issue internally without using Veeam ONE and Veeam B&R, so this is definitely a VMware issue. Our QC team has been emulating backup job behavior by creating and committing snapshots and during one of the commit operations we have caught the Latency spike for VM Virtual Disk and Highest Latency counter for 26-28 millions ms.

We will open a support case with VMware and inform them about the observed behavior, meanwhile our dev team will think about possible workaround for this issue.

P.S. if you see the same behavior, feel free contacting VMware support team as well. I will post our support ticket number here, so you could use it as a reference when talking to VMware technical team.

Thanks!

[stevenrodenburg1]

Had another one. Ironically, this time the VM in question is the VeeamOne VM itself.

Object: Veeam-One
Object Type: Virtual Machine
Parent Object: Normal
Status: Error
Alarm: VM total disk latency
Time: 09.11.2013 23:23:30
Details: "Datastore Highest Latency" (14941280 ms) is above a defined threshold (75 ms)

14 millions milliseconds is indeed a bit higher that 75 so the alert is valid

And again, nothing to see in vCenter.

There were no backups running or anything. The database sits on a SQL server on another spindle-group.

I hope it get's fixed soon. These emails are starting to become annoying and they never occurred in VeeamOne 6.5 and the vSphere version did not change.

[Vitaliy S.]

Steven,

You didn't see this alarm in Veeam ONE v6.5, because we didn't monitor this metric - Datastore Highest Latency before. Can you please tell me what tabs in vSphere Client did you check?

As to possible reasons of this weird behavior, I believe snapshots is not the only factor that can cause it.

Thanks!

[stevenrodenburg1]

Hmmm, there was no backup going on. But this VM, the VeeamOne VM, did indeed have a snapshot (as a means of fallback in case the upgrade to v7 had a problem).
This has been the only time i got that wierd error outside the backup window. Then again, it was the only VM to have a "regular, user initiated" snapshot. All other snapshots are taken by Veeam B&R v7 and the times of these alerts more or less correspond with those. Of the many backup jobs that are running, only a very few trigger such an alert.

In vCenter, i look at the performance counters of the VM -> advanced -> datastore.

[Vitaliy S.]

Take a look at Virtual Disk tab, this should record all the spikes occurred. Once I have information from VMware support team on these spikes I will let everyone know.

[athome]

I see the same issue.

vSphere 5.1U1B with latest patches.
Veeam B&R 7 clean install.
Veeam ONE 7 upgraded from 6.5

A

[Vitaliy S.]

Anders, have you been able to open a support case with VMware on this, as I have advised in the previous posts?

[athome]

Anders, have you been able to open a support case with VMware on this, as I have advised in the previous posts?

I have, but I didnt see any casenumber from VMwarethat I could refer to.

[Vitaliy S.]

Here is a VMware case number you can use as a reference - 13402946911

[Vitaliy S.]

Just another update - VMware has acknowledged this behavior. I'm not sure they are going to fix it soon, but we have come up with a workaround that should address this issue in the upcoming Veeam ONE v7 R2 update.

Here is a Veeam KB article for more info > http://www.veeam.com/kb1836

[athome]

Thanks for update Vitaliy!

Do you have an eta of Veeam ONE v7 R2?

Is there a workaround for it until then? (There were someone earlier in the post that stated that issue only was seen when alert was enabled in Veeam ONE.)

[Vitaliy S.]

These earlier statements were not accurate and during WebEx sessions with our engineer we couldn't confirm it. Veeam ONE v7 R2 is going to be available very soon. As a workaround for now you may want to suppress this alarm during heavy datastore I/O operations.

[Mazzotta]

Here is a VMware case number you can use as a reference - 13402946911

Any news on VMware side about this?

[Vitaliy S.]

No news, they have created a KB I have posted above and that's all I know from our support team.

[hauckm]

I just now got off the phone with VMware support for a ticket we opened for this issue.

We opened the ticket a week or so ago and at that time gave VMware the Veeam ticket as a reference.

I was told today by VMware support the Veeam ticket has been closed and this is not a VMware issue. I was also told if I wanted to pursue the open ticket we would have to do so without reference to the now closed Veeam ticket. In other words we would have to pursue this as a latency issue with our storage.

Now this is the information I received from one guy, so it may or may not be VMware's stance on this issue. I am just passing along what I was told.

I will be disabling this alarm until R2 is available.

[Vitaliy S.]

Hauckm,

Thank you for the update. I think I can understand the point VMware engineer was heading to, they also observe these high values in the vSphere Client and other monitoring tools, but I'm not sure it is correct to blame a storage vendor for that.

Anyway, I have just talked to our QC team and the RC build of Veeam ONE v7 R2 looks stable, so the release is almost there.

Thank you!