VeeamOne & Disk Latency Alarms

[nmace]

I've been getting a LOT of alarm emails from Veeam regarding Host Datastore latency and VM disk latency, most reads but sometimes writes. We have it monitoring a Nutanix vSphere 5.5 cluster. Nutanix takes a somewhat non-traditional approach to the shared filesystem. It's presented as NFS shares, but under the hood it's some got dedup/compression/replication/SSD tiering going on. Also, the Nutanix console doesn't report any IO alarms or anything that resebles what VeeamOne is reporting. Nor do the VMs themselves seem to experiencing any sluggish-ness.

I've tried to increasing both the number of milliseconds that has to be hit before VeeamOne alarms as well as the amount of time. Changing this helped, but I still got 300+ emails over the weekend. Is there anything else I can do besides just disable those 2 alarms?

[Vitaliy S.]

Hello,

What are you current alarm thresholds and latency performance graphs on the hosts? Are these alarms triggered at a particular time or it can happens just randomly?

Thank you!

[nmace]

I have Host Disk Read Latency (currently disabled) set to warn @ 100 ms, error @ 250 ms and time period 30 mins.

I have Virtual Machine VM Total Disk Latency (disabled) set to warn 60 ms and error for 85 ms.

Not all of the triggered alerts (but most) happen at night during the backups (using Veeam B&R), so yes these alerts are happening during a period of increased IO....but by the same token I can't believe some of the readings I'm getting. A datastore with 100+ ms latency? Seems like if that were really happening....lots of stuff would be broken.

[Vitaliy S.]

Well...we are getting this information through VMware API and it must be something with how Nutanix reports on its performance. Anyway, if this happens during backup windows I would recommend setting a suppress period for the entire window. Should address the massive email notifications issue.

[nmace]

Nice! I didn't even notice that suppress tab, AND there is the option to suppress it during a Veeam Backup. VeeamOne being aware of Veeam B&R backups for the win!

[Vitaliy S.]

Of course we are aware about Veeam B&R, moreover you can monitor/report on your Veeam backups with Veeam ONE.

[cstemaly]

My latency occurs on a single datastore during SureBackup verification jobs. Does the suppress for "Veeam Backup Activity" also apply to SureBackup jobs?

[Vitaliy S.]

No it does not, it only tracks backup job activity. BTW, do you schedule your SureBackup jobs? It might be worth trying to suppress this alarm during SB job via advanced alarm settings.

[cstemaly]

Yes they're scheduled - I guess I'll go that route, thanks for the reply!

[danas]

I am seeing this issue with replication, which is ongoing. I would like to be able to monitor the latency, but it is reporting in such a away that changing thresholds wont help. Our SAN is a Dell MD3220i and we haven't had any issues with it. Somehow i don't think the latency below is accurate, is this a known issue that can be addressed? Thanks!

"Datastore Highest Latency" (3552916809269 ms) is above a defined threshold (75 ms)

[Vitaliy S.]

Hello Dana,

Yes, you're correct that these values are inaccurately reported by VMware. Please check out this topic for the solution > V7 - VMware Issue - False alerts

Thank you!

[danas]

Ah, thanks. I JUST downloaded in installed VeeamONE about 6 days ago, didn't realize i was missing an update! I love it by the way, fantastic access to stats. I do have a question but i'll start a new thread. Thanks!!

Dana