Veeam One Hyper-V alert confusion

[ekhouser614]

I have been using Veeam One for some time now in testing and have only a week ago went full production and seem be either confused on a couple report/errors or they are mis-reporting data. The first one is the "Host Average Memory Pressure" . alert. I have two Hyper-V hosts, each with 192 GB of RAM and each host only lightly used. If I select one of the hosts and select the Memory tab, the Data is as follows:
Units Latest Min Avg Max
Hyper-V Services Memory Consumed GB 49.56 49.39 49.73 51.77
Hyper-V Services Memory Usage % 25.81 25.72 25.90 26.95
Memory Consumed GB 44.78 44.62 44.96 47.00
Memory Usage % 23.32 23.23 23.41 24.96

These metrics are, to me, very small. I have a couple VMs which periodically run into memory pressure issues, but not the hosts themselves and this alert is for the host itself. The second host is very close to this one as far as usage goes.

The second alert that confuses me is, for the VM itself, some will receive the "Hyper-V snapshot size" alert. We do not use snapshots as a practice so I am not understanding where this alert comes from. I will though, it only appears for VMs that have gone through a P2V migration. All other VMs do not reflect snapshot alerts

[Vitaliy S.]

Hi Eric,

1. Do you use dynamic or static memory for VMs located on the host in question?
2. What VMs do you see these alarms for? Are you running backup jobs by any chance against these VMs?

Thanks!

[ekhouser614]

For the VMs in question, dynamic and this happens whether or not a backup is running. The confusing part is, there is an alert for VM memory pressure, which I understand since the VM is reaching it's threshold but this alert says the host itself is always at approximately 90%--which it's no where close.

[Vitaliy S.]

Memory pressure is calculated based on the potential memory demand from VMs configure with dynamic memory.

As for the second issue (with snapshots) presence, can you confirm that you don't have any avhdx files present on the volume?

[Shestakov]

Eric,
I`m also curious what thresholds are set for the alarmed hosts? Have you changed the default settings? What values(pikes) are reported by the triggered alarms?
Thanks!

[ekhouser614]

The alerts are default, but in looking at the Alarm settings for Hyper-V>Host Average Memory Pressure, it is a warning at 80% and Error at 90%. The aggregate of all VMs on the individual host would not total 80 or 90% even at max on the dynamic memory. Now, I will say, this is a fail-over cluster, and with all VMs on one host, this may be, but not with the separation of VMs on different hosts. Would it be looking at all VMs on one host to calculate this? Also, there are no avhdx or avhd files on the hosts at all.

[Vitaliy S.]

Also, there are no avhdx or avhd files on the hosts at all.

In this case open a support case, cause snapshot alarm only works if it detects VM snapshots (aka checkpoints).

[ekhouser614]

I will, but any idea about the memory alert?

[Shestakov]

Memory Pressure is a calculation of how much memory the host wants divided by the amount of memory it currently has. I would check the next metrics same time when the Memory Pressure alarm was triggered:
Memory Available which indicates the amount of physical memory (in MB) immediately available for allocation.
Memory Pages showing if and how often system uses hard drive to store and retrieve memory associated data.

When applications host`s VMs want more memory than it currently has, the host swaps memory pages to hard drive. You can observe Memory pressure around 100% and low Memory available.
If these counters had the mentioned behavior, you had a lack of memory for a short period of time, if the counters were stable for that time, probably Hyper-V or Veeam ONE provided wrong numbers, you can compare Veeam ONE results and perfmon to find out the issue.
Thanks!

[ekhouser614]

And this is where this alert is confusing. I understand that when a child VM demands more memory than it has, or, it has a memory pressure above a threshold would send an alert but, this is "host" memory--not vm memory. If every vm went to 100% memory on this host, it can only have a maximum of 62 GB which is 30% of the total installed on the server. In looking at the performance of the host itself, it's barely touched since it has 192 GB of installed RAM so this is where the confusion is.

[ekhouser614]

And to further this, the explanation is also a little confusing since it states, the alert could in fact mean an "over-commitment" from all VMs, which on this host is impossible.

[Shestakov]

I see your point and you are right.
If so, I would compare Hyper-V and Veeam ONE results to check if VeeamONE collects correct data. Since you are up to contact technical support with another issue, I would ask them for an assistance with this one as well.

[ekhouser614]

What I notice is, every hour on the hour, the pages/sec go through the roof at over 46,000, but memory doesn't budge, and CPU stays below 10%. I do notice as well, using processexplorer, the veeamagent.exe is the process attributed to the high I/O I see on both disk and network. I have a backup that runs each hour on a VM with Exchange installed. It corresponds to these timeframes as well. Is this to be expected? We use off-host proxies, but the Veeam server is a VM. The disks these are on are 15k SAS drives using CSVs on a server.

[Shestakov]

Where do you have the high Memory pages/sec values(proxy VM/host/other VM)?

We use off-host proxies, but the Veeam server is a VM.

So you have another HyperV with a backup proxy VM which processes VMs from the primary host? That is not the best practice.

[ekhouser614]

The proxy is a physical server, not a VM on another non-clustered host. The high memory pages are on the hosts that are clustered. What I can't find though is a good explanation of how we have the proxy setup. I read one place the proxy must have "direct access" to the shared storage such as iscsi to a SAN and other places it just needs "access to the host and VMs". The proxy is just another physical server with Hyper-V role installed and the clustered hosts are on the same LAN. Is this also not best practice? If so, I will need to just use on-host backups and not use off-host proxies.

[Shestakov]

What I can't find though is a good explanation of how we have the proxy setup. I read one place the proxy must have "direct access" to the shared storage such as iscsi to a SAN and other places it just needs "access to the host and VMs".

What kind of storage(local/CSV/SMB3) do you have? For CSV and local storage the source Hyper-V host and the off-host backup proxy must be connected (through a SAN configuration) to the shared storage.

The proxy is just another physical server with Hyper-V role installed and the clustered hosts are on the same LAN. Is this also not best practice? If so, I will need to just use on-host backups and not use off-host proxies.

It`s the recommended approach, on-host proxy will slow down VMs and backup processing.

[ekhouser614]

The virtual infrastructure is on a Dell VRTX server which is a blade server and SAS drives shared by the blades. There is no way to physically connect another host unless I install another blade itself and use it that way since we are a small shop and do not have a SAN. That said, it appears I have no option other than on-host backups correct?

[Shestakov]

You can check what backup method is used by looking into backup job results for particular VM. It`s told what proxy and backup transport mode were used.
By default if VBR can not use off-host transport mode it fails over the on-host.

[ekhouser614]

This I have seen, but the question is, with the below described infrastructure, what would be required? Does the proxy have to have a physical or logical (iscsi) connection to the CSV of the host cluster? Is there no way to create a proxy that is a stand-alone host without this connection?

[Shestakov]

Yes, one of the requirements is a connection between source Hyper-V host, off-host backup proxy and the shared storage.
Please check the link for more information.