v9 - Pages per second errors

[adrianh]

Hello,

Upgraded to v9 a few weeks ago and i constantly get this error on all my hosts. The thing is I never used to on v8, and if i look at resource monitor on the server and perfmon my servers are all nearly zero, nevermind the predefined 500-1500 for 15 min threshold. Is there a bug in here? This is a hyper-v environment.

Alarm: Host Memory Pages Usage
Time: 3/9/2016 5:16:27 PM
Details: All alarm metrics are back to normal

KB article
Summary: The counter shows the rate at which pages are read from or written to disk to resolve hard page faults. This counter is a primary indicator of the types of faults that cause system-wide delays
Cause: To resolve hard page faults, the operating system swaps the contents of memory to disk, which negatively impacts performance. A high number of pages per second in correlation with low available physical memory may indicate a lack of physical memory
Resolution: Run Veeam ONE 'Oversized VMs' and 'Idle VMs' reports to optimize memory allocation. Remove superfluous software. Run maintenance tasks off peak times. If the alarm appears during backup operations, consider Off-host backup infrastructure

[Vitaliy S.]

Hello,

Does it happen during some backup/maintenance activity or it occurs randomly?

Thank you!

[adrianh]

Yes its random

[Vitaliy S.]

Can you please both screenshot to compare the numbers?

[Anguel]

I am also getting this error. Veeam is running on my Hyper-V Host. The host has 24 GB of RAM and there is only one VM configured with 4 GB. So there should be plenty of RAM available. Alarm is triggered mainly around the backup job times (Endpoint, VM and Backup Copy Jobs). Should I get even more RAM or is Veeam always using RAM to the max?

[Shestakov]

Proxy server tries to use all available memory resources during the backup activity. Best practice is to shift the load from the production Hyper-V offsite leveraging off-host backup.
Thanks!

[Anguel]

So it looks like in a small company we have to turn the warnings off. Not the most elegant solution.

[Shestakov]

You can suppress it for the time when backups jobs are running.

[Anguel]

FYI: I added additional 16 GB of RAM (which is probably overkill) and now I don't get the warning anymore.

[Shestakov]

Thanks for updating the thread, Anguel!
Have you added 16GB to host or VM?

[Anguel]

To host.

[Shestakov]

Glad it works but it still looks strange.
Having 1 VM with configured 4GB of RAM on 24 or 40GB should not cause any memory issues on the host.

[Anguel]

The memory pages error has reappeared again and appears multiple times per day - I have now just turned off the alarm.
Regarding the fact that I am running the probably most basic configuration of Veeam I am constantly getting some strange errors. The "copy job exceeded backup window" error is still being investigated by Veeam since months...
From time to time I get "host unknown" errors for endpoint backups for no particular reason. From time to time I also get "Source restore point is locked by another job" errors - somehow Veeam does not properly schedule things.
We have to do business and can't constantly report errors and send error logs to Veeam, regarding the fact that we use the most basic configuration possible - Veeam on Hyper-V host with backup to local storage and backup copy to NAS.
I am leaving it now as is - giving up...

[Shestakov]

Anguel,
I`m really sorry to hear that. Thanks for sharing your feedback though. Could you provide your support case number so I can discuss the issues with the support engineer?

The "copy job exceeded backup window" error is still being investigated by Veeam since months...

Do you use backup windows? Does exceed really happen with your backup jobs?

From time to time I also get "Source restore point is locked by another job" errors - somehow Veeam does not properly schedule things.

That happens if 2 jobs of different priorities want to work with the same backup file.
If these situations happen you just need to tune your Veeam B&R instead of turning off Veeam ONE alarms. Let me know if assistance is needed here.

[Anguel]

Just an update on exceeded backup window problem: This has now been confirmed as a bug, although it does not seem to have a negative impact on backup, and should be fixed in next releases. I still think that it must have been seen during proper testing. As I said, I have the most basic config ever and the bug always showed up.

Regarding the restore point locked problem: I still wonder why Veeam is not able to schedule jobs properly so they don't overlap.

[Shestakov]

Just an update on exceeded backup window problem: This has now been confirmed as a bug, although it does not seem to have a negative impact on backup, and should be fixed in next releases. I still think that it must have been seen during proper testing. As I said, I have the most basic config ever and the bug always showed up.

Could you provide the case number?

Regarding the restore point locked problem: I still wonder why Veeam is not able to schedule jobs properly so they don't overlap.

Veeam doesn`t schedule jobs. You set their start time, backup windows etc. You can set the job start one after another.

[Anguel]

Case for exceeded backup window problem: 01737723.
For backup copy jobs without manual scheduling Veeam does the scheduling and fails there - this is where I got the "locked" problems.

[max0664]

Hey Anguel,
have you some solution for the Pages per second errors?
I cannot make a Offhost proxy and have the same Problem.

[Shestakov]

Hi Max,
Could you describe your issue?
Does the alarm trigger only during backup operations?
Thanks!

[max0664]

Hi Shestakov,

yes i only see the alarm when the node make the backup of the VMs.
So i use Hyper-V and On-Host proxy because the VSS Offhost proxy is expensiv with my Storage and is rly tricky.

[Shestakov]

While we are waiting Anguel`s reply, you can run optimization reports to check if you have any Idle or Oversized VMs wasting memory resources.

[max0664]

I have one Idle VM and one Oversized but i dont think that i have to less Memory. Because on every Hyper-V Servers are ~150 GB Memory free.

[Anguel]

Sorry for the late reply, but I don't have a solution, had to turn off the alarm...

[cmartinmy]

I have the same issue

Hyper-V Failover Cluster
2 Hosts, Server 2016 128 GB RAM (60G free on each)
Veeam B&R 9.5.0.1038

Host showed no performance degradation, Memory usage was not high, Disk I/O was not above normal. Pretty sure it's a false alarm, perhaps an incompatibility with 2016.

Finally disabled the alarm, had it set not to trigger during backup jobs, still didn't help.

[Shestakov]

Hello Chris and welcome to the community!
Do you have a related support case number?

[ivoneta]

Seeing the same on our 5 node Hyper-V 2016 Cluster, everything seems fine hundreds of GB of memory free on each node.

Each node reporting memory pages usage errors randomly within the backup window, environment seems stable no degradation or issues.

Local disk is SSD for OS while or VM's stored on SMB3 storage, have had to disabled the alarm due to constant emails.

[Shestakov]

Hello Tom,
Are you also on v9?
Do you have a support case number? If yes, please post it here.
Thanks!

[ivoneta]

Thanks for the follow up, I was going to open a ticket today but realized these hosts were on 2012R2. Our 2016 Cluster does not seem to have the same issue, we will be upgrading this Cluster in the next week so will wait to confirm issue is resolved before logging a ticket.

[Shestakov]

Got it. If you decide to open a support case, please post it here for us to follow the investigation.
Thanks!

[dredd123]

I've got the same problem. It happens at random across multiple clusters and not just when Veeam backups are running. Had to disable the alarm as it was becoming a PITA.