Monitoring and reporting for Veeam Data Platform
adrianh
Novice
Posts: 8
Liked: 2 times
Joined: Nov 11, 2015 6:37 pm
Contact:

v9 - Pages per second errors

Post by adrianh »

Hello,

Upgraded to v9 a few weeks ago and i constantly get this error on all my hosts. The thing is I never used to on v8, and if i look at resource monitor on the server and perfmon my servers are all nearly zero, nevermind the predefined 500-1500 for 15 min threshold. Is there a bug in here? This is a hyper-v environment.

Alarm: Host Memory Pages Usage
Time: 3/9/2016 5:16:27 PM
Details: All alarm metrics are back to normal

KB article
Summary: The counter shows the rate at which pages are read from or written to disk to resolve hard page faults. This counter is a primary indicator of the types of faults that cause system-wide delays
Cause: To resolve hard page faults, the operating system swaps the contents of memory to disk, which negatively impacts performance. A high number of pages per second in correlation with low available physical memory may indicate a lack of physical memory
Resolution: Run Veeam ONE 'Oversized VMs' and 'Idle VMs' reports to optimize memory allocation. Remove superfluous software. Run maintenance tasks off peak times. If the alarm appears during backup operations, consider Off-host backup infrastructure
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v9 - Pages per second errors

Post by Vitaliy S. »

Hello,

Does it happen during some backup/maintenance activity or it occurs randomly?

Thank you!
adrianh
Novice
Posts: 8
Liked: 2 times
Joined: Nov 11, 2015 6:37 pm
Contact:

Re: v9 - Pages per second errors

Post by adrianh »

Yes its random
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v9 - Pages per second errors

Post by Vitaliy S. »

Can you please both screenshot to compare the numbers?
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

I am also getting this error. Veeam is running on my Hyper-V Host. The host has 24 GB of RAM and there is only one VM configured with 4 GB. So there should be plenty of RAM available. Alarm is triggered mainly around the backup job times (Endpoint, VM and Backup Copy Jobs). Should I get even more RAM or is Veeam always using RAM to the max?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov » 1 person likes this post

Proxy server tries to use all available memory resources during the backup activity. Best practice is to shift the load from the production Hyper-V offsite leveraging off-host backup.
Thanks!
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

So it looks like in a small company we have to turn the warnings off. Not the most elegant solution.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

You can suppress it for the time when backups jobs are running.
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

FYI: I added additional 16 GB of RAM (which is probably overkill) and now I don't get the warning anymore.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

Thanks for updating the thread, Anguel!
Have you added 16GB to host or VM?
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

To host.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

Glad it works but it still looks strange.
Having 1 VM with configured 4GB of RAM on 24 or 40GB should not cause any memory issues on the host.
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

The memory pages error has reappeared again and appears multiple times per day - I have now just turned off the alarm.
Regarding the fact that I am running the probably most basic configuration of Veeam I am constantly getting some strange errors. The "copy job exceeded backup window" error is still being investigated by Veeam since months...
From time to time I get "host unknown" errors for endpoint backups for no particular reason. From time to time I also get "Source restore point is locked by another job" errors - somehow Veeam does not properly schedule things.
We have to do business and can't constantly report errors and send error logs to Veeam, regarding the fact that we use the most basic configuration possible - Veeam on Hyper-V host with backup to local storage and backup copy to NAS.
I am leaving it now as is - giving up...
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov » 1 person likes this post

Anguel,
I`m really sorry to hear that. Thanks for sharing your feedback though. Could you provide your support case number so I can discuss the issues with the support engineer?
Anguel wrote:The "copy job exceeded backup window" error is still being investigated by Veeam since months...
Do you use backup windows? Does exceed really happen with your backup jobs?
Anguel wrote:From time to time I also get "Source restore point is locked by another job" errors - somehow Veeam does not properly schedule things.
That happens if 2 jobs of different priorities want to work with the same backup file.
If these situations happen you just need to tune your Veeam B&R instead of turning off Veeam ONE alarms. Let me know if assistance is needed here.
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

Just an update on exceeded backup window problem: This has now been confirmed as a bug, although it does not seem to have a negative impact on backup, and should be fixed in next releases. I still think that it must have been seen during proper testing. As I said, I have the most basic config ever and the bug always showed up.

Regarding the restore point locked problem: I still wonder why Veeam is not able to schedule jobs properly so they don't overlap.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov » 1 person likes this post

Anguel wrote:Just an update on exceeded backup window problem: This has now been confirmed as a bug, although it does not seem to have a negative impact on backup, and should be fixed in next releases. I still think that it must have been seen during proper testing. As I said, I have the most basic config ever and the bug always showed up.
Could you provide the case number? :)
Anguel wrote:Regarding the restore point locked problem: I still wonder why Veeam is not able to schedule jobs properly so they don't overlap.
Veeam doesn`t schedule jobs. You set their start time, backup windows etc. You can set the job start one after another.
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

Case for exceeded backup window problem: 01737723.
For backup copy jobs without manual scheduling Veeam does the scheduling and fails there - this is where I got the "locked" problems.
max0664
Influencer
Posts: 17
Liked: never
Joined: Sep 04, 2012 4:30 pm
Full Name: Max
Contact:

Re: v9 - Pages per second errors

Post by max0664 »

Hey Anguel,
have you some solution for the Pages per second errors?
I cannot make a Offhost proxy and have the same Problem.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

Hi Max,
Could you describe your issue?
Does the alarm trigger only during backup operations?
Thanks!
max0664
Influencer
Posts: 17
Liked: never
Joined: Sep 04, 2012 4:30 pm
Full Name: Max
Contact:

Re: v9 - Pages per second errors

Post by max0664 »

Hi Shestakov,

yes i only see the alarm when the node make the backup of the VMs.
So i use Hyper-V and On-Host proxy because the VSS Offhost proxy is expensiv with my Storage and is rly tricky.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

While we are waiting Anguel`s reply, you can run optimization reports to check if you have any Idle or Oversized VMs wasting memory resources.
max0664
Influencer
Posts: 17
Liked: never
Joined: Sep 04, 2012 4:30 pm
Full Name: Max
Contact:

Re: v9 - Pages per second errors

Post by max0664 »

I have one Idle VM and one Oversized but i dont think that i have to less Memory. Because on every Hyper-V Servers are ~150 GB Memory free.
Anguel
Expert
Posts: 193
Liked: 18 times
Joined: Apr 16, 2015 9:01 am
Location: Germany / Bulgaria
Contact:

Re: v9 - Pages per second errors

Post by Anguel »

Sorry for the late reply, but I don't have a solution, had to turn off the alarm...
cmartinmy
Lurker
Posts: 1
Liked: never
Joined: Aug 01, 2017 5:24 pm
Full Name: Chris Martin
Contact:

Re: v9 - Pages per second errors

Post by cmartinmy »

I have the same issue

Hyper-V Failover Cluster
2 Hosts, Server 2016 128 GB RAM (60G free on each)
Veeam B&R 9.5.0.1038

Host showed no performance degradation, Memory usage was not high, Disk I/O was not above normal. Pretty sure it's a false alarm, perhaps an incompatibility with 2016.

Finally disabled the alarm, had it set not to trigger during backup jobs, still didn't help.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

Hello Chris and welcome to the community!
Do you have a related support case number?
ivoneta
Novice
Posts: 8
Liked: 1 time
Joined: Feb 20, 2014 7:45 am
Full Name: Tom Ivone
Contact:

Re: v9 - Pages per second errors

Post by ivoneta »

Seeing the same on our 5 node Hyper-V 2016 Cluster, everything seems fine hundreds of GB of memory free on each node.

Each node reporting memory pages usage errors randomly within the backup window, environment seems stable no degradation or issues.

Local disk is SSD for OS while or VM's stored on SMB3 storage, have had to disabled the alarm due to constant emails.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

Hello Tom,
Are you also on v9?
Do you have a support case number? If yes, please post it here.
Thanks!
ivoneta
Novice
Posts: 8
Liked: 1 time
Joined: Feb 20, 2014 7:45 am
Full Name: Tom Ivone
Contact:

Re: v9 - Pages per second errors

Post by ivoneta »

Thanks for the follow up, I was going to open a ticket today but realized these hosts were on 2012R2. Our 2016 Cluster does not seem to have the same issue, we will be upgrading this Cluster in the next week so will wait to confirm issue is resolved before logging a ticket.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 - Pages per second errors

Post by Shestakov »

Got it. If you decide to open a support case, please post it here for us to follow the investigation.
Thanks!
dredd123
Novice
Posts: 7
Liked: never
Joined: Feb 24, 2016 10:10 am
Full Name: David Walsh
Contact:

Re: v9 - Pages per second errors

Post by dredd123 »

I've got the same problem. It happens at random across multiple clusters and not just when Veeam backups are running. Had to disable the alarm as it was becoming a PITA.
Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests