No-op power profile changes during windows agent backup?

[hpadm]

Whenever a windows agent-based backup job starts or finishes on our physical server hosts, the system event log records the following:

Event 12, UserModePowerService:
Process C:\Program Files\Veeam\Endpoint Backup\Veeam.EndPoint.Manager.exe (process ID:6472) reset policy scheme from {381b4222-f694-41f0-9685-ff5bb260df2e} to {381b4222-f694-41f0-9685-ff5bb260df2e}

This is the UUID for the 'Balanced' power scheme. I used 'powercfg /query' before and after and saw no changes. So it seems as if it's doing some kind of system-wide change to a power setting that's already set. Whatever it is, I suggest using some kind of temporary API if possible, instead of altering the system settings.

I checked the documentation, and for workstations the only related note I found was about enabling 'wake up timers' on workstations. The documentation for servers does not include this note. And our servers all have it enabled, which is the default setting.

(Originally submitted as free support case #05458116 but it timed out.)

[Dima P.]

Hello hpadm,

Server edition should not interfere with the power plans (indeed this is a functionality for free/workstation edition). I've asked support team to reopen the case and start the investigation. Thank you for your post!

[hpadm]

It seemed unusual, because even if it was related to wake timers, that's something you configure one time during agent installation, not just before and just after a backup job. The phrasing of the event message suggests that it actually tried switching power profiles, or something. The most logical reasoning is that it's trying to prevent the host from going to sleep in the middle of a backup job (but there are more appropriate APIs for that). Another weird thing is that it's resetting the power profile ... to itself, doing nothing.

Hm.

Code: Select all

Endpoint\...\Job.Backup.log
[06.06.2022 00:00:03] <01> Info     [SleepLocker] Enable sleep prevention
[06.06.2022 00:00:03] <01> Info     [SleepLocker] Sleep prevention power scheme params AC: True 4294967295, DC: True 300
...
[06.06.2022 00:38:38] <01> Info     [SleepLocker] Disable sleep prevention

There is a set of WinAPI calls - PowerCreateRequest, PowerSetRequest, PowerClearRequest - that are able to prevent sleep, however the documentation mentions some caveats, like a time limit when on battery power, or the ability of the user to force sleep mode. Maybe there is a more fitting API intended for backup jobs.

[Dima P.]

Thank you. Support team will reach you to collect all the logs from Veeam Agent side of things. Stay tuned!

[hpadm]

I've revisited this subject today since the described behavior is still present in 13.0. Based on the past conversation in #05470443 I have learned that the intent is to prevent sleep mode during a backup job, and that on top of conventional methods, Microsoft's modern standby initiative introduced 'a timeout after which the OS is allowed to no longer honor active power requests'.

Code: Select all

HKLM\SYSTEM\CurrentControlSet\Control\Power\User\PowerSchemes
 > 381b4222-f694-41f0-9685-ff5bb260df2e (Balanced)
   > 2e601130-5351-4d9d-8e04-252966bad054 (Idle Resiliency)
     > 3166bc41-7e98-4e03-b34e-ec0f5f2b218e (Execution Required power request timeout)
       > ACSettingIndex = -1 (infinite)
       > DCSettingIndex = 300 (5min)

VBR's SleepLocker calls PowerWriteACValueIndex/PowerWriteDCValueIndex to change both to -1 and then PowerSetActiveScheme to apply the change. It remembers the original values and wries them back once done. There are multiple problems with this:

• Changing global system state to solve an individual application problem is a very old antipattern.
• It fails the 'what if two programs did this' test, the outcome being -1 being permanently written to the OS.
• It does this even if the admin custom edits the values to already be -1. It doesn't bother checking and just does it.
• It does this on hosts using ordinary C3 instead of S0ix. It's unclear if these hosts even need this extra step.
• It does this on servers which do not have any sleep functionality.

I wonder how Microsoft expects backup software to operate under modern standby. Maybe it would be worth it to ask them. I wonder if VSS snapshots persist across sleep/hibernate and if Microsoft expects backup jobs to support pause/resume with disconnects.

All in all, the only visible artifact of the current logic are two power events for each backup session. I might have been overthinking it. Not attempting to detect what sort of system it's on and just doing it unconditionally is sloppy but also the least complex. It made me think back on Task Scheduler, which lets the admin decide - "start task only if on AC power (y/n)", "stop if switching to battery power (y/n)". It's tempting to just add more checkboxes, but in reality, there is only one good default and noone would probably want to change it.