Wrong registry key kills B&R Application aware processing (#08074972)

[ibschreiber]

Hi everyone,
After upgrading to V13, we ran into issues with Application aware processing through VBR.

No Hyper-V production Snapshots could be generated on multiple machines spread over multiple customers (we're VCSP).

After a long search and trying everything we knew of this issue with previous versions, the solution came in form of a redid post from mazvazzeg (thank you!)

The issue were not missing open firewall ports, user rights, failed vss providers or anything like this.

Veeam changed the service entry for the Hyper V Volumeshadow requestor which prevented it from running at all:

The failed entry for the registry data under

Code: Select all

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vmicvss

contained in the ImagePath key

Code: Select all

 %systemroot%\system32\svchost.exe -k VeeamHvVssGroupstemNetworkRestrictedRedirectionGuard -p

which prevented the service from starting at all.
after changing it back to the value

Code: Select all

%SystemRoot%\System32\svchost.exe -k VeeamHvVssGroup -p

the service is starting again and everything is back to normal.

I'm waiting for Veeam to clarify, if this is a viable solution (at least it's a working one for me) and if there is a different way to fix this.

I'll keep you posted
Michael

[ibschreiber]

This is the original post: https://www.reddit.com/r/Veeam/comments ... ing/?tl=de

[david.domask]

Hi Michael,

Thank you for sharing the solution you located, and sorry to hear about the troubles.

Please do continue with Support, I'm not able to quickly find any details on VeeamHvVssGroupstemNetworkRestrictedRedirectionGuard or how it would have been set like that, so while it's good backups are running, as you noted let's see the results of the investigation.

I've notated the case for Support with some additional details, so let's see how the investigation goes.

[Samuraj]

Hi,
I also encountered this problem on a newly installed SQL server on Hyper-V. Older SQL servers are backed up fine.

Code: Select all

Warning : Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) ... Production checkpoints cannot be created for ...

Unfortunately, I couldn't find this post, so I spent 2 days trying to figure it out. I found that the Hyper-V Volume Shadow Copy Requestor service (vmicvss) is not running and cannot be started.

Finally I found the key mentioned here in the registry, which had a different value than on the other servers. After editing the service started and everything works.

So I was now looking for what the VeeamHvVssGroupstemNetworkRestrictedRedirectionGuard value means and came across this post.

[cdubus]

Hi, Given the format of the value

Code: Select all

"VeeamHvVssGroupstemNetworkRestrictedRedirectionGuard"

I assume that Veeam tried to replace the value

Code: Select all

 "VeeamHvVssGroup"

with

Code: Select all

"VeeamHvVssGroup\ItemNetworkRestricted\RedirectionGuard"

The "s" in "groups" looks like a typo, as the "i" in "item" is missing.

[david.domask]

Thanks for the idea cdubus, may indeed be related.

Currently the issue is being reviewed in one case by our Advanced Support Team & RND; hopefully there should be an update soon on the behavior confirming the issue and validating the workaround.

[jasa004]

This issue still exists - also on Community Edition. It took me a full week to find the culprit as to why our Hyper-V production checkpoint creation fails on just 1 healthy VM as well as job fails with little logging. Today i saw the difference in vmicvss service path to executable and then this thread.
How is there still not an update?

[BrianBuchanan]

Thank-you. I just troubleshot my third freshly installed Server 2025 vm with this issue but unlike the other two I didn't give up and just wipe and reinstall.

Credit to Gemini that gave me this:

Code: Select all

Get-WmiObject win32_service | ?{$_.Name -like "*vmicvss*"} | Select PathName

which gave me this output:

Code: Select all

PathName
--------
C:\WINDOWS\system32\svchost.exe -k VeeamHvVssGroupstemNetworkRestrictedRedirectionGuard -p

Which Gemini deemed to be "junk".

I needed to do two things to get checkpoints working again. First was to fix that value with:

Code: Select all

sc.exe config vmicvss binPath= "C:\Windows\system32\svchost.exe -k LocalSystemNetworkRestricted -p"

and second was to add vmicvss to the registry value:
"HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost\LocalSystemNetworkRestricted"

Code: Select all

$registryPath = "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost"
$valueName = "LocalSystemNetworkRestricted"

# Check if the group exists and add vmicvss if it's missing
if (Test-Path $registryPath) {
    $currentValues = Get-ItemProperty -Path $registryPath -Name $valueName -ErrorAction SilentlyContinue
    if ($currentValues) {
        if ($currentValues.$valueName -notcontains "vmicvss") {
            $newValues = $currentValues.$valueName + "vmicvss"
            Set-ItemProperty -Path $registryPath -Name $valueName -Value $newValues
            Write-Host "Added vmicvss to Svchost group." -ForegroundColor Green
        }
    } else {
        # Create the value if Veeam deleted it
        New-ItemProperty -Path $registryPath -Name $valueName -Value @("vmicvss") -PropertyType MultiString
    }
}

After that the service would start and I could make production checkpoints.

Some more context, the host output this with "No Contact" for VSS.

Code: Select all

Get-VMIntegrationService -VMName "TESTVM"

VMName  Name                    Enabled PrimaryStatusDescription SecondaryStatusDescription
------  ----                    ------- ------------------------ --------------------------
TESTVM Guest Service Interface False   OK
TESTVM Heartbeat               True    OK
TESTVM Key-Value Pair Exchange True    OK
TESTVM Shutdown                True    OK
TESTVM Time Synchronization    True    OK
TESTVM VSS                     True    No Contact

And trying to start vmicvss on the guest:

Code: Select all

Start-Service -Name vmicvss -Verbose
VERBOSE: Performing the operation "Start-Service" on target "Hyper-V Volume Shadow Copy Requestor (vmicvss)".
Start-Service : Service 'Hyper-V Volume Shadow Copy Requestor (vmicvss)' cannot be started due to the following error:
Cannot start service vmicvss on computer '.'.
At line:1 char:1
+ Start-Service -Name vmicvss -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
   ServiceCommandException
    + FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand

I hope this helps someone else in the future!