extreme compression - RPC function call failed

[dcit]

We have Veeam Backup version 11.

When using Extreme compression, I have some job failures:
RPC function call failed. Function name: [GetSvcVersion]. Target machine: [server-name:6160].

I am suspecting high CPU usage is the problem.

Problem started manisfesting after I have enable Dell SC storage integration, appropriate proxy&(internal RAID)repository server has FC connection to storage, so data can be read so fast, that CPU is 100% utilized (we have AMD EPYC 7282 16 core in that proxy server). Veeam Backup master server is running at another server.

When I have changed compression to High, CPU is 100% utilized too, but not fully, I see little fluctuation in taskmgr, it is not full 100% all the time (or I can say, that cpu graph line is not flat at 100%), and backup is able to finish without any errors.

Is it the issue some (small) timeout in Veeam Backup (default) settings? Or is it problem that compression task is having higher CPU priority than controlling task?

[dcit]

Support Case #04770711

[PetrM]

Hello,

The error is quite generic and just informs us about the failed RPC call but the real reason can be explained only after debug logs inspection. I'd let our support engineers to work on it. Your idea about timeout does make sense, you could ask our support engineers to check it as one of the possible versions.

You may also try to apply throttling, for instance to limit data transmission speed between Data Movers. My hypothesis is that the reduction of overall conveyor speed could probably save some CPU resources. Also, it would be interesting to know the dependency of CPU load on proxy concurrent tasks number.

Thanks!

[dcit]

It seems like I got closer to the issue.

I have lowered Max concurrent tasks from 16 to 4 on VMware backup proxy, on server which has FC connection to storage, and which has local storage for backups. I think that problem was in that many backup proxy tasks for read from storage snapshot were started at once. I have tried to set it to 1, 2, 4 and 8 - when 8 errors were there again, so I settled on 4.

It behaves much better now, but there are still some infrequent errors, I have sent logs to support.

I think that compression level is not the issue, or at least not the primary issue, as I have got errors even when compression set to Optimal, and CPU usage was rather low (cca 30%, and big part of it in kernel, so FC and/or local RAID card operations I think).

Next issue (with I am not happy) is that Max concurrent tasks 4 started to be to high after I have enabled secondary target type Storage Snapshot: Dell EMC SC Series Snapshot. Errors started happening regularly again, this time:

Code: Select all

27.04.2021 18:22:14 :: Processing VMname Error: Access is denied.
27.04.2021 18:22:26 :: Using backup proxy servername for retrieving Hard disk 1 data from storage snapshot on SCv3020-xxxx
27.04.2021 18:22:38 :: Hard disk 2 (30 GB) 4 MB read at 4 MB/s [CBT]
27.04.2021 18:22:48 :: Error: Access is denied.

Solution seems to be to lower "Max concurrent tasks" for that backup proxy to 2. After that my test job completed without errors.

Btw selecting storage snapshot as secondary target is behaving for me in that way that it creates one additional storage snapshot (after job ends this one is left without expiration) (so two storage snapshot are created), so I see some correlation here (may even causation).

[PetrM]

Hello,

Looks like it's on a right track, the initial errors are not so frequent as they were. Honestly, I don't think that the "Access denied" error is related to proxy load level somehow. As far as I understand, you've managed to find some pattern and it might be a good hint for our support but I'd consider it as a coincidence unless the direct correlation is confirmed by our support engineers. I believe that the best action plan now is to wait for the result of analysis from our support team.

Thanks!

[dcit]

It seems that culprit was VBS (virtual based security), I have had it enabled on VM with VBR server (& inside in Windows Server), after I have disabled it, it seems that those "Access is denied" errors stopped.

(it was before set this way:
In the VM, open gpedit.msc and browse to:
Computer Configuration > Administrative Templates > System > Device Guard > Turn On Virtualization Based Security

Set it to Enable and configure the options as follows:
Select Platform Security Level: Secure Boot
Virtualization Based Protection of Code Integrity: Enabled with UEFI lock
not enabled: Require UEFI Memory Attributes Table
Credential Guard Configuration: Enabled with UEFI lock
Secure Launch Configuration: Enabled)

Btw before that I have created exceptions in Windows Defender, even stopped realtime protection - on VBR server and that problematic backup proxy - and it helped a bit, but those errors still occasionally happened.
After disabling VBS it works without those errors even when I have set Max concurrent tasks to 16.

Btw latest relevant support issue number is 04790849

[dcit]

better name of this forum thread would be now:
Virtualization-based Security (VBS) conflicts with Veeam Backup

[PetrM]

Hello,

Thanks a lot for sharing the detailed information about the root cause with the Community, I'm glad to hear that it is sorted out!

Thanks!