Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

[doktornotor]

Somehow, when playing with 9.5U4 and repo configuration on a test box, I managed to break the whole backup job by setting"Limit maximum concurrent tasks" for the backup repository to a number lower than the number of VMs that are being processed in a backup job that is using that repository, while that job had "Allow processing of multiple VMs with a single volume snapshot" enabled.

This reliably causes the backup job to fail. Say, you have concurrent tasks set to 2 and there are 3 VMs included in the job. The backup for the first two VMs fails like this:

Code: Select all

Failed to finalize guest processing. Error: Failed to call RPC function 'Vss.WaitForBackupDocsToBeFlushed': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc].

After two first 2 VMs failed, the backup continues with the third machine, and there the job just hangs indefinitely, doing nothing, job cannot be stopped either, the Veeam Backup Service fails to stop as well and needs to be killed via task manager. The checkpoint for the third VM if left behind and needs to be merged manually via PS - "Get-VMSnapshot <VM> | Remove-VMSnapshot"

The issue immediately goes away when "Allow processing of multiple VMs with a single volume snapshot" in unchecked from the job, the VMs get processed in batches corresponding to the concurrent tasks limit and everything finishes just fine.

Somehow I'd say users should be prevented from creating invalid configuration that completely breaks the backup. Someone else able to reproduce this?

[DGrinev]

Hi,

I've discussed your issue with the QA team, the described scenario shouldn't result as you've seen.
It's impossible to understand the root cause without deeper investigation.

Please contact the support team and provide the logs, then don't forget to share the case ID with us here. Thanks!

[doktornotor]

the described scenario shouldn't result as you've seen.

Yes, agreed that this is definitely not the desired outcome.

Please contact the support team and provide the logs, then don't forget to share the case ID with us here. Thanks!

Ok, will get to it after the weekend, hopefully...

Thanks!

[georgeinva2004]

I am having the same issue. Did you ever get this resolved?

[veremin]

The OP has never come back to update the topic with case ID, so we cannot check the status of the ticket and whether the resolution was provided.

So, kindly, open your own case and post its number here.

Thanks!

[georgeinva2004]

Created a support case. Ticket # 03449308

[Jeroen]

I have the same problem. However, disabling the "Allow processing of multiple VMs with a single volume snapshot" didn't change anything.
Created a support ticket: Case # 03455876

[georgeinva2004]

I found a fix for my issue.

While pulling the data requested by support, I noticed a number of CAPI32 errors (Event 513) while backups were in progress. I found this article: https://support.microsoft.com/en-us/hel ... ows-server, performed the steps and the issue stopped.

[georgeinva2004]

The problem came back today for me. Has anyone else found a resolution?

[foggy]

Please continue working with your support engineer and refer to another case mentioned above (# 03455876) to check if the fix discussed there will help in your particular case.

[melexis]

Same issue here, since update from 9.5U3a to 9.5U4 we have random timeouts:
"Failed to finalize guest processing. Error: Failed to call RPC function 'Vss.WaitForBackupDocsToBeFlushed': Der Remoteprozeduraufruf (RPC) wurde abgebrochen. RPC function call failed. Function name: [DoRpc]. Target machine"

Working with support on this topic since 8th of February, without any progress. Case # 03405318

Also tried the suggestions in this thread but without success.

[melexis]

Made some progress with support:
In my case, they confirmed it is a bug (came with Update 4) and a hotfix will be released soon.
Temporary solution is to kick VeeamPSDirectCtrl_X64 process on the affected Hyper-V Host, after that you're able to stop backup job (which was not possible before) and also backup retry works for me.

[BrannelSchool]

Have you had access to the hotfix yet? We are experiencing the exact same issue, was hoping it may have been resolved in U4a but we are still struggling.

[foggy]

Yes, hotfix has been provided but the outcome is still not clear (at least from the case notes). Please contact technical support directly. If the hotfix applies to your issue, you will get access to it.

[BrannelSchool]

Ahh thanks, I thought it would be a public hotfix. I have just opened a case #03506903

[rymsza]

We have exactly the same problem with update4a.
Temporary solution is to kick VeeamPSDirectCtrl_X64 process on the affected Hyper-V Host, after that you're able to stop backup job (which was not possible before) and also backup retry works for me.

[Gostev]

The issue is caused by PSDirect calls hanging under certain circumstances, and as far as I know it should be completely resolved in Update 4b.

[doktornotor]

Thanks, will test when 4b is out.

[Stegerius]

Update 4b is out.
Has anybody tried if the problem is resolved with it?

Thanks.

[georgeinva2004]

It did fix my issue