Comprehensive data protection for all workloads
Post Reply
doktornotor
Enthusiast
Posts: 94
Liked: 29 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by doktornotor » 1 person likes this post

Somehow, when playing with 9.5U4 and repo configuration on a test box, I managed to break the whole backup job by setting"Limit maximum concurrent tasks" for the backup repository to a number lower than the number of VMs that are being processed in a backup job that is using that repository, while that job had "Allow processing of multiple VMs with a single volume snapshot" enabled.

This reliably causes the backup job to fail. Say, you have concurrent tasks set to 2 and there are 3 VMs included in the job. The backup for the first two VMs fails like this:

Code: Select all

Failed to finalize guest processing. Error: Failed to call RPC function 'Vss.WaitForBackupDocsToBeFlushed': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc].
After two first 2 VMs failed, the backup continues with the third machine, and there the job just hangs indefinitely, doing nothing, job cannot be stopped either, the Veeam Backup Service fails to stop as well and needs to be killed via task manager. The checkpoint for the third VM if left behind and needs to be merged manually via PS - "Get-VMSnapshot <VM> | Remove-VMSnapshot"

The issue immediately goes away when "Allow processing of multiple VMs with a single volume snapshot" in unchecked from the job, the VMs get processed in batches corresponding to the concurrent tasks limit and everything finishes just fine.

Somehow I'd say users should be prevented from creating invalid configuration that completely breaks the backup. :idea: Someone else able to reproduce this? :?:
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by DGrinev » 1 person likes this post

Hi,

I've discussed your issue with the QA team, the described scenario shouldn't result as you've seen.
It's impossible to understand the root cause without deeper investigation.

Please contact the support team and provide the logs, then don't forget to share the case ID with us here. Thanks!
doktornotor
Enthusiast
Posts: 94
Liked: 29 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by doktornotor »

the described scenario shouldn't result as you've seen.
Yes, agreed that this is definitely not the desired outcome. :)
Please contact the support team and provide the logs, then don't forget to share the case ID with us here. Thanks!
Ok, will get to it after the weekend, hopefully...

Thanks!
georgeinva2004
Novice
Posts: 6
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 »

I am having the same issue. Did you ever get this resolved?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by veremin »

The OP has never come back to update the topic with case ID, so we cannot check the status of the ticket and whether the resolution was provided.

So, kindly, open your own case and post its number here.

Thanks!
georgeinva2004
Novice
Posts: 6
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 » 1 person likes this post

Created a support case. Ticket # 03449308
Jeroen
Service Provider
Posts: 4
Liked: never
Joined: Apr 27, 2015 11:43 am
Full Name: Jeroen Dhondt
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by Jeroen »

I have the same problem. However, disabling the "Allow processing of multiple VMs with a single volume snapshot" didn't change anything.
Created a support ticket: Case # 03455876
georgeinva2004
Novice
Posts: 6
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 » 1 person likes this post

I found a fix for my issue.

While pulling the data requested by support, I noticed a number of CAPI32 errors (Event 513) while backups were in progress. I found this article: https://support.microsoft.com/en-us/hel ... ows-server, performed the steps and the issue stopped.
georgeinva2004
Novice
Posts: 6
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 »

The problem came back today for me. Has anyone else found a resolution?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by foggy »

Please continue working with your support engineer and refer to another case mentioned above (# 03455876) to check if the fix discussed there will help in your particular case.
melexis
Influencer
Posts: 12
Liked: 2 times
Joined: Mar 31, 2018 5:40 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by melexis »

Same issue here, since update from 9.5U3a to 9.5U4 we have random timeouts:
"Failed to finalize guest processing. Error: Failed to call RPC function 'Vss.WaitForBackupDocsToBeFlushed': Der Remoteprozeduraufruf (RPC) wurde abgebrochen. RPC function call failed. Function name: [DoRpc]. Target machine"

Working with support on this topic since 8th of February, without any progress. Case # 03405318

Also tried the suggestions in this thread but without success.
melexis
Influencer
Posts: 12
Liked: 2 times
Joined: Mar 31, 2018 5:40 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by melexis » 2 people like this post

Made some progress with support:
In my case, they confirmed it is a bug (came with Update 4) and a hotfix will be released soon.
Temporary solution is to kick VeeamPSDirectCtrl_X64 process on the affected Hyper-V Host, after that you're able to stop backup job (which was not possible before) and also backup retry works for me.
BrannelSchool
Lurker
Posts: 2
Liked: never
Joined: Apr 09, 2019 10:49 am
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by BrannelSchool »

Have you had access to the hotfix yet? We are experiencing the exact same issue, was hoping it may have been resolved in U4a but we are still struggling.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by foggy »

Yes, hotfix has been provided but the outcome is still not clear (at least from the case notes). Please contact technical support directly. If the hotfix applies to your issue, you will get access to it.
BrannelSchool
Lurker
Posts: 2
Liked: never
Joined: Apr 09, 2019 10:49 am
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by BrannelSchool »

Ahh thanks, I thought it would be a public hotfix. I have just opened a case #03506903
rymsza
Lurker
Posts: 1
Liked: never
Joined: Jun 28, 2019 5:34 pm
Full Name: Rodrigo Rymsza
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by rymsza »

We have exactly the same problem with update4a.
Temporary solution is to kick VeeamPSDirectCtrl_X64 process on the affected Hyper-V Host, after that you're able to stop backup job (which was not possible before) and also backup retry works for me.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by Gostev »

The issue is caused by PSDirect calls hanging under certain circumstances, and as far as I know it should be completely resolved in Update 4b.
doktornotor
Enthusiast
Posts: 94
Liked: 29 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by doktornotor »

Thanks, will test when 4b is out.
Stegerius
Novice
Posts: 6
Liked: never
Joined: Dec 08, 2014 7:45 am
Full Name: Lorenz Steger
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by Stegerius »

Update 4b is out.
Has anybody tried if the problem is resolved with it?

Thanks.
georgeinva2004
Novice
Posts: 6
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 »

It did fix my issue
Post Reply

Who is online

Users browsing this forum: No registered users and 259 guests