Comprehensive data protection for all workloads
Post Reply
doktornotor
Enthusiast
Posts: 35
Liked: 10 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by doktornotor » Feb 07, 2019 9:18 am 1 person likes this post

Somehow, when playing with 9.5U4 and repo configuration on a test box, I managed to break the whole backup job by setting"Limit maximum concurrent tasks" for the backup repository to a number lower than the number of VMs that are being processed in a backup job that is using that repository, while that job had "Allow processing of multiple VMs with a single volume snapshot" enabled.

This reliably causes the backup job to fail. Say, you have concurrent tasks set to 2 and there are 3 VMs included in the job. The backup for the first two VMs fails like this:

Code: Select all

Failed to finalize guest processing. Error: Failed to call RPC function 'Vss.WaitForBackupDocsToBeFlushed': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc].
After two first 2 VMs failed, the backup continues with the third machine, and there the job just hangs indefinitely, doing nothing, job cannot be stopped either, the Veeam Backup Service fails to stop as well and needs to be killed via task manager. The checkpoint for the third VM if left behind and needs to be merged manually via PS - "Get-VMSnapshot <VM> | Remove-VMSnapshot"

The issue immediately goes away when "Allow processing of multiple VMs with a single volume snapshot" in unchecked from the job, the VMs get processed in batches corresponding to the concurrent tasks limit and everything finishes just fine.

Somehow I'd say users should be prevented from creating invalid configuration that completely breaks the backup. :idea: Someone else able to reproduce this? :?:

DGrinev
Veeam Software
Posts: 1650
Liked: 201 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by DGrinev » Feb 07, 2019 11:40 am 1 person likes this post

Hi,

I've discussed your issue with the QA team, the described scenario shouldn't result as you've seen.
It's impossible to understand the root cause without deeper investigation.

Please contact the support team and provide the logs, then don't forget to share the case ID with us here. Thanks!

doktornotor
Enthusiast
Posts: 35
Liked: 10 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by doktornotor » Feb 07, 2019 11:50 am

the described scenario shouldn't result as you've seen.
Yes, agreed that this is definitely not the desired outcome. :)
Please contact the support team and provide the logs, then don't forget to share the case ID with us here. Thanks!
Ok, will get to it after the weekend, hopefully...

Thanks!

georgeinva2004
Novice
Posts: 4
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 » Mar 05, 2019 9:39 pm

I am having the same issue. Did you ever get this resolved?

veremin
Product Manager
Posts: 16553
Liked: 1377 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by veremin » Mar 06, 2019 12:51 pm

The OP has never come back to update the topic with case ID, so we cannot check the status of the ticket and whether the resolution was provided.

So, kindly, open your own case and post its number here.

Thanks!

georgeinva2004
Novice
Posts: 4
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 » Mar 06, 2019 2:43 pm 1 person likes this post

Created a support case. Ticket # 03449308

Jeroen
Service Provider
Posts: 1
Liked: never
Joined: Apr 27, 2015 11:43 am
Full Name: Jeroen Dhondt
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by Jeroen » Mar 11, 2019 10:58 am

I have the same problem. However, disabling the "Allow processing of multiple VMs with a single volume snapshot" didn't change anything.
Created a support ticket: Case # 03455876

georgeinva2004
Novice
Posts: 4
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 » Mar 11, 2019 12:46 pm 1 person likes this post

I found a fix for my issue.

While pulling the data requested by support, I noticed a number of CAPI32 errors (Event 513) while backups were in progress. I found this article: https://support.microsoft.com/en-us/hel ... ows-server, performed the steps and the issue stopped.

georgeinva2004
Novice
Posts: 4
Liked: 2 times
Joined: Nov 21, 2016 5:51 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by georgeinva2004 » Mar 20, 2019 1:39 pm

The problem came back today for me. Has anyone else found a resolution?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by foggy » Mar 20, 2019 6:17 pm

Please continue working with your support engineer and refer to another case mentioned above (# 03455876) to check if the fix discussed there will help in your particular case.

melexis
Novice
Posts: 9
Liked: 2 times
Joined: Mar 31, 2018 5:40 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by melexis » Mar 22, 2019 10:14 am

Same issue here, since update from 9.5U3a to 9.5U4 we have random timeouts:
"Failed to finalize guest processing. Error: Failed to call RPC function 'Vss.WaitForBackupDocsToBeFlushed': Der Remoteprozeduraufruf (RPC) wurde abgebrochen. RPC function call failed. Function name: [DoRpc]. Target machine"

Working with support on this topic since 8th of February, without any progress. Case # 03405318

Also tried the suggestions in this thread but without success.

melexis
Novice
Posts: 9
Liked: 2 times
Joined: Mar 31, 2018 5:40 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by melexis » Mar 25, 2019 10:03 am 2 people like this post

Made some progress with support:
In my case, they confirmed it is a bug (came with Update 4) and a hotfix will be released soon.
Temporary solution is to kick VeeamPSDirectCtrl_X64 process on the affected Hyper-V Host, after that you're able to stop backup job (which was not possible before) and also backup retry works for me.

BrannelSchool
Lurker
Posts: 2
Liked: never
Joined: Apr 09, 2019 10:49 am
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by BrannelSchool » Apr 09, 2019 10:53 am

Have you had access to the hotfix yet? We are experiencing the exact same issue, was hoping it may have been resolved in U4a but we are still struggling.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by foggy » Apr 09, 2019 4:26 pm

Yes, hotfix has been provided but the outcome is still not clear (at least from the case notes). Please contact technical support directly. If the hotfix applies to your issue, you will get access to it.

BrannelSchool
Lurker
Posts: 2
Liked: never
Joined: Apr 09, 2019 10:49 am
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by BrannelSchool » Apr 10, 2019 11:39 am

Ahh thanks, I thought it would be a public hotfix. I have just opened a case #03506903

rymsza
Lurker
Posts: 1
Liked: never
Joined: Jun 28, 2019 5:34 pm
Full Name: Rodrigo Rymsza
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by rymsza » Jun 28, 2019 5:50 pm

We have exactly the same problem with update4a.
Temporary solution is to kick VeeamPSDirectCtrl_X64 process on the affected Hyper-V Host, after that you're able to stop backup job (which was not possible before) and also backup retry works for me.

Gostev
SVP, Product Management
Posts: 24304
Liked: 3335 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by Gostev » Jun 29, 2019 8:37 pm

The issue is caused by PSDirect calls hanging under certain circumstances, and as far as I know it should be completely resolved in Update 4b.

doktornotor
Enthusiast
Posts: 35
Liked: 10 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: Single volume snapshot for multiple VMs vs. Max. concurrent tasks allowed for repo

Post by doktornotor » Jul 01, 2019 7:07 am

Thanks, will test when 4b is out.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], BradI, Google [Bot], kloppo, Mgamerz and 42 guests