-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Failing Backups - Many Azure Tenants
Case #07049634
Case #07038738
We have many customers that now have failing backups for certain VMs. Other VMs in the same policy backup fine, but others won't. We have tried everything we can think of:
Refreshing Azure Account Permissions
Changing to Message Queue from Service Bus
Updating & rebooting appliances
Increasing size of workers
We are seeing errors including this text:
Error: Cluster with parameter {xxxxxxxxxxxxxxxxxx] already exist
Agent failed to process method {Cloud.DeleteCheckpoint}. in vcp::CVCPCommandArgsPtr vcp::CDispatchedCommandSet<TCommandSet>::Dispatch(vcp::CVCPCommandPtr) [with TCommandSet = frontend::CCloudCommandSet; vcp::CVCPCommandArgsPtr = boost::shared_ptr<vcp::CVCPCommandArgs>; vcp::CVCPCommandPtr = boost::shared_ptr<vcp::CVCPCommand>] at /mnt/VeeamAgentEx/Sources/CommonLib/vcp/DispatchedCommandSet.h:49. We will retry in 00:10:00...
[08.12.2023 23:08:04.520] Error 54 (1) Error: Worker VBA-1a7eabe1-c792-480c-8106-e88f3618e412 (xxxxxxxx): job completed with error. Error: Cluster with parameter {xxxxxxxxxxxxxxx] already exist
Agent failed to process method {Cloud.DeleteCheckpoint}. in vcp::CVCPCommandArgsPtr vcp::CDispatchedCommandSet<TCommandSet>::Dispatch(vcp::CVCPCommandPtr) [with TCommandSet = frontend::CCloudCommandSet; vcp::CVCPCommandArgsPtr = boost::shared_ptr<vcp::CVCPCommandArgs>; vcp::CVCPCommandPtr = boost::shared_ptr<vcp::CVCPCommand>] at /mnt/VeeamAgentEx/Sources/CommonLib/vcp/DispatchedCommandSet.h:49
So far, the only solution offered is to make all new st accounts and start over. This is not an accepatable solution. Many of these are less than 3 months old. My ticket is with R&D, which does not give me the warm and fuzzies when we have so many ongoing failures, and they went on for a bit here due to VSPC not alerting.....
Our VSPC did not alert us to all of these failures (separate issue and ticket), so we are only finding out now.
Is anyone else seeing errors like this? No clue why we would be hit across numerous environments with no clear answer from Veeam yet.
Case #07038738
We have many customers that now have failing backups for certain VMs. Other VMs in the same policy backup fine, but others won't. We have tried everything we can think of:
Refreshing Azure Account Permissions
Changing to Message Queue from Service Bus
Updating & rebooting appliances
Increasing size of workers
We are seeing errors including this text:
Error: Cluster with parameter {xxxxxxxxxxxxxxxxxx] already exist
Agent failed to process method {Cloud.DeleteCheckpoint}. in vcp::CVCPCommandArgsPtr vcp::CDispatchedCommandSet<TCommandSet>::Dispatch(vcp::CVCPCommandPtr) [with TCommandSet = frontend::CCloudCommandSet; vcp::CVCPCommandArgsPtr = boost::shared_ptr<vcp::CVCPCommandArgs>; vcp::CVCPCommandPtr = boost::shared_ptr<vcp::CVCPCommand>] at /mnt/VeeamAgentEx/Sources/CommonLib/vcp/DispatchedCommandSet.h:49. We will retry in 00:10:00...
[08.12.2023 23:08:04.520] Error 54 (1) Error: Worker VBA-1a7eabe1-c792-480c-8106-e88f3618e412 (xxxxxxxx): job completed with error. Error: Cluster with parameter {xxxxxxxxxxxxxxx] already exist
Agent failed to process method {Cloud.DeleteCheckpoint}. in vcp::CVCPCommandArgsPtr vcp::CDispatchedCommandSet<TCommandSet>::Dispatch(vcp::CVCPCommandPtr) [with TCommandSet = frontend::CCloudCommandSet; vcp::CVCPCommandArgsPtr = boost::shared_ptr<vcp::CVCPCommandArgs>; vcp::CVCPCommandPtr = boost::shared_ptr<vcp::CVCPCommand>] at /mnt/VeeamAgentEx/Sources/CommonLib/vcp/DispatchedCommandSet.h:49
So far, the only solution offered is to make all new st accounts and start over. This is not an accepatable solution. Many of these are less than 3 months old. My ticket is with R&D, which does not give me the warm and fuzzies when we have so many ongoing failures, and they went on for a bit here due to VSPC not alerting.....
Our VSPC did not alert us to all of these failures (separate issue and ticket), so we are only finding out now.
Is anyone else seeing errors like this? No clue why we would be hit across numerous environments with no clear answer from Veeam yet.
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
This cannot be a solution, fully agree with you. I will talk with the RnD team once those tickets are escalated to us. Based on what I see now, our QA team has requested additional info from you to troubleshoot it.RobMiller86 wrote:So far, the only solution offered is to make all new st accounts and start over. This is not an accepatable solution.
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
Thanks Vitaliy, although I just checked both of those tickets and I don't see where they have requested anything. Both simply say waiting on R&D.
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
Yes, QA is communicating with the our support now. Let's see what they can find out from the uploaded logs.
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
We are having similar issues, all started on Monday or Tuesday.
Ticket has been escalated to RnD like yourself. Snapshots are running fine, but it's weird that 7 out of 35 of our servers are backing up fine... Same repo, same storage accounts, same region, everything...
Ticket has been escalated to RnD like yourself. Snapshots are running fine, but it's weird that 7 out of 35 of our servers are backing up fine... Same repo, same storage accounts, same region, everything...
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
Hi Will,
Yes, I see that the pattern might be the same. Our RnD team is actively looking into this..., so thanks for sharing your case ID in another thread.
Thanks!
Yes, I see that the pattern might be the same. Our RnD team is actively looking into this..., so thanks for sharing your case ID in another thread.
Thanks!
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
I feel for you Will, but at least I'm not alone. I personally feel that one of the updates to VBAZ caused this. Which would align with VCC updating the appliances and then they all broke at once for us. I suspect as time goes on, Veeam support is going to see many more of these tickets unless the update is identified and pulled.
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
Rob,
I think your right, although I did try to roll back to 5a (by deploying an new appliance), and restore the config, although I am almost 100% that on boot the appliance installs any updates which pushed it to 6.
I was on with a T2 Agent for 2hrs the other day trying all sorts so I guess its reassuring that it is getting attention as you say, the more people with the issue the more pressing it becomes and again, glad to see it's not just me.
Fingers crossed.
I think your right, although I did try to roll back to 5a (by deploying an new appliance), and restore the config, although I am almost 100% that on boot the appliance installs any updates which pushed it to 6.
I was on with a T2 Agent for 2hrs the other day trying all sorts so I guess its reassuring that it is getting attention as you say, the more people with the issue the more pressing it becomes and again, glad to see it's not just me.
Fingers crossed.
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
And for that matter, with our ongoing VSPC issues, default alarms that aren't working properly, I wonder if those VSPC issues are causing other service providers to not even be aware. Our VSPC is fairly new, it was deployed as v7. It's well maintained and not customized much at all yet. I don't think the issues with it are unique to us. A bug with VBAZ, that then is missed by VSPC, this could be a far larger issue than it seems currently.
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
One pattern I see is that an update is applied to VBAZ and it reboots, then the next backup is successful, and then the following backup has the error.
We had many with a policy for hourly snapshots, and backups to blob only at 11am and 11pm. On Dec 5th when I upgraded Cloud Connect, the 11am backups were successful, then updates were applied in the afternoon when Cloud Connect was upgraded 12.1, then 11pm backups were successful, then the next 11am backup has many VM failures, and it seems to be larger VMs.
So that first backup runs, but it does something to the backup data in the repo that causes the following backup to fail for what seems to be larger VMs.
We had many with a policy for hourly snapshots, and backups to blob only at 11am and 11pm. On Dec 5th when I upgraded Cloud Connect, the 11am backups were successful, then updates were applied in the afternoon when Cloud Connect was upgraded 12.1, then 11pm backups were successful, then the next 11am backup has many VM failures, and it seems to be larger VMs.
So that first backup runs, but it does something to the backup data in the repo that causes the following backup to fail for what seems to be larger VMs.
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
I have experienced similar, the machines that are failing are predominantly those with storage and compute higher than average 1TB+ or thereabouts.
I'm quite surprised that it's taking so long, and also that more users aren't experiencing these issues.
I'm quite surprised that it's taking so long, and also that more users aren't experiencing these issues.
-
- Product Manager
- Posts: 5838
- Liked: 1220 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Failing Backups - Many Azure Tenants
Hi all, please continue working via support. We are still actively working on this and hope to deliver a fix soon.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
I am but, but I have received not a single update from support even when I asked. I wouldn't know that a root cause has been identified if it weren't for these forums and Wills update. Have had failing backups since before 12/5 that were missed due to VSPC alarm issues, and then many more after 12/5. It's now 12/19 so this is becoming a significant issue. I'm sure they are busy working on it, but having some kind of info explaining the cause and an ETA for a fix would be much appreciated.
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
The fix is being tested right now, so it depends on the QA team. It should be signed off today or tomorrow.
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
Any update on this? I guess it's gonna be tomorrow now?
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
Received the fix this morning, applied and backups are running.
5 out of 7 failing machines have complete with the final 2 still running. All signs seem to indicate we are good.
5 out of 7 failing machines have complete with the final 2 still running. All signs seem to indicate we are good.
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
Perefct! I was about to write that the fix has been shared
. Please keep us updated on the results.

-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
I'm actually on PTO until the 26th. I did just see the fix. Although it also seems that if you just upgrade the appliance, you don't need this fix.
I guess I will try an upgrade first rather than trying the fix. When I have some time here. I'll prob test one myself later today and then have my team perform upgrades.No, the fix is for v6 only. If when you upgrade your 5a to v6 the version is 6.0.0.237 or above, you don't need to apply the fix. If when you upgrade the version is the current public available one then yes.
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
Got it! Keep us informed on how it goes too.
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
Having some confusion with support. They are telling me this appliace is v6 and not v5a so I'm good now to apply the fix. They also state it was upgraded when I upgraded Cloud Connect to 12.1. Makes sense, but all of the image plans in Azure still state v5a. In VSPC, all of our CC connected appliances show as using version 6.0.0.234. The one that I deployed fresh as v6 is showing 6.0.0.234, and all of the ones deployed as v5a also show 6.0.0.234. I always thought that to go up a major version, you need a new image plan. So is Veeam now updating appliances automatically like this and leaving the older image plan in place?
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
Support is saying it doesn't update the image plan, so I guess I'm good to go. They state they are all in fact v6, even though the image plan wasn't changed. Time to get crackin' I guess. Thanks for the fix Veeam!
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
I have had great success with the fix, 6 out of 7 larger servers have now completed with the final looking like its going to follow suit.
My appliance is now showing 6.0.0.237.
Thanks for the speed of the fix guys.
My appliance is now showing 6.0.0.237.
Thanks for the speed of the fix guys.
-
- Service Provider
- Posts: 192
- Liked: 38 times
- Joined: Oct 28, 2019 7:10 pm
- Full Name: Rob Miller
- Contact:
Re: Failing Backups - Many Azure Tenants
Thanks for the confirmation Will!
-
- VP, Product Management
- Posts: 27463
- Liked: 2825 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Failing Backups - Many Azure Tenants
Yes, the GA image remains the same, but upon new deployments, all necessary updates are also installed. The same is done for existing installations (all fixes are gathered and installed from the Veeam repo via Veeam Updater).RobMiller86 wrote:So is Veeam now updating appliances automatically like this and leaving the older image plan in place?
-
- Influencer
- Posts: 11
- Liked: 3 times
- Joined: Dec 15, 2023 6:44 pm
- Full Name: Will Holmes
- Contact:
Re: Failing Backups - Many Azure Tenants
Once you deploy a fresh appliance these days, if you log onto the webui quick enough you will see it installs all available updates on initial boot
Who is online
Users browsing this forum: No registered users and 4 guests