Agentless, cloud-native backup for Microsoft Azure
Post Reply
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Failing Backups - Many Azure Tenants

Post by RobMiller86 »

Case #07049634
Case #07038738

We have many customers that now have failing backups for certain VMs. Other VMs in the same policy backup fine, but others won't. We have tried everything we can think of:

Refreshing Azure Account Permissions
Changing to Message Queue from Service Bus
Updating & rebooting appliances
Increasing size of workers

We are seeing errors including this text:

Error: Cluster with parameter {xxxxxxxxxxxxxxxxxx] already exist
Agent failed to process method {Cloud.DeleteCheckpoint}. in vcp::CVCPCommandArgsPtr vcp::CDispatchedCommandSet<TCommandSet>::Dispatch(vcp::CVCPCommandPtr) [with TCommandSet = frontend::CCloudCommandSet; vcp::CVCPCommandArgsPtr = boost::shared_ptr<vcp::CVCPCommandArgs>; vcp::CVCPCommandPtr = boost::shared_ptr<vcp::CVCPCommand>] at /mnt/VeeamAgentEx/Sources/CommonLib/vcp/DispatchedCommandSet.h:49. We will retry in 00:10:00...
[08.12.2023 23:08:04.520] Error 54 (1) Error: Worker VBA-1a7eabe1-c792-480c-8106-e88f3618e412 (xxxxxxxx): job completed with error. Error: Cluster with parameter {xxxxxxxxxxxxxxx] already exist
Agent failed to process method {Cloud.DeleteCheckpoint}. in vcp::CVCPCommandArgsPtr vcp::CDispatchedCommandSet<TCommandSet>::Dispatch(vcp::CVCPCommandPtr) [with TCommandSet = frontend::CCloudCommandSet; vcp::CVCPCommandArgsPtr = boost::shared_ptr<vcp::CVCPCommandArgs>; vcp::CVCPCommandPtr = boost::shared_ptr<vcp::CVCPCommand>] at /mnt/VeeamAgentEx/Sources/CommonLib/vcp/DispatchedCommandSet.h:49

So far, the only solution offered is to make all new st accounts and start over. This is not an accepatable solution. Many of these are less than 3 months old. My ticket is with R&D, which does not give me the warm and fuzzies when we have so many ongoing failures, and they went on for a bit here due to VSPC not alerting.....

Our VSPC did not alert us to all of these failures (separate issue and ticket), so we are only finding out now.

Is anyone else seeing errors like this? No clue why we would be hit across numerous environments with no clear answer from Veeam yet.
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

RobMiller86 wrote:So far, the only solution offered is to make all new st accounts and start over. This is not an accepatable solution.
This cannot be a solution, fully agree with you. I will talk with the RnD team once those tickets are escalated to us. Based on what I see now, our QA team has requested additional info from you to troubleshoot it.
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

Thanks Vitaliy, although I just checked both of those tickets and I don't see where they have requested anything. Both simply say waiting on R&D.
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

Yes, QA is communicating with the our support now. Let's see what they can find out from the uploaded logs.
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS »

We are having similar issues, all started on Monday or Tuesday.

Ticket has been escalated to RnD like yourself. Snapshots are running fine, but it's weird that 7 out of 35 of our servers are backing up fine... Same repo, same storage accounts, same region, everything...
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

Hi Will,

Yes, I see that the pattern might be the same. Our RnD team is actively looking into this..., so thanks for sharing your case ID in another thread.

Thanks!
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

I feel for you Will, but at least I'm not alone. I personally feel that one of the updates to VBAZ caused this. Which would align with VCC updating the appliances and then they all broke at once for us. I suspect as time goes on, Veeam support is going to see many more of these tickets unless the update is identified and pulled.
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS »

Rob,

I think your right, although I did try to roll back to 5a (by deploying an new appliance), and restore the config, although I am almost 100% that on boot the appliance installs any updates which pushed it to 6.

I was on with a T2 Agent for 2hrs the other day trying all sorts so I guess its reassuring that it is getting attention as you say, the more people with the issue the more pressing it becomes and again, glad to see it's not just me.

Fingers crossed.
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

And for that matter, with our ongoing VSPC issues, default alarms that aren't working properly, I wonder if those VSPC issues are causing other service providers to not even be aware. Our VSPC is fairly new, it was deployed as v7. It's well maintained and not customized much at all yet. I don't think the issues with it are unique to us. A bug with VBAZ, that then is missed by VSPC, this could be a far larger issue than it seems currently.
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

One pattern I see is that an update is applied to VBAZ and it reboots, then the next backup is successful, and then the following backup has the error.

We had many with a policy for hourly snapshots, and backups to blob only at 11am and 11pm. On Dec 5th when I upgraded Cloud Connect, the 11am backups were successful, then updates were applied in the afternoon when Cloud Connect was upgraded 12.1, then 11pm backups were successful, then the next 11am backup has many VM failures, and it seems to be larger VMs.

So that first backup runs, but it does something to the backup data in the repo that causes the following backup to fail for what seems to be larger VMs.
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS »

I have experienced similar, the machines that are failing are predominantly those with storage and compute higher than average 1TB+ or thereabouts.

I'm quite surprised that it's taking so long, and also that more users aren't experiencing these issues.
nielsengelen
Product Manager
Posts: 5838
Liked: 1220 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Failing Backups - Many Azure Tenants

Post by nielsengelen »

Hi all, please continue working via support. We are still actively working on this and hope to deliver a fix soon.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

I am but, but I have received not a single update from support even when I asked. I wouldn't know that a root cause has been identified if it weren't for these forums and Wills update. Have had failing backups since before 12/5 that were missed due to VSPC alarm issues, and then many more after 12/5. It's now 12/19 so this is becoming a significant issue. I'm sure they are busy working on it, but having some kind of info explaining the cause and an ETA for a fix would be much appreciated.
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

The fix is being tested right now, so it depends on the QA team. It should be signed off today or tomorrow.
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS »

Any update on this? I guess it's gonna be tomorrow now?
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS »

Received the fix this morning, applied and backups are running.

5 out of 7 failing machines have complete with the final 2 still running. All signs seem to indicate we are good.
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

Perefct! I was about to write that the fix has been shared ;). Please keep us updated on the results.
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 » 1 person likes this post

I'm actually on PTO until the 26th. I did just see the fix. Although it also seems that if you just upgrade the appliance, you don't need this fix.
No, the fix is for v6 only. If when you upgrade your 5a to v6 the version is 6.0.0.237 or above, you don't need to apply the fix. If when you upgrade the version is the current public available one then yes.
I guess I will try an upgrade first rather than trying the fix. When I have some time here. I'll prob test one myself later today and then have my team perform upgrades.
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

Got it! Keep us informed on how it goes too.
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

Having some confusion with support. They are telling me this appliace is v6 and not v5a so I'm good now to apply the fix. They also state it was upgraded when I upgraded Cloud Connect to 12.1. Makes sense, but all of the image plans in Azure still state v5a. In VSPC, all of our CC connected appliances show as using version 6.0.0.234. The one that I deployed fresh as v6 is showing 6.0.0.234, and all of the ones deployed as v5a also show 6.0.0.234. I always thought that to go up a major version, you need a new image plan. So is Veeam now updating appliances automatically like this and leaving the older image plan in place?
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 » 1 person likes this post

Support is saying it doesn't update the image plan, so I guess I'm good to go. They state they are all in fact v6, even though the image plan wasn't changed. Time to get crackin' I guess. Thanks for the fix Veeam!
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS » 2 people like this post

I have had great success with the fix, 6 out of 7 larger servers have now completed with the final looking like its going to follow suit.

My appliance is now showing 6.0.0.237.

Thanks for the speed of the fix guys.
RobMiller86
Service Provider
Posts: 192
Liked: 38 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Failing Backups - Many Azure Tenants

Post by RobMiller86 »

Thanks for the confirmation Will!
Vitaliy S.
VP, Product Management
Posts: 27463
Liked: 2825 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failing Backups - Many Azure Tenants

Post by Vitaliy S. »

RobMiller86 wrote:So is Veeam now updating appliances automatically like this and leaving the older image plan in place?
Yes, the GA image remains the same, but upon new deployments, all necessary updates are also installed. The same is done for existing installations (all fixes are gathered and installed from the Veeam repo via Veeam Updater).
WillholmesNBS
Influencer
Posts: 11
Liked: 3 times
Joined: Dec 15, 2023 6:44 pm
Full Name: Will Holmes
Contact:

Re: Failing Backups - Many Azure Tenants

Post by WillholmesNBS » 1 person likes this post

Once you deploy a fresh appliance these days, if you log onto the webui quick enough you will see it installs all available updates on initial boot
Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests