11a Upgrade Problems

[vertices]

Our Veeam B&R Server is in Azure. It connects to hyper-v servers and hardened Ubuntu repos in 2 sites.

We've been having some performance issues with backups running long, so I went to upgrade to 11a. 11.0.1.1261.

The upgrade went ok. I made a mistake and used the wrong account for one time password on the linux repos so the upgrades of the components on those were failing. I straightend that out and that worked for the first 2 repos but the 3rd repo won't upgrade. It's up to date on Ubuntu 20.04, has been rebooted, I can run through the properties on it just fine. But when I try to upgrade the components on it it fails with:

Failed to upgrade host components. Error: 'Invalid package type DeployerSvc'

I've tried everything I can think of but it always fails. The first two, that I set up at that same time in the same way had no problems upgrading.

The other issue I have is that my main B&R server, which is in Azure, says it has to upgrade Hyper-V components on it. But it can't as Hyper-V role is not on it. It never was. I can't add that role to an Azure VM. I don't know why with 11a it thinks it needs to add Hyper-V components but it does.

On the B&R Server I get:

Failed to upgrade host components. Error: 'This server is not a Hyper-V host.'

So I have 1 repo showing can't upgrade as well as the B&R server.

Any ideas?

Ticket ID: #02444580

Thanks.

[vertices]

Man I can't even get support to respond. My initial notes didn't appear to be fully read when the first response came. I asked again to call me but no call. I try to call in but you need a case number to talk to ANYONE. I enter my case number, which is clearly still showing open on the portal, but the auto attendant tells me it's closed and hangs up on me.

If I call 1 800 774 5124 that attendant tells me it's closed.
If I call 1 800 913 1940 that attendant tells me the case ID is invalid.

Neither provide anyway to get through this.

This is not a very good tech support experience Veeam. I love you, but this really sucks.

[vertices]

I don't care who I call it won't let me through. I get that you guys need people to make cases online, but this is pretty ridiculous that I can't get a response, and can't get through the auto attendant without it hanging up on me. God forbid I had production down right now. I'm not a very happy Veeamer at the moment. Case is set as open and severity 2 but the AA tells me to take a hike. There has to be a better way. I am fully unable to contact support.

[vertices]

Well, I guess I will go sit on the couch with my phone by my side hoping that my ticket notes are read this time and someone calls me back. Super frustrating. I will definitely be requesting mgmt at the end of this to find out what to do in the future. Like I said, if production was down right now, I would be losing my mind with this broken system.

[Egor Yakovlev]

Hi Rob,

Your opened support case ID is 05115636, it is different from contract ID.

Currently case severity level is set to "2", so please go to my.veeam.com and raise your ticket to Severity 1 + attach product logs as was requested by email reply from support.
They are eager to help.

/Thanks!

[vertices]

Well that helps. I didn't even see case up there. I was looking at the subject and saw ID# 02444580. Confusing I guess. Thanks, I will call now.

[vertices]

I think it might be nice to have this system changed in some way. When she says on the phone "Enter your ticket ID that starts with 0" and you are stressed and tired and looking at your case, up top you see ID# 0xxxxxxx and enter that, and she just tells you the case is closed and hangs up on you, idk, I think that process could be improved. Perhaps clearly stating that contract numbers may also start with 0 and don't enter that. Or maybe once entering the wrong number, she then prompts some additional info such as "Your contract ID may also start with 0, ensure you are not entering that, try again". Idk something other than "goodbye" would be nice there.

Either way, I did finally talk to someone. But just gathering more logs. No fixes for anything yet. Thanks for letting me know though Egor.

[Egor Yakovlev]

Passed your feedback up - it is not the first time I see people are being confused with Contract ID and Case ID, so some tuning might be done indeed.
Thanks for bringing that up and hope your case will get resolved asap!

[vertices]

Ok I am coming back to this post and starting fresh.

Case ID: 05115636

Design:

Azure Site: Veeam B&R Server with 8C/32GB running 11a 11.0.1.1261
Site 1: - SOBR-1 with a single Ubuntu 20.04 Hardened Repo on Dell Hardware, repo1, XFS Fast clone
- SOBR-2 with a single Ubuntu 20.04 Hardened Repo on Dell Hardware, repo2, XFS Fast clone
- Primary Hyper-V cluster
Site 2: - SOBR-3 with a single Ubuntu 20.04 Hardened Repo on Dell Hardware, repo3, XFS Fast clone
- DR Hyper-V Cluster


General flow:
Everything is controlled from B&R in Azure. Tunnels to both sites. Both sites have 1G internet. Sites also have a L2 650mbps private link between them. Backups roll from primary HV cluster to SOBR-1 every 2 hours. Every 6 hours they are copied from SOBR-1 to SOBR-2. SOBR-2 has a capacity tier to Wasabi. Backups are also copied every 8 hours from SOBR-2 to SOBR-3 across the private L2 link. Replication jobs run to DR cluster from SOBR-3 source every 12 hours. About 65 VMs total protected, and none are huge as in more than maybe 1.5TB. Most are in the 300GB range or less. About 14TB of total VMs.

Problem 1:
The copy jobs from SOBR-1 to SOBR-2, which are right next to each other are slow. They are slow due to just general Veeam slowness. It can take 30 minutes just to build the backup list of VMs when there are only 37 in the job. So 30 minutes lost just figuring out what VMs to copy from SOBR-1 to SOBR-2. Once it decides to actually copy, it flies by at high speed. Then spends oodles of time “finalizing”. A disk will copy, once it decides to copy, in like 9 seconds, then it will spend an hour “finalizing” that copy. While it's finalizing, it’s consuming job slots on the repos, so everything slows and gets backed up. There is simply lots of pausing going on during these copy jobs. During this slowness, analyzing both repos, and the Veeam server, there is nothing going on. No disk or network IO, no CPU or mem usage. Just idle servers sitting around thinking about maybe making a copy.

Problem 2: In an attempt to fix problem 1, I upgraded from Veeam 11 to 11a. That didn’t fix any problem. Instead, it brought new problems. The upgrade went off without a hitch, no issues. Logged in first time and of course it tries to upgrade infrastructure components. That worked for all but 2 hosts.
- The Veeam B&R server itself, Veeam is saying it needs Hyper-V components. Why, idk, I am assuming it’s incorrectly identifying client services running in it since it’s a VM on Azure. It has never had a Hyper-V role, didn’t have it during version 11, but now 11a incorrectly IDs this as a Hyper-V server, so tries to installs components, which it can’t since Hyper-V role isn’t added. I can’t add the role either as MS doesn’t allow that. The error is:
- veeam1.xxx.xxx.com: Failed to upgrade host components. Error: 'This server is not a Hyper-V host.'
- Repo3 won’t upgrade. It’s up and running, it’s been rebooted. It’s at the same exact OS level and patch level, with the same configuration, as repo1 and repo2 which upgraded fine. The error is:
- repo3.xxx.xxx.com: Failed to upgrade host components. Error: 'Invalid package type DeployerSvc'

Problem 3: I also now have errors in the capacity tier offloads from repo2 in SOBR-2. This appears to be a known issue, as I have the same errors peppered throughout our offloads now. object-storage-f52/attempt-to-overwrite ... 76913.html I just haven’t received the fix from support yet.

Problem 4: Since upgrading to 11a, I now often get VM backup failures. The backup of the whole VM fails with “ Error: Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again”. This happens randomly to pretty much all VMs. A retry fixes it. It’s not terrible, as it’s usually successful at the next run, but this is new and annoying behavior.


I think that’s it. I’ve been trying to work with support, but between schedules, and such it’s slow going. I now have just 1 guy I am working with, which helps, but so far we have made no progress. I would really like to start putting some of these problems behind us. We are scheduled for another session tomorrow morning, but I think we could use some guidance.

If anyone has any advice, I would appreciate it.

Thanks,

Rob

[PetrM]

Hello,

I can advice: you should ask your support engineer to spin off the main case so that you have exactly one case for every problem. So far, you've reported 4 different issues and all of them require a precise analysis and debug logs examination. You'll get fix or workaround sooner if different problems are processed in parallel in multiple cases. We usually try to keep different issues separately in order to increase the efficiency of interaction with our support team. One more option is to ask escalation to a higher level of support if you feel that deeper technical analysis is needed. Also, I'll ask our support leaders to pay attention to your case and perhaps find more resources to accelerate the investigation.

Thanks!

[vertices]

Thanks. We are splitting it up now. Yes we need to find a way forward, 11a has been the buggiest version of Veeam I've ever used and I've been using it since v4 IIRC. Not trying to throw stones at it, but I've never seen it just fail in so many ways. It's just so slow in every way. In addition to taking forever just to build VM lists, it will sit with a copy job at 99% completed for over an hour. All VMs show success, nothing else to do. Job at 99% and just sitting there with no activity, no clue what it is doing. Just idle at 99%. There are just so many things wrong that it's hard to keep track. I'm working now on deploying the fixes everywhere for Problem 3, but just trying to get jobs to complete so I have a consistent environment in which to upgrade is painful.

[vertices]

Well the fix for problem 3 completely broke my linux hosts. All backups are now unable to run. Time to call back in I guess.

[Gostev]

11a has been the buggiest version of Veeam I've ever used and I've been using it since v4 IIRC. Not trying to throw stones at it, but I've never seen it just fail in so many ways.

Just bad luck as 11a is actually the first release ever in our history which didn't result in a spike of support cases. Contrary to that, they have been on the consistent downward trend since launch with every week bringing less cases than previous. Although this is expected as 11a fixes thousands of bugs.

[vertices]

Ok repos are up again. We had to update the instructions for the hotfix as they didn't work for the linux repos. Just had to run them one at a time rather than chained together.

All components are showing up to date now, so problem 2 is resolved, and the hotfix is installed everywhere so I suspect that will fix problem 3. I still have problem 4 above, so many VMs fail to backup with that error and then run fine on a retry. I suspect I still have problem 1 above, working on determining that. At least we have some forward progress!

[PetrM]

It's absolutely expected that you have progress I've no doubts that all of the problems will be resolved in the end, just ETA of the fix depends on technical complexity of the issue.

Thanks!

[vertices]

Well, in the end, there were just too many issues. I felt it would waste too much of Veeam's time, and my time, trying to get everything squared away. So I took out the sledgehammer. I stood up a new Veeam B&R server in production, as opposed to in Azure. Dropped the db and started fresh, since I had that db since at least v8 and it was loaded with just so many old resources, so I didn't want to migrate it. I took the opportunity to clean everything up and organize the jobs better than before, and applied the hotfix above before putting it production. I can say that this act appears to have solved every single problem I had. Everything is fast again. Backups don't hang at the beginning or end, and it's just all very performant. Sometimes, it's nice just to start over.