Reaching for straws to avoid identifying and solving the problem

[TitaniumCoder477]

Almost a decade ago, when I took over the backup responsibilities for my employer, I dealt with mostly VMware and the occasional Hyper-V. Back in those days, our Hyper-V clients were tiny with typically a single host and a few VMs. Veeam was installed by many engineers directly onto the host (which <gasp> was often domain-joined at that time!). This worked fine with few issues. But fast forward nine years, and we literally have one legacy client that is setup that way. They have literally one VM running on the host on which Veeam lives. And it's still working fine.

But I get it. Veeam made it official (I think in v11? https://helpcenter.veeam.com/docs/backu ... ml?ver=110) with a bullet that said, "Veeam Backup & Replication must not be installed directly on a Hyper-V host. Such installation may lead to unpredictable system behavior. Instead, create a VM on the host and install Veeam Backup & Replication on the VM."

I was not aware of this, and while my practice (and indirectly my employer since I was architect and project engineer, among many other things) of installing Veeam on Hyper-V hosts had long since passed, when Veeam introduced the support for virtualization of VMware backups on Hyper-V hosts and vise versus, I was delighted! Finally, we could create a sort of "BDR", i.e. a physical Windows server w/ Hyper-V enabled to conduct our yearly backup verification tests for our clients. We could also use it in a pinch for emergencies. This was direct competition against Datto. No longer would we need to always repurpose old VMware hosts or Hyper-V hosts. We could literally create an AIO appliance.

So I made sure to scope physical servers that had plenty of RAM and CPU. Then after deploying Veeam, I would enable the Hyper-V role, create a "Sandbox Network" for testing purposes, and the next time we'd use it would be during the yearly tests.

Recently I opened case 06270093 when I noticed Veeam not retentioning a Windows Agent Backup job. I was told it was a bug (KB4420), so I patched BNR. Weeks later, I noticed the issue was still not fixed, but since my case had closed, I created follow up case 06322891. I was told to run an active full. I explained we do not have space to do that. But I ended up getting approval from the client to remove one asset from the chain entirely and start a whole new backup chain of it. This one asset was so large that starting it from scratch allowed us to get backup job going again as a whole, but I noticed another asset doing the same thing, i.e. not retentioning. Support told me it appeared to be the same failure of the synthetic full backup and again recommended an active full. I explained the situation again and also that I cannot just go back to the client and ask for permission to severely affect their overall security posture by wiping their local backup job for a production asset, especially when I just did managed to get permission for this on another asset. I laid out a plan to Veeam support and asked for their thoughts. For some reason, they instead chose to focus on the fact that Veeam was installed on a Hyper-V host. Then I got a "worth a try" answer which immediately ticked me off. It came across very much like Veeam doesn't know, Veeam doesn't care, and Veeam will instead find anything else to blame to delay a resolution. I asked for the case to be escalated.

The next support rep didn't even respond to my plan. He did, however, explain how to ignore the "Missing Updates" bug (have I mentioned how plagued with bugs V12 is??) and then immediately proceeded to reiterate the previous engineers words about Hyper-V and and recommend I do a backup server migration! This is without even acknowledging that I merely have the Hyper-V role enabled, that I'm not protected any assets on the backup server (of which there are none), and that the backup job in question is Windows Agent Backup. In fact, the target repo is a simple Windows server packed with managed disks that happens to live in Azure. The symptom is about as remote to Hyper-V as one can get. And yet, that is what they chose to focus on and recommend a backup server migration!

And speaking of that bullet point, it's subject to interpretation. I think it means installing Veeam on a Hyper-V host you are protecting. But I can see how Veeam Support might interpret it differently, i.e. by the "letter of the law". However, they have yet to draw a correlation between any log entry regarding the failing sythetic merge and Hyper-V.

I called support a short while ago and explained the situation. Thankfully someone saw my point of view that the requirement probably was for production hosts that Veeam is protecting. He didn't think this symptom had anything to do with Hyper-V. But he wanted to play Veeam's one and only trump card, and I agreed to do it. I have since remove the Hyper-V role and have rebooted the server. When we find out tomorrow that this had no impact on the symptom whatsoever, I will be livid. Why? Because Veeam will have once again wasted my time and theirs and will be no closer to the root cause.

[TitaniumCoder477]

UPDATE: Removal of the Hyper-V had zero affect on the symptom, as I suspected. So Veeam support effectively wasted my time and theirs and is no closer to resolving the issue.