Broadcom / VMware debacle

[tinto1970]

> And companies born from russian devs?

if you mean Veeam, it's no longer a russian corporation.

>That is quite a statement... and can be said about any system, even USA ones, not open to public scrutiny (closed source).

USA it's not a dictatorship.

[OmiFreak]

HV is not an alternative for me that I would like to start over with now. Microsoft is no longer interested in on-premises products, there is no significant further development of HV.
Proxmox also feels like it has been treading water for a long time, maybe the problems with VMware and encouragement from people desperately looking for a VMware alternative could make a difference.
If Oracle is the best alternative for software that has become too expensive, it should be clear how serious the situation is.

Bernd

[JamesNT]

I have a few responses to this thread. A thread I have found interesting. Like many of you here, I've been working in the SMB space for decades.

First, I would like to point out that I miss "The Word from Gostev" on the Sunday emails. I always enjoyed reading those and now they are gone. And that sucks.

Second, I agree with @Moopere: there is a lot of hating on HYPER-V that is completely unjustified. Hyper-V is a mature product that can go toe-to-toe with VMWare and the others out there. Hyper-V is the ONLY hypervisor I support. I steer clear of VMWare and, especially, all FOSS hypervisors.

* FOSS has a bad habit of making changes or abandoning things with little notice. Remember CENTOS? Remember when Linux changed its printer model? And I'm certain Veeam has had so much fun chasing after all the different Linux API changes over the years for the Linux stuff they do support.
* FOSS often lacks enterprise support. If they do offer enterprise support, by the time you pay for it, you're better off just using HYPER-V.

As for VMWare, having a new owner every few years doesn't work for me. First it was EMC, then Dell, now Broadcom. Further, those changes in ownership have resulted in a less stable product. From Wikipedia: In January 2016, in anticipation of Dell's acquisition of EMC, VMware announced a restructuring to reduce about 800 positions, and some executives resigned. The entire development team behind VMware Workstation and Fusion was disbanded and all US developers were immediately fired.

Changing teams up like that is disruptive. I'm not only IT, but also a software developer myself, and I've seen first-hand the damage that just changing teams up like that can do.

Further, I have never been able to justify the additional cost of VMWare to a client unless that client must run an OS not supported by Hyper-V. Which is rare for me.

Lastly, I defend HYPER-V because it is cost effective, more so than any other hypervisor out there.
* Easy to set up and maintain.
* For clients that want cheap redundancy, HYPER-V replica is hard to beat. Just get a couple of servers from XByte, handle your licensing (which you have to do either way) and you have redundancy in the event of server failure.
* For clients that want to go all the way, a Hyper-V cluster is far cheaper to purchase and maintain than a VMWare cluster is. I maintain currently 6 Hyper-V clusters and not a single issue out of any of them.

I am against Veeam spending time (READ: MONEY$$$$) developing backup for any other Hypervisor because of the cost of both initial development and ongoing maintenance.

For those of you hating on Hyper-V, I have noticed that many who hate on it are using their experience from the Windows Server 2008 days or even the Microsoft Virtual Server 2005 days to justify that hate. Whether this applies to any of you is unknown. However, I would nonetheless encourage you to setup a new 2019/2022 Hyper-V server and give it a run.

As for patching, using group policy to set your guests to patch on Friday morning at 3am and your hosts to patch Saturday morning at 3am. Or whatever you need to do to separate them.

James

[nmdange]

HV is not an alternative for me that I would like to start over with now. Microsoft is no longer interested in on-premises products, there is no significant further development of HV.
Proxmox also feels like it has been treading water for a long time, maybe the problems with VMware and encouragement from people desperately looking for a VMware alternative could make a difference.
If Oracle is the best alternative for software that has become too expensive, it should be clear how serious the situation is.

Bernd

If that were true there wouldn't be any new features coming in the next release of Windows Server, especially the ones that are Hyper-V specific (that are currently Azure Stack HCI-only) https://redmondmag.com/articles/2023/12 ... vnext.aspx

[Gostev]

there is a lot of hating on HYPER-V that is completely unjustified. Hyper-V is a mature product that can go toe-to-toe with VMWare and the others out there. Hyper-V is the ONLY hypervisor I support.

Just curious James, are you really not affected by any of these issues?

[tyler.jurgens]

For those of you hating on Hyper-V, I have noticed that many who hate on it are using their experience from the Windows Server 2008 days or even the Microsoft Virtual Server 2005 days to justify that hate. Whether this applies to any of you is unknown. However, I would nonetheless encourage you to setup a new 2019/2022 Hyper-V server and give it a run.

To be fair, I only hate Hyper-V because I've used it in every one of its iterations, from long ago - 2003, until the latest version 2022. Including cluster deployments and iterations such as Azure Pack and Azure Stack.

It does its job from a very basic perspective, running a few VMs. It is not feature complete against VMware by any means, but it can work.

[JamesNT]

@Gostev,
No, I have not had those issues. And I have a client with an Exchange 2019 server on prem and I have several SQL Servers not to mention one law firm with a Hyper-V cluster in the main office and servers in 5 branch offices all doing DFS and SQL Server Replication. I didn't even know about those issues until just now.

James

[jcross@hh]

I've been fortunate enough to use XenServer (but not XCP-NG specifically yet), VMWare since VI 3.5 through 7.x, and Hyper-V in SMB and enterprise environments. I will grant that I was not a fan of Hyper-V in the 2008 and older days, but I find it more than sufficient in 2012R2 and newer. I have never spent any time with Nutanix, so I don't have any comments to make about it. I am a fan of the 3 HV's I listed as I've found them all to be stable and do their job well. I have run Hyper-V across multiple clusters with hundreds of VM's with great success, and as someone else has mentioned here, Hyper-V is what underpins Azure. It can handle quite a bit. With Hyper-V, I absolutely had to have good PowerShell skills, so that could be a barrier to adoption. Of the 3 I've listed, Veeam only really supports VMWare (which is what we're on now) and Hyper-V. Considering I really like Veeam, and I'm not liking the direction of VMWare's new owners, it is very possible that we'll end up on Hyper-V. We're still kicking the tires on some of the other FOSS HV's, but if it comes down to level of support, again Hyper-V might be the only option. Just my $.02.

[nitramd]

Another option is TrueNAS Scale sponsored by IXSystems: https://www.truenas.com/truenas-scale/ ; this platform is Hyperconverged storage with KVM ability.

The underlying OS is Debian and TrueNAS uses ZFS as the underlying filesystem I believe.

Here's a link to the virtualization section of their documentation: https://www.truenas.com/docs/scale/scal ... onscreens/

Great posts everyone. I've been watching this merger with a wary eye.

[coolsport00]

Veeam only really supports VMWare (which is what we're on now) and Hyper-V.

This isn't true. They support RHV and Nutanix as well. But, I think most features are only in VMW and HV

It would be nice to see a post of what Sys Architects/Engrs find "wrong" or "bad" about HV vs VMW.

[jcross@hh]

Hi @coolsport00

This isn't true. They support RHV and Nutanix as well. But, I think most features are only in VMW and HV

I apologize for the lack of clarification. What I was stating was of the 3 HV's I was talking about (Xen, VMWare, and Hyper-V), Veeam supports VMWare and Hyper-V. I hope that clears up my comment. Please let me know if there are any questions.

[jcross@hh]

Another option is TrueNAS Scale sponsored by IXSystems: https://www.truenas.com/truenas-scale/ ; this platform is Hyperconverged storage with KVM ability.

The underlying OS is Debian and TrueNAS uses ZFS as the underlying filesystem I believe.

Here's a link to the virtualization section of their documentation: https://www.truenas.com/docs/scale/scal ... onscreens/

Great posts everyone. I've been watching this merger with a wary eye.

TrueNAS Scale is one that we're looking at as well, but it would mean a move away from Veeam (I'm not a huge fan of installing agents within the VM's for backup) which is why it's not higher on our list. If Veeam was able to support it, that may change its ranking for us.

[JamesNT]

For high availability and storage in general I stay away from TrueNAS. They are expensive and I've never had any good luck with TrueNAS regarding stability.

All of my hyper-v clusters run Dell servers with Dell ME5 storage arrays. The Dell PowerVault series just cannot be beat. I know MD3200s still in production and we are talking almost 15 years later.

James

[jronnblom]

If that were true there wouldn't be any new features coming in the next release of Windows Server, especially the ones that are Hyper-V specific (that are currently Azure Stack HCI-only) https://redmondmag.com/articles/2023/12 ... vnext.aspx

Hotpatching is great news. However if it will require Azure Arc then Im not so sure about it. And depending on pricing also.

MS should be ashamed even thinking about separate pricing for it.

-J

[virtualguru]

I work for a public higher-ed institution. When we renewed our VMware ELA in 2023 (prior to the official change of ownership to Broadcom), VMware forced us to convert all our perpetual licensing to subscription licensing. This resulted in an 85% cost increase over our previous ELA. Since then, we've been having internal discussions around what our future might look like, since we're not willing to risk another 85% increase on our next VMware ELA. This is prompting us to do a full analysis of all the systems we run on-prem to see:
1. Are there any we no longer need?
2. Are there any we can move to (or replace with) a SaaS offering?
3. Are there any that are a good fit to move to cloud hosting (and can we afford cloud hosting)?

Additionally, we'll be looking at ways to eliminate or replace all the VMware components currently in our environment. On the data center side, we run vSphere, vSAN, and NSX. Our NSX implementation is very small, and ended up only being used for load balancing. We'll be using some surplus budget to purchase replacement virtual load balancers. Likewise, our vSAN implementation is small (a single four-node cluster), so easy enough to move away from, though we may need to wait until the next hardware refresh cycle on those hosts. That leaves us with finding a different hypervisor.

In addition to data center workloads, we're heavily invested in VDI, and are currently using Horizon, App Volumes, and DEM. In my opinion, finding equivalent replacements for these products will be considerably more difficult. If VDI stays on-prem, I would guess that whatever hypervisor we run the desktops on will also be the hypervisor we run remaining data center workloads on.

I personally have never used the Microsoft hypervisor, so all I have at this point is anecdotal information. However, my personal opinion of Microsoft is that they do a lot, but they're not the best at any of it. Also, Server 2019 is the last version to include hyper-v. Microsoft's future is Azure Stack HCI. While hyper-v was free, Azure Stack is not. This won't factor into our decision, I'm just sharing this info for the benefit of others.

Based on previous experience with anything from Oracle, I'd guess their solution will be out of our price range also. But those F1 cars and sailboats aren't cheap to sponsor.

On a final note, I find it ironic that VMware may have finally persuaded us to move to a hybrid cloud, and now they likely won't be part of it.

[nathano]

@nathano - have you used HV recently?...ever?...or, just looking to begin a switchover now? What's your experience with it? Like you, we don't use many features with VMW. DRS & some affinity/anti-affinity rules, VMotion, SVMotion, and that's pretty much it. Big features for us though. No vSAN, NSX, Aria, SRM, Replication, etc.

used it many years ago on single server sites, really simple stuff. I use it on my PC at home I manage windows servers day to day so would probably be less workload for me overall, famous last words I also manage all the other infrastructure as well so I'm across all the networking and storage etc, so no drama dealing with those teams

Currently setting up a couple of old servers in a cluster to get "current" with it.. like you we do very little use the same features as you. I am setting up AD as part of it as from the reading I've done that is required for the live migration feature to work.

if I was to test one of the others is would be XCP-NG, looks like they have really stepped things up recently and the feature set is pretty comprehensive (again, won't use much of it) and they have commerical support at very reasonaly rates from what I can see. CEO and cofounder of XCP is active on X and is keen to work with Veeam by the looks of things (https://x.com/OlivierLamber12/status/17 ... 39333?s=20)

agree, the discussion is interesting.

[nathano]

Server 2019 is the last version to include hyper-v. Microsoft's future is Azure Stack HCI.

HV is in server 2022 https://learn.microsoft.com/en-us/windo ... ows-server that's what I'm currently spinning up for testing and a refresher on HV

Agree that the direction is Azure stack which I'm not really keen on. Server 2022 has mainstream support to Oct 2026 and extended support to Oct 2031, so hopefully during that time Veeam might announce a new h/v support??

[dasfliege]

@Gostev,
No, I have not had those issues. And I have a client with an Exchange 2019 server on prem and I have several SQL Servers not to mention one law firm with a Hyper-V cluster in the main office and servers in 5 branch offices all doing DFS and SQL Server Replication. I didn't even know about those issues until just now.

James

I think your were just a very lucky guy then. We run our own datacenter on Hyper-V since more then 6 years now. Quite a few hosts and >700 VMs. We started with HV2012 and are on 2019 atm. Years back, we had to replace all the NICs in our HPE proliant hosts two times because of incompatiblities with Hyper-V. With the actual Gen10 hosts, we have problems again as they crash out of nothing because of VMQ/RSS failures. Already had cases with Microsoft premier support for months. No solution so far. This is not the only issue we had. I guess we basically run into every single issue that occured with Hyper-V so far. The RCT I/O problem that Gostev was referencing was known at MS for almost 3 years! As far as i know, it was easily reproducable in pretty much every installation with a little higher load and it took them 3 years to fix it!

Just wanted to add my 2 cents to that discussion and follow it in the future, as we are designing our new datacenter right now and unfortunately had to move away from our plans to build it on VMWare. So we may go with Hyper-V again if there is no other bulletproof alternative. We're also discussing an hybrid solution with Hyper-V for the main workload and a dedicated proxmox cluster for Linux VMs that don't need a MS license or Veeam Backup.

[coolsport00]

@nathano - thanks mate!

[hobbit]

How are you finding it so far? What are you using to backup the VMs?

From what I've seen so far, I really like it (Proxmox). With Ceph integrated, it can basically do what our VMware/VSAN cluster does, and more. On a personal view it feels more natural, as it's closer to managing the standard linux boxes I'm used to. Added benefit is that it's using the same hypervisor (KVM) I'm familiar with from using on my desktop anyway.

As I said, the elephant in the room is the backups. While for the current workload the integrated backup feature was sufficient, we'll need something better if we consider the full transition. Maybe the Proxmox Backup Server product, maybe additional Veeam Agent Backups inside the guests, still have to explore these options. The optimal solution for us would be a Veeam product that works the same as we are used to for VMware hypervisors.

[hobbit]

..
If Proxmox has potential VM corruption issues with backup..hmm...then not sure how that can even be a possibility for anyone now, other than lab'ing. Maybe engineers for them can get that taken care of soon. May still at least give it a glance. @hobbit - have you had any VM corruption issues in your testing?
..

No, haven't experienced this yet. Are there more details under which circumstances this could be happening? Basically, hot/live backups work the same way as in the Vmware case - freeze, snapshot, thaw. Afterwards you got a crash-consistent backup, not much different from how Veeam does it in our VMware-setup.

[dloseke]

I don't think VMware is going away any time soon, or at all. Especially for enterprises. But as a MSP solidly in the SMB space, we're going to have to start having conversations about the cost of renewals/conversions with VMware, and for some of my clients, that may be a large pill to swallow. I'm going to have to start looking at alternatives like Proxmox and XCP-NG, but lack of Veeam support for these hypervisors has stopped me from doing so. If Veeam were to be compatible with these, it would put a lot of energy behind my considering of these for my customers. I'd really prefer to avoid Hyper-V if possible, but that's my other low-cost alternative.

[dloseke]

It would be nice to see a post of what Sys Architects/Engrs find "wrong" or "bad" about HV vs VMW.

I find the setup, especially failover clustering not as clear or intuitive to setup in Hyper-V. Now granted, as noted by others, most of my experience with Hyper-V was from 2008 - 2016, but some it still feels kind hacked together at times. I also find that it's really easy to store VM's....wrong? Inconsistently? Almost every Hyper-V environment I walk into has some VM's stored in one location, others in another, some VM's using a specific folder, some using a subfolder under that folder, etc. I'm also concerned about Microsoft moving away from premise computing as they focus on Azure, or focus on rolling out on-premise software with the intent of funneling you to Azure. It's not bad, but it's also not ideal. VMware historically has just done a really good job of standardizing deployments, making things pretty clear overall, and just baking a lot of reliability into their products. Not to say that there haven't been issues (such as ESXI 7 update 2). I'm also sure that if I spent more time with Hyper-V, I'd be more comfortable with it, but I'm still not sure it's the right product going forward.

[jcross@hh]

I also find that it's really easy to store VM's....wrong? Inconsistently? Almost every Hyper-V environment I walk into has some VM's stored in one location, others in another, some VM's using a specific folder, some using a subfolder under that folder, etc.

I experienced the same which is why I ended up writing scripts to create VM's with proper placement . Once we leveraged scripts for automation, these types of issues went away. We scripted the entire process from setting up hosts, creating clusters, storage, networking, creating VM's, etc. to minimize these types of scenarios and errors that arise from manually using the GUI to set up each one. To me, that's a good direction regardless of the HV being used though. Again, just my $.02.

[infused]

XCP is new to me but I would vote for this too. any alternative for VMware would be welcome!

It's buggy and not production ready. imo, this would be wasted resource.

[jcross@hh]

It's buggy and not production ready. imo, this would be wasted resource.

Hi @infused, would you care to elaborate? I used the previous Citrix Xen Server (before they blew things up, and what XCP-NG is based off of) and found it to be very reliable. Please provide details.

[infused]

I was going off a couple of forum posts on this current VMWare issue and where to go from here. XCP-NG had a lot of praise, but the comments from people whom had tried to get this going in to production said it was very difficult with a lot of issues. I'll try and dig the sites out again. I realise this is 3rd hand information, but there was enough of it there to have me seriously questioning it.

[edh]

Sad news today... VMware asking for 3500 cores at 350$ per core equal 1.2M$ to be partner. Checking CloudStack with Hyper-V Cluster Integration and Veeam Pluggin (Outdated)

https://docs.cloudstack.apache.org/en/l ... lugin.html

[m.novelli]

Today 1.2M$ , what was previous request? Just to put in perspective their greed

Marco

[tyler.jurgens]

Still unknown is what happens to smaller VCSPs who don't meet that 3500 core minimum.

Previously they billed points per RAM utilization. Now they've flipped it to billing on (host) CPU Cores.