Feature Request: Proxmox

[CHR15M]

With the recent VMWare licensing consolidation and Broadcom moving to a subscription/core model, it has caused many SMBs to explore new options for a hypervisor. (such a shame.. pouring one out for my years of VMWare experience.) XCP-NG and ProxMox keep coming up in various communities. I personally have tested both and they perform really well. My only issue with them is that currently there is no native Veeam integration for the two platforms. Are there any plans for Veeam to support XCP-NG and/or ProxMox?

[Mildur]

Hello Chris

Welcome to the forum. I moved your request to the existing topic.
As of now, there's no information to provide. We'll keep a close eye on the market.

Best,
Fabian

[Gostev]

@CHR15M what is your general conclusion about Proxmox vs. XCP-NG (beyond that they both will do the job for SMB)?

[f.neumann]

As current broadcom-changes will probably let vSphere die for SMB and most of their partners...
+1 for Proxmox Support, we really don't want to miss Veeam!

[A.J.]

+1 for Proxmox Support

my company runs ~2000 VMs on vmware in our datacenters and protect the data with Veeam. Know with the latest decisions from broadcom we have to find a new solution for our customers in the enterprise environment.
Since Veeam has served us successfully as a service provider for many many years, we are of course reluctant to do without it. Hence our vote for Proxmox. We will also check the hypervisors that are already supported, but we are now of course unsettled by the behavior of VMWARE/Broadcom and don't want to get into such a dead end again....

[Gostev]

We're researching and doing some prototyping around Proxmox to see what's possible there as far as backup goes.

It would be helpful if we replace future +1s in this thread with information on:

1. Proxmox infrastructure size.

2. Virtual disk/storage type you're using (QCOW2 vs. RAW vs. ZVOL vs. other).

3. Any other infrastructure info that might be directly relevant to Proxmox VM interaction/backup/restore. Not knowing Proxmox yet I simply don't know what to ask, but for example with Hyper-V you can use standalone hosts, failover clusters or SCVMM - looking for this kind of infrastructure info/peculiarities.

[MelanieTanaka]

@Gostev here's some info for our use case for proxmox. MSP type company serving a large amount of SMB customers, 100% of which are all currently running vmware. We deploy vmware on every server just as a best practice to abstract the hardware away and give us flexibility in migrations and hardware refreshes. Even for environments where 1 server = 1 VM.

Usually the lowest tier of licensing just to enable the use of Veeam, for 1 to 3 hosts max, between 1 to 20 VMs each.

We also have some large enterprise customers who are less likely to switch but as our experience with proxmox and SMB space usage grows we will likely start proposing it instead of vmware during the consulting/sales phases, or when hardware/software refreshes/migrations happen. So within the next 5-7 years we will be shifting basically everyone. VMware is firing us as a partner so we're not particularly motivated to continue giving them business.

Estimate about 3500 hosts and 20k VMs across all of them for SMB customers only, if including enterprise maybe 50% more than that.

Our sales teams are going crazy right now trying to figure things out but estimates are that over the next 2-3 years most SMB customers will need to transition to something else as their licenses or support contracts expire, while a substantial number will actually need to take action this year (2024). Enterprise sized customers likely won't feel a need to change immediately but many will still want to save money and do so eventually.

For more technical specs on proxmox use these are the use cases being tested by our R&D:


scenario 1 (very common): single host, 1 - 5 VMs, local storage backed by zfs. dataset/zvol created automatically by proxmox for each virtual disk.
scenario 2 (very common): single host, shared storage accessed via NFS. qcow2 disks.
scenario 3 (less common): 2 - 3 hosts, ~20 VMs, clustered. NFS shared storage, qcow2 disks.
scenario 4 (rare/unlikely at current levels of testing): multiple hosts, clustered, ceph storage, qcow2 disks.

The use of ceph may get more relevant for enterprise customers who make use of vsan, as it's a replacement of that feature.

Some general info about proxmox that may be relevant for veeam integration, specifically how it differs from vmware vsphere:

1) there is no equivalent to vcenter to be deployed. A standalone host is managed the same way as a cluster. In a cluster each host's web UI (you can connect to any of them) manages every other host. What many people do is deploy a reverse proxy so something like proxmox.domain.com points to each host in a round robin type of situation. So many people may be inputting this fqdn into veeam as the management address rather than a single host. It's all just a big API so this should be fine as long as you're expecting it.

2) the VM configs are stored separately from the virtual disks and other VM files. so in a proxmox cluster your equivalent of a .vmx is on dedicated clustered storage accessible to all hosts, while your disk files are stored wherever you decided to put them.

3) proxmox has its own built-in backup system that works very well for simple backup& restore of whole VMs. It just doesn't have anything other than that which is where veeam needs to come in. It may be possible to leverage these backups to do the things that veeam does instead of reinventing the wheel for the backup layer. Worth checking out.

4) in addition to the built in backups, they also offer a backup server which is used as a backup destination, that ties in to the built in backups but offers better retention management and data integrity checks etc. Still none of the features veeam offers though. This may actually be the component you want to compete with. Drop in replacement for proxmox backup server, where the backups are sent to it, and then using those backup files veeam can work its magic for file level recovery, active directory/exchange/sql and all the good stuff.

[fga]

There is also a Scenario 5 that might be quite common.
Multiple hosts, clustered, LVM Storage on top of a shared FC LUN, RAW Datasets per Virtual disk.
This is imho the only setup that allows a Shared Storage via FibreChannel that does not rely on async replication.
Also this setup might be the closest you get to an actual vmware cluster with a FC SAN Backend. Its basically the same, just replace esx with proxmox and vmfs with lvm.

Also +1 from my side for Proxmox.

[MoritzG-Seidemann]

Proxmox vs. Hyper-V also became an idea for us for future virtualization if VMware fully cancels the free option (which is usually sufficient for most of our clients).
So support for Proxmox would be a nice feature for both home lab users to become more familiar with the product family of Veeam and in addition veeam has a stand-out feature as a one-for-(nearly) all solution.

Might be a bit of simplifying and wish-thinking from me

[saintdle]

How are you currently protecting your clients on ESXi free? As there's no APIs for backup using Veeam or another software.

[Gostev]

APIs are only needed for host-based backups. Agent-based backups (with a backup agent inside of each VM) do not require them.

[m.novelli]

I'm following this thread

[chris.childerhose]

Posting to follow the thread and see where this goes with Proxmox.

[galbitz]

We have about 200 server vms in our env, mostly small sites of 2-4 host clusters at a few offices. We utilize veeam backup,copy, and replication jobs to a dr site. We just renewed VMware for 3 years but will not renew it again.

Looking at proxmox as a possible alternative. Cloud is always on the table as well but the opex hasnt made sense for us to date despite what you read online =)

[topbanana]

Long time lurker, first time poster. We've got a very significant VM estate, split between VMware and Hyper-V and using Veeam with VMware for about two thirds of it. Hyper-V was via an aquisition with an existing backup solution with a slow transition.

With the Broadcom changes, I'll wager most are in my position of hoping it all works out, but planning around it not - and I've asked my team to evaluate options since the 22nd of December. As we're not hyperconverged, it whittled down to expanding Hyper-V, or looking at XCP-NG or Proxmox - with the latter favoured after preliminary use case testing. Hyper-V works, but it's got many problems and there's little chance of those problems being addressed.

However, whilst I know Proxmox has it's own backup system, which works to some extent, I've slept very well for over a decade with Veeam doing hypervisor level backups and I'm very reluctant to move away from that level of reassurance. I think that if Veeam did offer support for Proxmox then I don't think it would be an understatement to say that it would move Proxmox into another tier overnight, and make it somewhat of a no-brainer to adopt in a number of scenarios.

Even if it does "work out" with Broadcom, I think the days of relying on VMware remaining commercially viable have now passed and other options will need to be explored.

[PTide]

Hi,

I've slept very well for over a decade with Veeam doing hypervisor level backups and I'm very reluctant to move away from that level of reassurance. I think that if Veeam did offer support for Proxmox then I don't think it would be an understatement to say that it would move Proxmox into another tier overnight, and make it somewhat of a no-brainer to adopt in a number of scenarios.

I am glad to hear that! However, I would like to ask you to tell us more details about your experience with Proxmox.

Which features of Veeam are especially valuable for you and that are missing from Proxmox backup?

Thanks!

[nathano]

We have a small setup relatively speaking but also concerned with the BC drama, having been Broadcommed once before with Symantec I'm not keen doing that again. For our small cluster I am looking at HyperV as would be covered with the windows data centre licensing. Also keen to explore XCP-NG and also Proxmox but my non negotiable at this stage is hypervisor support by Veeam, so that limits things. Don't want to move to agent based backups.

Like @topbanana I sleep well knowing Veeam is in place. we also use tape (on the cloud connect side of things) and have never had any problems restoring from disk or tape as requested by clients. When it comes to infrastrucutre replacement, DC moves etc.. Veeam just makes life easy, don't want to go away from it.

As a very small partner in the eyes of VMware I don't know if we will get any invitation to join any program and would proably have to do our rental/CSP thing through another partner if we need to continue with VMware.

Very keen to see where Veeam goes here, not sure if Proxmox being KVM based and XCP being Zen based means proxmox would be the preference?? (I think I have that right )

[bstoyanov]

It's great to see VEEAM doing some research and looking up to support KVM as we see more and more people migrating away from VMware into pure KVM, there's also plenty of open source orchestration tools to come handy as a vCenter replacement, for example Apache CloudStack already has support for VEEAM latest releases. Btw, it also covers all together orchestration/offering/usage all together on for both hypervisor and backup components side so it's much easier to operate.

[ITP-Stan]

1. Proxmox infrastructure size.

2. Virtual disk/storage type you're using (QCOW2 vs. RAW vs. ZVOL vs. other).

3. Any other infrastructure info that might be directly relevant to Proxmox VM interaction/backup/restore. Not knowing Proxmox yet I simply don't know what to ask, but for example with Hyper-V you can use standalone hosts, failover clusters or SCVMM - looking for this kind of infrastructure info/peculiarities.

I'm very worried about the future of vSphere for the SMB market. So looking at Proxmox as well.
The only thing holding me back-up is Veeam support.

I've only used Proxmox in my homelab, not yet in production or at customers.
But I'm very pleased with the management GUI and the stability.
The only strange thing, is the fact that VM's have an ID starting at 100 and not really a name, but you can add a description and labels so not an issue really.

And besides VM's you can also add LXC containers. LXC containers sit somewhere between Docker containers and actual VM's.
Proxmox has the ability to create clusters and to use CEPH for shared storage.
They also have Software Defined Networking options, useful with clusters I guess.
Also a popular option is the ability to create a ZFS (raid-like) storage pool to use as storage for VM's and containers.

1) Single standalone host, have not tried a cluster with ceph shared storage yet.

2) I have just a single SSD in the homelab PC, configured as LVM-thin to store the data for VM's and containers.
I've added a disk to a VM as a "VirtIO SCSI single" disk on the LVM-thin storage.
This adds a virtual LVM type disk. Using fdisk on the hypervisor host actually shows me this virtual disk and the partition layout.

3) Proxmox has a built-in free-to-use basic back-up utility, called vzdump.
You can maybe look at this tool to get some insight. The tool allows to back-up a running VM, for example to an SMB share.

Happy to share whatever more you want.

[ITP-Stan]

Which features of Veeam are especially valuable for you and that are missing from Proxmox backup?

I have no experience with the full fledged back-up software from Proxmox, only the basic vzdump utility.

What I value about Veeam and what is needed:
- File Level Restore
- Application Aware Processing and Back-up
- Application Item Restore
- Back-up copy, offload to cloud storage
- ReFS/XFS space savings with block cloning

[vbussiro]

@PTide : Not yet an user of ProxMox but certainly going in after Vmware. And my answer on ProxMox config will be : I followed Veeam best practice : )

[olivierlambert]

We have a small setup relatively speaking but also concerned with the BC drama, having been Broadcommed once before with Symantec I'm not keen doing that again. For our small cluster I am looking at HyperV as would be covered with the windows data centre licensing. Also keen to explore XCP-NG and also Proxmox but my non negotiable at this stage is hypervisor support by Veeam, so that limits things. Don't want to move to agent based backups.

Like @topbanana I sleep well knowing Veeam is in place. we also use tape (on the cloud connect side of things) and have never had any problems restoring from disk or tape as requested by clients. When it comes to infrastrucutre replacement, DC moves etc.. Veeam just makes life easy, don't want to go away from it.

As a very small partner in the eyes of VMware I don't know if we will get any invitation to join any program and would proably have to do our rental/CSP thing through another partner if we need to continue with VMware.

Very keen to see where Veeam goes here, not sure if Proxmox being KVM based and XCP being Zen based means proxmox would be the preference?? (I think I have that right )

No, you don't, because the hypervisor doesn't matter. What matters is the toolstack you have on top. You could have a Proxmox-like solution with Xen and XCP-ng with KVM, that wouldn't be a fundamental change from a backup/API consumer perspective.

On the security aspects, there's big different between Xen and KVM though (security wise, because KVM+virtio is opening many holes with DMA, reducing the memory isolation between the guest and the host, but it's another topic).

[je1000]

Veeam is uniquely positioned to chose a vSphere successor, specifically in the SME space, since a lot of MSP’s and SME customers are using veeam and they love it, and veeam is a huge part of the their virtualization strategy, they will choose a hypervisor that veeam supports. In the same time if Veeam doesn’t include more hypervisor options soon enough it will force a lot companies to give up on-prem infrastructure as the added VMware tax will make it a none viable option and make the cloud more financially attractive this will result in a shrinking market for Veeam.

From my research it seems that XCP-NG is the closes to vSphere both in reliability, scalability and support options. With Veeam’s support it will make Broadcom regret buying VMware. I personally hope that Veeam moves fast to announce its intention to fully support XCP-NG

[SkyDiver79]

We're researching and doing some prototyping around Proxmox to see what's possible there as far as backup goes.

It would be helpful if we replace future +1s in this thread with information on:

1. Proxmox infrastructure size.

I have many Customers with Proxmox. One small Customer with 15 VMs and a 2 Node Cluster with ZFS Replication. (Using ZVol)
One Customer with an iSCSI shared Storage and 70 VMs. (Using LVM)
All other Customer with up to 700VM use PVE Cluster with Ceph. (3-11 Node Cluster) . (Using RBD)

2. Virtual disk/storage type you're using (QCOW2 vs. RAW vs. ZVOL vs. other).

On all Setups RAW. The big benefit from proxmox to vSphere are using Block Devices instead of Files on a Filesystem. (Except vSAN)
And on Benchmarks i have better Latencies and Throughput then ESXi with the same Server.

3. Any other infrastructure info that might be directly relevant to Proxmox VM interaction/backup/restore. Not knowing Proxmox yet I simply don't know what to ask, but for example with Hyper-V you can use standalone hosts, failover clusters or SCVMM - looking for this kind of infrastructure info/peculiarities.

On all Clusters we use PBS (Proxmox Backup Server). This Tool provides us to make incremental Snapshot Backups on all Storage Types, also on LVM without Snapshot Support.
I Think, the fastest integration from Veeam perspective is, use PBS as Backend. Scheduling and Guest Interaction a the Veeam Job and the trigger per API the PBS incremental Backup.
For Application Restores, we can mount the virtual Disks from PBS per API and mount for Application Restores on the Backupserver.

We don't need a hardened repository either, the PBS has an Option for Sync, that Sync Process is an Pull from Secondary PBS. This Server can be complete deny all incomming Connections on the Firewall.
The only Pain-Point, that i see ist Backup Copy to the Cloud, like S3 or Blob.

[SkyDiver79]

scenario 1 (very common): single host, 1 - 5 VMs, local storage backed by zfs. dataset/zvol created automatically by proxmox for each virtual disk.
scenario 2 (very common): single host, shared storage accessed via NFS. qcow2 disks.
scenario 3 (less common): 2 - 3 hosts, ~20 VMs, clustered. NFS shared storage, qcow2 disks.
scenario 4 (rare/unlikely at current levels of testing): multiple hosts, clustered, ceph storage, qcow2 disks.

For NFS is qcow2 OK, but dont Use qcow2 on Ceph. To store qcow2 Images you must activate CephFS and an virtual Disk operate not so perfect on a Object Storage.
In Proxmox is the default to use RBD (Rados Block Device) like ZFS zvol. With RBD (in classic storage speak) every VM Disk is an initiator and every OSD (ceph Disk/Deamon) are a target. That Scale Out performs very good.

[cgsm]

I am a manager of a 2-node vSphere cluster for an SMB using StarWind VSAN for the storage layer. Our VMware subscription is up in about 1 year. We are actively looking at Hyper-V and Proxmox as alternatives.

We have experimented with Hyper-V previously, and have a bunch of Windows servers, so I don't think it would be too difficult for us to transition to Hyper-V. However, we do like the developments going on with Proxmox, as well as not giving MSFT any more money! I think Proxmox will eventually be the way we go, as soon as we can figure out the storage layer, support, and backups.

[fbaligant]

1. Proxmox infrastructure size.

~20 clusters ranging from 2 nodes to 5 nodes per cluster.

Before the Great Move to the Cloud, clusters with ~30 nodes where common and worked just fine.

2. Virtual disk/storage type you're using (QCOW2 vs. RAW vs. ZVOL vs. other).

3 types:
- 1 node "cluster": RAW on top of LVM on top of LUKS
- 2 nodes cluster: RAW on top of LVM on top of DRBD on top of LUKS
- 3+ nodes cluster: RAW on top of NFS on top of HA-NFS cluster on top of DRBD on top of LUKS

There is other storage options (zfs, Ceph RBD, etc...) but mostly not used in our deployments.

3. Any other infrastructure info that might be directly relevant to Proxmox VM interaction/backup/restore. Not knowing Proxmox yet I simply don't know what to ask, but for example with Hyper-V you can use standalone hosts, failover clusters or SCVMM - looking for this kind of infrastructure info/peculiarities.

Design principles behind the built-in Proxmox backups are described here:

https://git.proxmox.com/?p=pve-qemu.git ... backup.txt
https://git.proxmox.com/?p=pve-qemu.git ... a_spec.txt

Consistency is handled using the QEMU Guest Agent (Qemu-GA) so it should be similar to your work supporting Red Hat Virtualization.

There is also an official add-on called Proxmox Backup Server that offers more advanced functions (deduplication, incremental, object level recovery, encryption, replication). Subscription is strongly recommended for production usage but source code is available under AGPL v3.

Veeam B&R for Proxmox could bring advanced features like more complex gfs, instant restore, storage tiering, restore to the cloud.

Best regards,
François

[fga]

I have no experience with the full fledged back-up software from Proxmox, only the basic vzdump utility.

What I value about Veeam and what is needed:
- File Level Restore
- Application Aware Processing and Back-up
- Application Item Restore
- Back-up copy, offload to cloud storage
- ReFS/XFS space savings with block cloning

So here is the thing. Proxmox already ticks 3 boxes here with their Proxmox Backup Server (PBS). That's offloading to somewhere else, space savings by block deduplication on a storage level and file level restore via hypervisor webinterface (for NTFS: Non dedup only).
Application Level Restore is completly not possible. no integration whatsoever besides basic file level restore.
Also Application Aware Processing is not possible, they don't have any dynamic loading agent that speaks with an installed application or VSS in the VM. At least not that i am aware of.
And the logic of Backups is somewhat back to front on Proxmox.
PBS is basically just a datastore with an api and dedup built in. The backup job itself is configured and ran from the hypervisor hosts.
That job takes care of everything and just throws data at the PBS to ingest.
So imho to make veeam work with proxmox there has to be development done on both sides.

[SkyDiver79]

I have migrate any HyperV VMs to Proxmox with Veeam.
It’s very easy, you need a small ESXi (VM is OK) and trigger a instant restore. Then I mount the NFS to my Proxmox Host, attach the vmdk to the VM and boot.
With this way, you can migrate big VMs with less downtime.
Veeam can help us and allow a instant Recovery direct to Proxmox, then we don’t need a dummy ESXi.

[SkyDiver79]

Filerestore Only way is to go via shell.
Application Level Restore is completly not possible. no integration whatsoever besides basic file level restore (see above).
Also Application Aware Processing is not possible, they don't have any dynamic loading agent that speaks with an installed application or VSS in the VM. At least not that i am aware of.

You can simply file Restores over the PVE GUI.
I like this concept, backup and restore tasks including the Hypervisor GUi. The PBS are only a good Datastore.
My VMs are all consistent through the qemu VSS integration.
For AD and Database Server I use Veeam Agent.