Feature Request: Proxmox

[SkyDiver79]

Which features of Veeam are especially valuable for you and that are missing from Proxmox backup?

Hi,

the PBS only lacks an application-aware control of the backups with log backup and truncate.
And of course also the application restore (SQL, AD, etc.).

At the moment I can live with having agents in the VMs, but it bothers me that I have to boot an ISO or install a dummy ESXi for the restore. Connecting NFS and mounting the VMDK in Proxmox works wonderfully.

[SkyDiver79]

Proxmox is too SMB for us.

Hello Anton,

the problem is, you won't find the admins of the big Proxmox installations here in the forum.
There are more enthusiasts here, I have already had contact with companies that have several Proxmox Ceph clusters in use, with a total of several thousand VMs.

These people would certainly take a look at Veeam if you announced that you support Proxmox, but with luck you will find them in the proxmox forum.
Have you ever spoken to the guys from Proxmox in Vienna? They might be able to give you some approximate figures for customers with subscriptions.
But I also know well-known German companies that have been using Proxmox for years without a subscription. This is actually a gray area that is difficult to calculate and I can't predict whether these customers will then spend money on the backup.

I don't want full support for Proxmox with all Veeam features, the RHEV integration isn't really that good either.
But the control of the backups with the Application Aware features and the Single Item Restore are missing. If you would only bring an addon for the Proxmox PBS, then I would immediately provide every customer with it.

[rohitsb]

Does anybody know what the limitations around scale with Proxmox? For example, maximum of hosts it can handle in a zone/dc or a cluster? Max number of disks in a VM etc? Can it handle separate network (storage, management, guest)?

[SkyDiver79]

Hi,
There is a recommendation not to operate more than 64 nodes per cluster. The limit here is the Corosync service, which does not like latencies.
If you have a fast network for the corosync (Cluster Heartbeat), there is theoretically no limit.

For Ceph you should always have an odd number of nodes, the "decider" can also be a small host without extra disks. This is about the topic of quorum.
For even clusters you should configure a qdevice, as well as quorum.

For the disks per VM, I haven't seen a limit yet, only limitation IDE=4 and SCSI=14, Virtio SCSI controller you can configure as many as you want, so probably the guest OS is the limiting factor.

Of course you should separate networks, as far as the network configuration is concerned, HyperV is more similar than vSphere. Anyone who knows Linux networking will get to grips with it straight away.

For further questions I recommend the Wiki:

https://pve.proxmox.com/wiki/Cluster_Manager

[DRoN]

Hi,


I am glad to hear that! However, I would like to ask you to tell us more details about your experience with Proxmox.

Which features of Veeam are especially valuable for you and that are missing from Proxmox backup?

Thanks!

Application-aware backups are going to be one of the biggest things missing from Proxmox Backup Server for most Veeam users, and I cannot see them implementing it any time soon given the complexity of writing an agent to perform these backups. Most other PBS limitations can be worked around, app aware can't. (Except via agent-based backups in Veeam)

[mconte]

It's great to see VEEAM doing some research and looking up to support KVM as we see more and more people migrating away from VMware into pure KVM, there's also plenty of open source orchestration tools to come handy as a vCenter replacement, for example Apache CloudStack already has support for VEEAM latest releases. Btw, it also covers all together orchestration/offering/usage all together on for both hypervisor and backup components side so it's much easier to operate.

Apache CloudStack could be really interesting for service providers looking for a VMware vCloud alternative.
So Veeam support for CloudStack could be very attractive to service providers.

[Andrew M]

We definitely have interest in Veeam supporting Proxmox.

We currently use vmware+veeam for all of our backup functionality but the writing is on the wall it seems.. At least we have some time with our vSphere 8 perpetual licenses to find a viable solution.

Currently I have a small 3-Node Proxmox+Ceph cluster. The VMs all use VirtIO Block (QEMU image format).

We are planning on setting up a 4-6 Node proxmox+ceph cluster which will house over 100 VMs at least and eventually will plan to replace our VMware environment before it comes to "renewal" time which will be another multi-node HA cluster.

I'm fully down with using my lab for testing if/when the time comes though.
I suppose it will mostly boil down to interfacing with Proxmox and working with the QEMU images / snapshots.

[topbanana]

I am glad to hear that! However, I would like to ask you to tell us more details about your experience with Proxmox.

Which features of Veeam are especially valuable for you and that are missing from Proxmox backup?

At present we're just labbing Proxmox, and I think a lot of people have come in behind me with more experience of PBS. However, what PBS does or does not do is somewhat of a secondary consideration for myself and my organisation.

From my current Veeam experience, the key features on a Day 1 release would be CBT backup, Instant Restore, GFS archiving with S3 shipping and it using the same Windows host server to leverage ReFS block cloning and space saving. The current deployment of Veeam on Windows Server is a known and reliable quantity, and at this I want as many known and reliable quantities as I can get if I'm looking to change hypervisors.

I would prefer the backup solution to be from a third-party, so the risk is spread between multiple providers. When I've had issues with Veeam in the past, a ticket has resolved that satisfactorily and with speed - and the forums are weekly reading, irrespective of me having issues or not. Having the backup system and the hypervisor provided by the same developers worries me from the perspective of making sure that backup system is thoroughly tested, and there is continuous innovation in key areas as a backup product.

[SkyDiver79]

Currently I have a small 3-Node Proxmox+Ceph cluster. The VMs all use VirtIO Block (QEMU image format).

Hello Andrew,

either you have made a mistake or you are using a setup that is not recommended.
If you are using Ceph pools, you can only store your disks in RAW format. To use qemu images (qcow2) you would need to install Ceph Filesystem and it is not recommended to run virtual machines on object storage. The performance suffers significantly.

[Andrew M]

Hi SkyDiver79,

Apologies, You're correct that I was in error assuming it was using QCOW2 (I hastily verified, welp).
Going back, I can see that the selection is grayed out for "Format" and it's selecting "Raw" when targeting the replicated RBD pool (But the controller is VirtIO)

I'm still in the early stages of rolling through proxmox (This cluster is very non-production).

[SkyDiver79]

These are important Informations for Veeam.
VSphere and HyperV are Filesystem agnostic and Proxmox in all default setups Block agnostic.
This makes a little bit harder to use processes from other Hypervisors.

[Andrew M]

Indeed it does add extra complexity. I suppose that returns me to thinking about working with Ceph RBD snapshots and the storage integration stuff.
Of course, that only covers HCI systems.

At least for now we can sit on our *final* perpetual vSphere+vSAN8 licenses.

I suppose in case of emergency I can pull out Hyper-V as a fallback option for backup purposes (So long as it also doesn't go the way of the greedy cloud), the rest I suppose could live as agents.

[SkyDiver79]

I would prefer the backup solution to be from a third-party, so the risk is spread between multiple providers. When I've had issues with Veeam in the past, a ticket has resolved that satisfactorily and with speed - and the forums are weekly reading, irrespective of me having issues or not. Having the backup system and the hypervisor provided by the same developers worries me from the perspective of making sure that backup system is thoroughly tested, and there is continuous innovation in key areas as a backup product.

If you had a background in software development or were a service provider like me who is often tasked with troubleshooting, you would think differently.
Especially if the backup comes from the same developer, there are of course advantages. It is by no means worse tested, but if you find a bug, the bug is fixed much faster if the developers of both solutions are the same people.
As soon as two manufacturers and APIs are on board, troubleshooting always takes longer. Of course Veeam does a good job and I have been using the backup for my customers since 2008 and FastSCP since 2007. but proxmox also does a very good job with the PBS. unfortunately the Proxmox development team lacks the know-how for the many applications and therefore there is no integration for them.

[infused]

We are considering our options now. We were just about to purchase another cluster for vmware when the news hit. It's still on hold. It's basically Hyper-V (Because we already have the licensing under SPLA) or something like Proxmox.

[aderumier]

We're researching and doing some prototyping around Proxmox to see what's possible there as far as backup goes.

It would be helpful if we replace future +1s in this thread with information on:

1. Proxmox infrastructure size.

2. Virtual disk/storage type you're using (QCOW2 vs. RAW vs. ZVOL vs. other).

3. Any other infrastructure info that might be directly relevant to Proxmox VM interaction/backup/restore. Not knowing Proxmox yet I simply don't know what to ask, but for example with Hyper-V you can use standalone hosts, failover clusters or SCVMM - looking for this kind of infrastructure info/peculiarities.

Hi,
I'm a proxmox user and also dev contributor.

I'm running in production around 5000vms with 100 proxmoxhypervisor, mostly with ceph block storage (around 500TB datas).

proxmox itself don't use default qemu backup blockjob (they are still available, same than ovirt, rhev,olvm), but they use their own code (with different output format) to send datas to their proxmox backup server. (This is always incremental backup with dirty bitmap inside qemu for block tracking, like for native qemu backup).
Also, proxmox backup don't use storage snapshots (as proxmox use a lot of different storage, block or file, and block storage snapshot is done through api at storage level). It's using copy before write technique. (if vm need to write a block when backup is running, this block will be sent in priority to backup storage before to be overwriten).
The most difference with ovirt, rhev,olvm is that proxmox don't use libvirt api, but their own api, so it need to be implement. (start/stop backup, manage dirty bitmap, call qemu-agent for consistency).


Restore currently is full vm or file restore (but only download to browser, as they are no agent currently to restore data at source).

I have a lot of customer running also vmware, mostly onprem, with classic windows features need (restore AD object, restore email in a an exchange mailbox,...).
Proxmox backup don't have this feature, and it's really missing for this kind of customer (also current filerestore don't work with windows Refs, as it's unsupported on linux)

You can reach proxmox developper to proxmox mailing list (pve-devel <pve-devel@lists.proxmox.com>) ou through the forum too

[PTide]

Thank you @aderumier, that kind of overview of Proxmox internals from a person with hands on experience is invaluable for us, much appreciated.

Cheers

[Gostev]

Interesting Proxmox I/O performance comparison vs. ESXi on NVMe storage > https://kb.blockbridge.com/technote/pro ... e-nvmetcp/
Although I'm sure a bunch of other test scenarios can be created where ESXi would pull well ahead of Proxmox as well.
But nevertheless, it's good to know that Proxmox is able to put up a serious fight at least in some cases

[SkyDiver79]

I have been a VMware fan since 2006, unfortunately VMware has been disappointing more and more for several years now.
I have also tested with other storages and the results were always better with Proxmox.
The big advantage is that we can save the file system with Proxmox and the first file system is created in the VM. I did a single VM performance test 2 years ago.
With a lot of tuning, you could get 1 million I/O (4k) out of a vSphere VM. With Proxmox without tuning, directly with the standard best practice 1.2Mio I/O.
More serious was the SQL test, vSphere 265k SQL I/Os and Proxmox 660k SQL I/Os. With Proxmox the Intel CPU was limited and ran at 100% .

[m.novelli]

Wow

I’ve started to feel since some years that VMware is slow out of the box even with SSD storage (Dell Server with local storage or Dell PowerVault ME4 / ME5)

Marco

[SkyDiver79]

This is more the Problem from low Cost Storages like Dell ME or HPE MSA

[sf4692]

Hello Guys, interesting forum entry. WE are from Austria so we always like to support locally made Software but never started to go further with Proxmox (apart from lab testing) since Veeam together with sphere was always a "NoBrainer" for us.

Now things start to change and we looked into Proxmox together with a backupsoftware called Storware (which looks and feels a little like Commvault) which supports Proxmox natively. Nevertheless feature wise miles away from Veeam/Vsphere.

So we would really appreciate to see Veeam support Proxmox in future, right now we are testing RAW (super fast) vs. Qcow2 (medium speed - better serviceability in terms of feature richness). Also Veeam would be great to be able to take a backup of a VM from vsphere VM and use Veeam to restore it under Proxmox (maybe with a little manual tweak or automatical virtio driver ingestion) since migration path is not that good.

For Context: We are a medium sized shop (for Austria) with 100 people, 4 vsphere servers (HPE) and 2 HA storages (Pure ISCSI).

[azpets]

I'd love to have Veeam also on Proxmox. I'm betting a shrinked userbase of ESX from Broadcom behavior.

[SkyDiver79]

Hi,

What STOREWARE can do, you can also get with Proxmox PBS, except that a live restore also works and you have a reasonable sync integration in PBS.
All other backup and restore features are not yet available from any manufacturer. Veeam's restore options in particular are unsurpassed.
A restore of an ESXi VM in Proxmox works in theory, I have used this so far to migrate large HyperV VMs to Proxmox. You simply take the NFS share from ESXi instant recovery and attach the VMDK to the new Proxmox VM.

[iansim]

Hi,

We are also willing to help in developping Veeam for Proxmox if the feature request is kept.

We have a small private cloud running on a 10 nodes cluster for about 4 years now (2592Ghz CPU, 10TB RAM and Ceph Hyperconverged 500TB SSD Storage pool) with around 350 nice size vms. We actually run Veeam CloudConnect managed agents which works, but is certainly limited compared to hypervisor-level backups.

We are also capable of running another lab environnement to help testing new features if needed.

Do not hesitate to contact us if needed

Thank you

[johannesr]

Proxmox support from Veeam would be awesome! What I would like to have is a similar experience like we have with VMware. CBT and Instant Restore would be features I´d consider critical. Application awareness and support for object restore (AD and MSX for example) would be needed as well.

[SkyDiver79]

CBT and Live Restore, are already available with the Proxmox Backup Server. App Aware is the big thing, that all People missing.

[PTide]

What kind of AppAware (AAIP) are you looking for the most? Any particular apps/functions in mind?

I get it that in a perfect world all AAIP features should be just ported 1:1, but in case it turns out that we live in a non-prefect world, I would appreciate knowing your priorities (AAIP-wise).

Thanks!

[SkyDiver79]

The most wanted is MS SQL.
Second priority has PostgreSQL and MS AD.
Exchange Servers are going more and more in the Cloud. Oracle Server are often physical an we use Veeam Agent.

[Gostev]

Very logical ^

[fractalenergy]

What kind of AppAware (AAIP) are you looking for the most? Any particular apps/functions in mind?

I get it that in a perfect world all AAIP features should be just ported 1:1, but in case it turns out that we live in a non-prefect world, I would appreciate knowing your priorities (AAIP-wise).

Thanks!

+1 on SQL/AD AAIP features.

Proxmox support from Veeam would be awesome! What I would like to have is a similar experience like we have with VMware. CBT and Instant Restore would be features I´d consider critical. Application awareness and support for object restore (AD and MSX for example) would be needed as well.

I think he mean CDP. But I think restore/replication from other platform for example would be very useful feature as Veeam can do in VMware.