Feature Request: Proxmox

[sf4692]

+1 on MS SQL/AD AAIP features. AS said above we also just ported exchange to cloud/o365.

[liwicom]

Hi all,

I am working at a small IT Systems House. For Virtualization we have used the VMWare - Veeam Combo for years as it is a very reliable combination. In the beginning of last year 2 admins from other companies recommended proxmox. As the Broadcom VMWare deal loomed we started to explore alternative Hypervisors in oder to have a plan B in case something happens with Vmware. After a lot of evaluation of differend hypervisors, we decided to give proxmox a try, tested it inhouse and used it in 2 very small deployments. It worked like a charm. When the Broadcom - VMware drama started to unfold we decided to bet our virtualization future on proxmox and invested time to train on the plattform for desaster recovery and try different backup and recovery scenarios in order to be absolut certain and to know what works and what does not so that we can use it in small and middle deployments to replace vmware. In the mean time we have moved our complete in house infrastructure to Proxmox.

Proxmox is a great plattform, PBS is a good solution. Veeam is better. What is missing? For example back up on rotating Harddrives (RDX), Support for S3 Object Storage (however I have read that this is beeing implemented by a third party this year), Application Aware Backup, USB Drives as rotating hard drives (at least not easy). PBS is very ressource intensive storage wise. The Backups are stored in junks with a max size of 4MB. Some user have reported in forums that the garbage collection takes a day.

If you have several PVE (Proxmox Virtualization Environment) Servers (for example 3) and you configure a backup job, all three PVEs backup to the PBS storage concurrently. So lets say you have 10 PVEs in the cluster that generates a lot of load on the PBS because all 10 will start to backup on the PBS at the same time. A sysadmin told me that he had problems with PBS when using slower SSDs because the VMs which were beeing backup up had been slowed down.

So I think there is plenty of room for a great Backup Solution for Proxmox which is known for reliability and a comprehensive feature set.

[PTide]

Hi and welcome to the community

A sysadmin told me that he had problems with PBS when using slower SSDs because the VMs which were beeing backup up had been slowed down.

I am pretty sure it was happening due to how PVE backup works.:

Code: Select all

The be more efficient, we simply need to avoid unnecessary steps. The
  51 following steps are always required:
  52 
  53 1.) read old data before it gets overwritten
  54 2.) write that data into the backup archive
  55 3.) write new data (VM write)

One of the possible ways to avoid that would be to place snapshot data in some temporary storage.

Since the snapshot storage must be reasonably fast (not too much slower than the production storage), it seems logical to use the same storage where the VMs' disks reside.

However, there is a catch - block storage is not suitable for snapshot placement. It might be a problem for those who don't have any fast file-level storage at their disposal.

Thanks!

[SkyDiver79]

That is normal and the Sizing of PBS and optimized Scheduling are not so easy like Veeam.
But for the Slowdown is a new Feature on the Way. The easiest Option is, Veeam supports Proxmox VE.

[mdws]

Hmm, as far as I understood, the Backup with PBS does not use snapshots, no matter on which storage the VM resides.

[PTide]

First of all, t's just a matter of terminology - no matter what you call it, they still need to intercept writes to the VM's disk if they want to have any sort of consistency. So they chose to intercept them and send them to the backup storage without any "landing" zone. If any part of the path from the production VM to the backup storage (i.e. the network, or the target storage) is slower than the rate at which writes are coming to the VM, it is destined to slow down/freeze the VM.

And secondly, I never said PVE uses snapshots : )

@SkyDiver79,

Well, we still need to solve the problem of storing writes somewhere : )

Thanks!

[blockbridge]

@PTide - the most efficient way is to delegate to storage creation of a snapshot and then attach that snapshot/clone directly to backup server, which I think Veeam is able to do. You get multiple advantage points:
a) production data flow is not affected b) the consistent view of the entire VM data set c) the data does not flow from storage to production to backup, but directly to backup d) no resource usage on your hypervisor
Good storage backend is already very efficient with dealing with snapshots.

[blockbridge]

Interesting Proxmox I/O performance comparison vs. ESXi on NVMe storage > https://kb.blockbridge.com/technote/pro ... e-nvmetcp/
Although I'm sure a bunch of other test scenarios can be created where ESXi would pull well ahead of Proxmox as well.
But nevertheless, it's good to know that Proxmox is able to put up a serious fight at least in some cases

Hi Anton, we are very pleased that you enjoyed our article. We tried to be very fair in our testing as a 3rd party that works with both solutions. The results surprised us as well. Keep in mind that the testing was storage centric.
You might be interested in our latest technote as well: https://kb.blockbridge.com/technote/pro ... art-1.html

Cheers

[ravib123]

Proxmox integration would be awesome.

Honestly, https://www.bdrsuite.com/kvm-backup/ claims to be able to do agentless backup for KVM.

I suspect this is because they are tying into KVM/QEMU directly and skipping the proxmox API layer that the proxmox backup server uses.

The biggest challenge is making sure agentless backup is software-aware (for SQL/exchange/etc).

Currently, the proxmox backup server product isn't software aware and only has very basic backup capabilities, though it can do quick image-based or file-level restore and leverages ZFS for compression/deduplication.

We use proxmox in many client locations with clusters both hyper-converged with CEPH and without HCI.

Veeam is a robust backup product, with a maturity level that provides a granular level of control. If we could get veeam for KVM (ideally distro agnostic) that would be a deal breaker because it would easily cover proxmox (my personal choice due to how active they are in development and how rock solid the product is) and other KVM/QEMU based products.

Currently looking to move away from VMware and solid backup regardless of platform alternatives has been the single biggest challenge.

We want the ability to do live migration from the backup server to production, agentless backup that is application-aware, and the other features of veeam are a plus.

Currently, KVM/QEMU is 100% capable of live migrations, and many backup products (like datto bcdr for example) use QEMU to spin up the backup to live use before performing the live migration back to the host cluster.

Some folks might say that you already support Nutanix so direct KVM should be easy but I suspect that Veeam just leverages the PRISM API calls for everything and never actually touches KVM, but I am not sure how this is currently implemented.

I'd love to beta-test anything the Veeam dev team comes up with as my preference is to maintain agentless backups.

[azpets]

As forseeable, Broadcom dropped the axe.
https://kb.vmware.com/s/article/2107518?lang=en_US

ProxMox integration and support might come in a reasonable future?

[PTide]

In a reasonable future, for reasonable configurations : ) speaking of which, how do yall deal with 'root' credentials on PVE? Do you leave them as is, or do some tweaking (e.g. disallow ssh logon) and hope for the best?

So far it seems that there is no way to completely make do with just some "admin" (be it linux, AD, or PVE) account as "root" seems to stay mandatory for some operations (I have never seen a definitive list of those operations though).

So, what's your experience/best practice?

Thanks

[SkyDiver79]

For the daily doing a @pve user is sufficient. For updates and similar activities, there are personalized @pam users with sudo rights. Root is then not used. In the GUI you can simply use 2FA and on CLI I also sometimes use the Google Authenticator tool for 2FA on CLI.

[DerOest]

We're a bit off topic here, excuse me for keeping that rolling =)

So far it seems that there is no way to completely make do with just some "admin" (be it linux, AD, or PVE) account as "root" seems to stay mandatory for some operations (I have never seen a definitive list of those operations though).

So, what's your experience/best practice?

There were several enhancements in this area (see below). And I think I've read several more changes on the Mailing List that might have already been rolled out as regular updates in 8.1, but then they'll show up in the next releases release notes only.
IIRC it goes in the direction of "only apt updates still require root@pam because of the direct console access".

from https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_8.0

Create, manage and assign resource mappings for PCI and USB devices for use in virtual machines (VMs) via API and web UI.

Mappings allow you to give out access to one or more specific device(s) to a user, without them requiring root access.
For setups that require VMs, with hardware passthrough to be able to (offline-) migrate to other nodes, mappings can be used to ensure that the VM also has a valid device for passthrough on the target node.

[dw432948jkk]

If we could get veeam for KVM (ideally distro agnostic) that would be a deal breaker because it would easily cover proxmox (my personal choice due to how active they are in development and how rock solid the product is) and other KVM/QEMU based products.

+1 for a distro agnostic veeam for KVM !

[brasmussen]

+1
I have been playing around with Proxmox and it isn't a bad hypervisor. It seems to meet most of the requirements I am looking for. It is not VMware but it does have a lot of features for the price but I do wish it had Veeam support.

[dwright1542]

We're researching and doing some prototyping around Proxmox to see what's possible there as far as backup goes.

It would be helpful if we replace future +1s in this thread with information on:

1. Proxmox infrastructure size.

2. Virtual disk/storage type you're using (QCOW2 vs. RAW vs. ZVOL vs. other).

3. Any other infrastructure info that might be directly relevant to Proxmox VM interaction/backup/restore. Not knowing Proxmox yet I simply don't know what to ask, but for example with Hyper-V you can use standalone hosts, failover clusters or SCVMM - looking for this kind of infrastructure info/peculiarities.

+1
Small boutique MSP, about 40 hosts running ESXi, some on Essentials Plus, many on aggregated ROBO. Definitely considering Proxmox, but can't until Veeam is ready, especially with immutable storage.
A few are running hyperconverged software from Stormagic (who support KVM)

[f.neumann]

+1 for Proxmox
Small Customers 1-3 Hosts, with local Storage or FC-SAN. Around 50 VMs per Environment.
We’ll probably need to switch customers that need to renew this year to Hyper-V until Veeam supports Proxmox. But not really happy with that. Proxmox looks much better - sadly without veeam currently!

[Gostev]

Could you share in which specific areas do you find Proxmox to be "much better" than Hyper-V? Are you comparing on paper at this time or have you already deployed some production environments? Asking because I'm observing repeating feedback that Proxmox is quite "unfinished" in most departments, possibly due to trying to check all the boxes and spreading dev resources too thin across all of them. I don't think I ever heard similar sentiments about Hyper-V or AHV for that matter, this type of feedback seems to be unique to Proxmox...

[SkyDiver79]

So far I have not only migrated customers with vSphere but also a few from HyperV to Proxmox.
With HyperV, the main reason was usually the better and, above all, smoother management. Sometimes also the support of unusual operating systems.
What is currently giving some German companies pause for thought is the cloud obligation with AzureStackHCI 23H2.
Some companies have voluntarily made the Azure Arc connection, but the obligation is particularly unpopular in Germany.

[eits]

+1 for following this topic, also +1 on xcp-ng and others for diversity which imho should always be the drive

as for now we do vmware and hyper-v clusters with fc and local storages and plan for testing alternatives as the clouds are darkening, so no details provideable on those yet

[SkyDiver79]

XCP-NG is not Enterprise ready for me. When you would use Snapshots and incremental Backups, you have a Limit of 2TB per Virtual Disk. With larger Disks, you loose all good Features. The old Kernel and the not so good maintained XEN are personal Pain Points for me. The most other Projects include Nutanix use KVM and I think this the Future.

[je1000]

Could you share in which specific areas do you find Proxmox to be "much better" than Hyper-V? Are you comparing on paper at this time or have you already deployed some production environments? Asking because I'm observing repeating feedback that Proxmox is quite "unfinished" in most departments, possibly due to trying to check all the boxes and spreading dev resources too thin across all of them. I don't think I ever heard similar sentiments about Hyper-V or AHV for that matter, this type of feedback seems to be unique to Proxmox...

A few easy ones that comes to mind.
-usb passthrough
-no need to create a domain controller for live migration
-no need for SCVMM
-light web interface vs MMC

There is much more, but I only have a few days of Proxmox experience.

[kavaa]

Think veeam needs to expand their game to other playforms. Agent based is not a great solution if you have some smaller vms. The agent will take quite some resources. Yes Proxmox has their own backup server. But still would rather have veeam

[Steve-nIP]

As forseeable, Broadcom dropped the axe.
https://kb.vmware.com/s/article/2107518?lang=en_US

ProxMox integration and support might come in a reasonable future?

That change is the absolute least of what Broadcom has done, and isn't relevant to Veeam anyway, since the Free version doesn't expose the APIs Veeam needs

[ITP-Stan]

All perpetual licensing has been terminated and renting for MSP's is a disaster with a huge min CPU cores and price increase.
It's probably the end of VMware dominance in the SMB market.

Exactly the same that happened to Symantec when Broadcom acquired them.
Nobody in the SMB market is using Symantec anymore, at least here where I live.

[JustBackupSomething]

Adding a +1 for Proxmox or open source KVM / QEMU Support from this side.

Ill add the following minor problems with Veeam / Hyper-V support.
- Hyper-V MMC can sometimes "block" snapshot / merge operations. Additionally, if a merge fails. it typically requires a hyper-v management service restart or (sometimes more commonly) a restart of the entire box. This isnt great if the client hasnt got this in a failover cluster and needs to shut down their entire operation.
- WMI - Ive seen several WMI related issues recently for the Veeam Management Agent and permissions and Veeam after some upgrades. the addition of 2FA has seemed to exacerbate the issue.
- Adding a +1 for a dependency on a domain controller for cluster operations. - from the security point of view, this is an additional machine that needs to be closely managed / protected. Especially in service provider situations and for large businesses. - ive also seem lots of companies "share" the same DC with their production environment.

For VMWare,
- Cost - im not involved in the cost side of things, but from what i can see, its a significant jump in cost to have the platform in operation.
- Partner Program - The recent changes have seemed to leave a sour taste in everyone's mouth for partners and the future seems unsure.

Regardless of what platform businesses go for, lots of companies will see the recent news and be looking for more options in the next few years.

New Windows machines appear not to have a dependency on the exact same hardware (as much) anymore and moving a system from one platform to another is relatively painless.

[RolfG]

Adding also a +1 for Proxmox

We have a lot of small Customer with 1-10 VM's. All have the ESXi Backup Api for Veeam B&R. Missing in Proxmox AD or SQL Object and the File Level recovery could be much better. If you use a single host it would be nice you can restore the config or everything if you have a Hardware crash or meet the Linux Crack

[Exrace]

Veeam is uniquely positioned to chose a vSphere successor, specifically in the SME space, since a lot of MSP’s and SME customers are using veeam and they love it, and veeam is a huge part of the their virtualization strategy, they will choose a hypervisor that veeam supports. In the same time if Veeam doesn’t include more hypervisor options soon enough it will force a lot companies to give up on-prem infrastructure as the added VMware tax will make it a none viable option and make the cloud more financially attractive this will result in a shrinking market for Veeam.

From my research it seems that XCP-NG is the closes to vSphere both in reliability, scalability and support options. With Veeam’s support it will make Broadcom regret buying VMware. I personally hope that Veeam moves fast to announce its intention to fully support XCP-NG

I second this. XCP-NG is the closest to VM$$$ in function and scalability for SME. Veeam could create a whole package around this open-source solution including backups and host orchestration.
I would love to see this, and I would love to test this. Migrating workloads over to this might make me come out of retirement!

[sf4692]

Hello All, is there anymore info from Veeam Side about Proxmox integration? I am testing also testing another solution for their Proxmox Support, but would really, really love to stay with Veeam.

[Gostev]

No additional official info at this point, however the results of our research and prototyping efforts are promising. Meanwhile, do keep in mind that you can already protect Proxmox VMs with Veeam using agent-based backups. As I assume you probably need a proven solution today already, and not at some point in future!