Veeam backup of PVE vm on spinning disks ceph issues

[squebel]

Hello, case 08166423 is open with Veeam support. We are running 12.3 and plugin version 1.5 and we have a couple of PVE clusters running Proxmox CEPH but using a hybrid setup with spinning disks. In some situations, the snapshot that Veeam is creating of that vm is causing major IO performance issues. I am being asked to immediately upgrade to the latest version hoping that it will fix the issue. Is there anything in the newer version and plugin that fundamentally changes the way a PVE vm backups are run?

[vnikiforov]

Hello,

No - the general backup flow is the same. The upgrade advice still stands, though. V12 passed End of Fix in November 2025 under the product lifecycle policy, so bug fixes now ship on V13.1+ only, and there were snapshot handling changes between PVE plugin versions.
With 13.1 expected in the coming weeks, I'd go straight to that. If the IO impact reproduces on the latest version, open a new support case with new logs from a new version.

[squebel]

I deployed a fresh v13 instance using the latest VBR appliance and set up a backup job to backup a single Windows vm on the proxmox cluster with a worker on the same node as the vm. IO impact is still the same. The vm locks up hard after a few minutes to the point it has to be powered off and restarted after the backup job is stopped and cleaned up.

[vnikiforov]

Hello,

Please be advised, It was recommended to test that on the upcoming Veeam Backup & Replication 13.1. If the results are the same, I recommend opening a support case with logs from the 13.1 version. This will allow us to engage proper support resources relevant to the case.

[squebel]

Vladimir, are there functionally different processes behind the scenes on how the backup process works for Proxmox in 13.1 that could potentially result in less impact to the running vm? If not, why would 13.1 make any difference?

Also, we're aren't blaming Veeam. We do know that the design of this PVE cluster is not optimal when it comes to the underlying disks but we're just trying to figure out the best way to protect the data without causing impact. And we do not want to have to look at other options; we want Veeam to work for this.

[vnikiforov]

Hello,

Please refer to my initial message: There are no fundamental differences, BUT in v.13.1 there were many changes including PVE plugin, specifically related to snapshots handling, which might have effects on the IO operations. In any case, as per Veeam support policy, the latest version is required to be used to consider changes. That is why testing that on the latest version (once it is released) and 13.1 logs and new case are needed.

[squebel]

Are you able to provide some information on what those "many changes including PVE plugin, specifically related to snapshots handling" means and why it could possible have effects on the IO operations?

[vnikiforov]

Hello,

Unfortunately, not, but I strongly recommend you try the new release. And if the issue persists, we have all the resources to address it.
I can't speculate on the root cause or the potential effects, as no research has yet been done on a technical side, and on this forum we do not provide technical support.
That is why I recommended the appropriate approach to handle that.

[squebel]

13.1 has been released and we have upgraded the environment. Backup issues still persist. We are seeing the exact same issue as the users in this forum post https://forum.proxmox.com/threads/vm-be ... up.185199/

[squebel]

I've updated the ticket 08166423 with the information. Also the link to that proxmox user forum post. We have tried the NBD backup and it does seem to work without impacting the vm but it is very slow. PVE version 9.2.3

[edh]

+1 Seen this behaivor maybe since 9.2

[squebel]

I have asked support to close 08166423 as there does not seem to be any resolution possible and the only suggestion is to convert the hosts to using flash media instead of spinning media and no real desire to dig into why it is happening exactly. I can say that we do not have this problem when using Proxmox Backup Server. If Vladimir @vnikiforov wants to dig in, please do and I'll be monitoring this thread.

[vnikiforov]

Hello,

Thank you for your feedback. It was noted, and I directed this information to the relevant department.

[gasherbrum]

I upgraded our test/DR cluster past 9.2.3 and also confirm that this issue is present for me, crashes the VM during backup maybe 50% of the time. We are using SATA SSDs for storage rather than HDDs so it may not be limited to low iops
Case 03777075

[vnikiforov]

Hello, Joe,

Thank you for your additions to that matter.
Your case number is 08204008. The 03777075 is your company account ID.
Please provide full logs to the case otherwise engineers will not be able to help - logs are mandatory.

[gasherbrum]

Added logs, cheers

[rovshan.pashayev]

Hi @gasherbrum

I believe Support Team was able to help you with the issue.

[vnikiforov]

Hello, @squebel

As my colleague Rovshan stated above, we might have a solution for the issue (changing internal parameter) please open a new case supply fresh full logs and specify case number here, I will comment on it internally and you may refer to the existing case# 08204008

[squebel]

Case 08221019 has been opened

[vnikiforov]

Hello,

Yes, I see it and support has the reference to a possible solution already, just need logs now to confirm the setup.

[squebel]

Can the solution be shared before we try the next backup please? We do not want to crash out a production server just to test this out and prove it is still happening before we get the possible fix.

[sakrizz]

Hello,

Had to make a account here just to comment on the same matter as we found the solution yesterday.

We had the similar issue even with SSD Ceph solution (VM got completely stunned during HotAdd backup operations with Veeam) and after through investigation we were able to isolate the issue to librbd driver that is used by Ceph, if you are running Ceph on default settings. As far as my investigation and now bugzilla report towards Proxmox VE goes - slowness issue seems to be related to librbd that Proxmox VE uses as default Ceph driver. Once the backup starts, librbd seems to hang almost completely. Enabling I/O Threading on VM hard drive options helps a bit, but without changing the default Ceph options you are going to run into poor VM performance when running HotAdd backup of the VMs.

There is a thread on Proxmox forums regarding to exactly same issue that is described here - I don't know if i am allowed to paste any links here so you can find it with words "VM becomes unresponsive during Veeam backup".

Anyways if links are allowed, solution can be found from this thread: https://forum.proxmox.com/threads/vm-be ... 199/page-2
and there is a bugzilla report on same matter: https://bugzilla.proxmox.com/show_bug.cgi?id=8010

Issue is isolated to librbd which Ceph uses by default. Issue started occuring with Veeam B&R after Proxmox VE was upgraded to 9.2.4 or later.

If you change librbd to KRBD in CEPH RBD storage settings - there is a tick box for this (Cluster -> Storage -> Edit Ceph Storage) - next time you either migrate or shutdown the VM = it will use KRBD driver instead of default librbd driver when starting VM that is using Ceph as storage and issue does not occur with KRBD driver. We had serious performance issues with VMs that were running the backups with HotAdd (100kbps disk write performance when testing out with dd on linux systems) and after changing the driver to KRBD performance skyrocketed back to 400-1000MB/s write / read operations during backup.

Also backup job runs lot faster, now reporting target as bottleneck for us as it should be (atleast for our part).

Had a support case open regarding to this as well with Veeam, but we kinda got cold towel with that one (case: #08170275).

Anyways, we found the solution and filed a ticket to Proxmox VE and their bugzilla on this matter. Hopefully someone finds this information useful.

[squebel]

Support has supplied the suggested change. We will attempt the backup when we have an approved window where we can impact the production server and I will update the case. thank you!