Searching advice - Slow restore speeds towards PVE?

[HenrikS.]

When restoring to PVE, we see slow transfer speeds, approx 500Mbps.
There are no obvious reasons for it, dual 25GbE links, no other running jobs, system capable of delivering 10+Gbps to VMware and PVE hosts have no load and NVMe disks.
Looking at metrics, I see that 1x CPU core on the Veeam AiO Windows 2022 is maxing out while running restore and no other jobs.

So, question: is the function of restoring to Proxmox VE single-threaded? Are there other obvious tweaks to implement?
Also, is it possible to change MTU for the Proxmox VE workers, as we see 6x performance increase due to interrupts handling in firewalls that lie inbetween by switching from 1500 to 9000

[david.domask]

Hi Henrik,

Hard to tell without more details, but guess is that the backup server is being used as part of the restore and handling a lot of the processing.

Similarly, depending on your worker deployment configuration, you might be seeing slower speeds if the network mode is being used instead of hotadd; hotadd requires the worker to be on same node as where you're backing up / restoring to, so probably check that first. If the worker is using network, it may explain the slow speeds.

If you're not able to make headway on it, open a Support Case to check the behavior, Support will be able to tell more from a review of the debug logs and understand why you're seeing less than stellar speeds.

[HenrikS.]

It is a cluster of 6 proxmox hosts, one worker pr host, restore to one host with local worker.
I see the restore job is using the local worker and hotadd mode.
I see no special CPU usage on the Backup server during my run today,
CPU usage on the worker, deployed with default 6c/6G is at about 50% reported by PVE, and incoming network traffic is at about 60MBps.
If I try two restores at the same time, then the incoming speed is doubled? So where is are the limitations...?

[david.domask]

Hi Henrik,

Right now not able to say, so please open a Support case as noted above; a review of the debug logs will likely help shed light on what's happening here.