Trying to deploy veeam proxmox workers. getting errors during test.

[gdvm]

Hi,

I can add the PVE hosts and deploy the PVE host veeam worker/proxy VMs without issue, but during the test phase they both (have two pve hosts in POC) fail with

Code: Select all

Failed to connect to the worker core service: Failed to connect to the backup appliance

and the worker VMs are shut down again.

In the logs which are saved on the veeam server I can only find the following error:

Code: Select all

error:141A318A:SSL routines:tls_process_ske_dhe:dh key too small

I can see in the logs the workers try to connect back to the veeam server and it fails with that error. No other errors are visible, afaict.

Same results when I follow the add host wizzard and check 'add worker' or manually try to add a worker.

Any leads?

goran

[PTide]

Hi,

This one looks very odd as Proxmox does not require an appliance.
Are you running the latest VBR 12.2? Which Windows version is used?

Also, which checkbox are you referring to?

Same results when I follow the add host wizzard and check 'add worker' or manually try to add a worker.

Thanks!

[gdvm]

Well.. proxmox suggests you should install 'workers (== proxies)' on your pve hosts when you add them through the wizzard (it's a checkbox). Then, when it's installed it will test them and starts them. I see all that in the proxmox events. The deploy/test window in veeam reports the error after a while. I'll assume that it is not a big thing in a test environment with two hosts and 6 VMs, but as the option is offered, I'd like to try.

Also you can deploy the workers in the backup proxies after the fact under backup infrastructure, if you have pve hosts added.

Fwiw. I've been reinstalling veeam on windows 2022 since noon, because veeam might have missed 2012r2 (my POC server version) testing for proxmox support, even though it the documentation suggests that 2012r2 is supported for v12. There might be some windows old-school ssl cert used, but I didn't figure out yet if this is the case or on which service. Almost there. We shall soon know.

[gdvm]

well.. there you have it. I tested the same thing on win2022, which I figured might halp.

30-8-2024 15:41:29 Connection to the worker core service was established successfully
30-8-2024 15:41:29 Connection between the worker and backup server was established successfully
30-8-2024 15:41:34 Connection between the worker and cluster was established successfully
30-8-2024 15:41:38 The worker VM was powered off successfully
30-8-2024 15:41:39 The worker veeam-worker1 was tested successfully


So as speculated, it's likely an old school ssl/cert thing in 2012r2 which veeam uses but has not tested during proxmox development?

[gdvm]

Hi,

This one looks very odd as Proxmox does not require an appliance.
Are you running the latest VBR 12.2? Which Windows version is used?

Also, which checkbox are you referring to?



Thanks!

Just found this veeam blog that suggests the same thing I was/am doing. https://community.veeam.com/blogs-and-p ... art-1-8016

Anyway, it seems to work in a newer windows.

[Javier Lobato]

Hi,

Same error here with the worker .

Code: Select all

Failed to connect to the worker core service: Failed to connect to the backup appliance

But after parsing log, it seems to be a communication failure from Windows Firewall ( in my case ) seeing a drop traffic on port 19001 and 10006 from Worker to VBR.

Example:

Code: Select all

2024-09-02 15:45:34 DROP TCP 192.168.71.123 192.168.71.101 39854 19001 60 S 1111399525 0 29200 - - - RECEIVE
2024-09-02 15:46:06 DROP TCP 192.168.71.123 192.168.71.101 39854 19001 60 S 1111399525 0 29200 - - - RECEIVE

Code: Select all

2024-09-02 15:53:39 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE
2024-09-02 15:53:40 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE
2024-09-02 15:53:42 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE
2024-09-02 15:53:46 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE
2024-09-02 15:53:55 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE
2024-09-02 15:54:11 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE
2024-09-02 15:54:43 DROP TCP 192.168.71.123 192.168.71.101 43992 10006 60 S 353682441 0 29200 - - - RECEIVE

Referring to the official doc the 10006 port doesn't seem to be reported

https://helpcenter.veeam.com/docs/vbpro ... html?ver=1

Maybe it can help.

[PTide]

Hi,

Right, there are some issues with cyphers lengths in the older Windows versions (2012/2012R2).
There will be a KB issues on the topic soon.

Thanks!

[urbanovits]

Same issue here

***Failed to connect to the endpoint***

Case ID: 07431601

Setup: Windows 2016 fully patched // dual homed!!
Veeam 12.2 community

Veeam backup asked to install worker I cannot backup PMX VMs without this.

Veeam 192.168.100.189 (main link) / 192.168.200.189 (storage)
ProxMox 192.168.100.184 (main link) / 192.168.200.184 (strorage)
Vorker 192.168.100.190 (fixed IP)

Veeam server config
Network traffic rules set to use 192.168.100.0/24network

Name resolution set by host file AND DNS

Test failed al the time
Deep drive into logs (C:\ProgramData\Veeam\Backup\Plugins\PVE\workers)

Worker unable to resolve veeam server by name or even by ip => Register backup server [169.254.105.234]:[10006]
<logpath>\<worker-unicID<GUID>>\veeam_backup\test_connection_service\vmb_api log

Code: Select all

default option 'VmbApiPreReadBufSize': 1048576
[22.09.2024 11:33:13.356] <140079801358080>          |     Non-default option 'VmbApiPreReadQueueDepth': 32
[22.09.2024 11:33:13.356] <140079801358080>          |   /etc/VeeamAgentConfig was found.
[22.09.2024 11:33:13.357] <140079801358080> vmbcli   | Registering backup platform PVE. ok.
[22.09.2024 11:33:13.358] <140079801358080> vmbapi   | Registering host logging name [Veeam]
[22.09.2024 11:33:13.359] <140079801358080> vmbapi   |   Registering host logging name [Veeam]
[22.09.2024 11:33:13.359] <140079801358080> vmbapi   | Registering host logging name [Veeam] ok.
[22.09.2024 11:33:13.362] <140079801358080> vmbcli   | Register backup server [169.254.105.234]:[10006]
[22.09.2024 11:33:13.362] <140079801358080> cryptlib |   Load [libPkcs12Converter.so] library
[22.09.2024 11:33:13.393] <140079801358080> vmb      |   Using stub connection params provider.
[22.09.2024 11:33:13.395] <140079801358080> vmb      |   Connect to vbr[{602f5396-dcaf-42bc-9989-73b1fc5a79fb}] by certificate with fingerprint: 3F64E0B20412F2072F1119B2E02CED8EE504A94B
[22.09.2024 11:33:13.395] <140079801358080>          |   [NetworkPolicy] Global network policy has been set (default):
[22.09.2024 11:33:13.395] <140079801358080>          |   [NetworkPolicy]     IPv4 disabled: False
[22.09.2024 11:33:13.395] <140079801358080>          |   [NetworkPolicy]     IPv6 disabled: False
[22.09.2024 11:33:13.397] <140079801358080>          |   Trying to connect to the endpoint [169.254.105.234:10006]
[22.09.2024 11:35:21.808] <140079801358080>          |   Connection status: system:110 ( Connection timed out ).
[22.09.2024 11:35:21.826] <140079801358080> vmbcli   | Register backup server [169.254.105.234]:[10006] Failed.
[22.09.2024 11:35:21.827] <140079801358080> vmbcli   | ERR |Failed to connect to the endpoint [169.254.105.234:10006]. Connection timed out

**NOTES
IPv6 disable false?? I set to disable
169.254.105.234 ?? Is'n my Veam server IP address

It seems to be some malfunction between Veeam server and Worker, because network policy settings not transferred or even updated by applied Cloud init

Symptom identical if i get rid of netBIOS or even DNS FQDNs , and try to run on IP addresses only. Worker unable to resolve Veeam server IP address, always try to use APIPA (169.254.x.x.) to get connected, then fail.

I did not find way to log on to worker machine, username/password that Rocky Linux different (set by Veeam) as my ProxMox root account

Worker VM investigation pointing out , that is an Rocky linux ISO, using Cloud Init to set config. Ok lets do some hack, Ubuntu iso insterted and lets see what is inside

The worker VM gas a sybolic link pointig to link-local 169.254.0.0 (etc/networks)
That is the first problem!

Switch over to Cloud Init iso I found network config sees to be OK
(Cloud init set to proper DNS and DNS suffix on PXE level)
In addition found Worker needs to have FQDN (user-data file in Cloud Init)

BTW another bugs:
-Worker set to not to upgrade on Veeam server level, and cloud init show opposite settings upgrade-yes
-Network Config set to static IPv4 and No thanks IPv6, but no value set there, and IPv6 still waiting as enabled

Lets fix all, and try

Worker FQDN name set on DNS
IP Set on cloud init (probably this will fix Rocky networks settings 169.254. whatever )

Failed, symhtom still same same
log say
<140063284184832> | Trying to connect to the endpoint [169.254.56.190:10006]

Worker stiil in scizopremic mode, does not have a purple guess where the hell Veeam server is.



Bigger slap coming
Lets get hardcode the ip address inside Roky Linux's hosts file
No easy task, boot from Ubunti iso, then edit as sudo

No change, still not working.

Out of ideas

[rovshan.pashayev]

Hello Gyorgy,

Thank you for detailed report, keep in touch with Support team for now.

[urbanovits]

Root cause here

I found in the logs, that not connected network cards have APIPA IP addresses
In my server I have 6 LANs and only two connected yet the rest 4 left . Veeam server believe those LANa are up and operate regardless of the status down state.

see:
Network Interface, Name: Embedded FlexibleLOM 1 Port 2, Description: HP FlexFabric 10Gb 2-port 554FLR-SFP+ Adapter, Interface Type: Ethernet, Operational Status: Down;
Unicast IPAddresses: fe80::91b53cf:69ea%3; 169.254.105.234;
Network Interface, Name: Embedded LOM 1 Port 2, Description: Broadcom NetXtreme Gigabit Ethernet, Interface Type: Ethernet, Operational Status: Down;
Unicast IPAddresses: fe80::f905:cc0:8502:38be%2; 169.254.56.190;
Network Interface, Name: Embedded LOM 1 Port 4, Description: Broadcom NetXtreme Gigabit Ethernet #3, Interface Type: Ethernet, Operational Status: Down;
Unicast IPAddresses: fe80::d9ef:b71f:dccc:d0ca%11; 169.254.208.202;
Network Interface, Name: Embedded LOM 1 Port 3, Description: Broadcom NetXtreme Gigabit Ethernet #4, Interface Type: Ethernet, Operational Status: Down;
Unicast IPAddresses: fe80:c35a:f529:7518%10; 169.254.117.24;
Network Interface, Name: Embedded FlexibleLOM 1 Port 1, Description: HP FlexFabric 10Gb 2-port 554FLR-SFP+ Adapter #2, Interface Type: Ethernet, Operational Status: Up;
Unicast IPAddresses: fe80::f855:8577f971%7; 169.254.249.113;
Network Interface, Name: Embedded LOM 1 Port 1, Description: Broadcom NetXtreme Gigabit Ethernet #2, Interface Type: Ethernet, Operational Status: Up;
Unicast IPAddresses: fe80::7cb3:8adf:6442:b8b5%6; 192.168.100.189;
Gateway IPAddresses: 192.168.100.253;
Network Interface, Name: Loopback Pseudo-Interface 1, Description: Software Loopback Interface 1, Interface Type: Loopback, Operational Status: Up;
Unicast IPAddresses: ::1; 127.0.0.1;
Network Interface, Name: isatap.{99BF1CEF-1943-4580-B2C2-4C31C702B32A}, Description: Microsoft ISATAP Adapter, Interface Type: Tunnel, Operational Status: Down;
Unicast IPAddresses: fe80::5efe:192.168.100.189%8;
Network Interface, Name: isatap.{A235E8C3-31AB-4906-A17D-03E928DFDC8C}, Description: Microsoft ISATAP Adapter #2, Interface Type: Tunnel, Operational Status: Down;
Unicast IPAddresses: fe80::5efe:169.254.249.113%4;
UTC offset: 2,00 hours

When i disabled all not connected LAN cards on Windows ProxMox Proxy agent test succeed.

IT IS A BUG.
Veeam dose not lookup for Link down status of LAN cards when proceed a list of available connections info to Veeam server. This fake info passed to Proxy agent, and then agent getting failed.

CASE opened with ID of 07433641 including all of this.

[rovshan.pashayev]

Hi Gyorgy,

Thank you for the detailed reply and the assigned case number. We will review it in-depth.

[urbanovits]

Thanks Rovshan

Let me share my worries about this issue. This problem occurred because missing input validation before build array in which Veeam server IP addresses are collected. And regarding to the log records, the very first IP address selected (probably depend index of that variable/array) and passed to the agent. Instead of any valid one -which have link UP-, PLUS the IP range of backup network does not have any effect here.

Let's think forward
If my statement true, the possibility to find missing input validation in Veeam code somewhere else too should be high. Let's assume a potential attacker targeting a Veeam backup server (or even underlying Windows or .NET ), and he/she able to modify the record list ( which stored in variable) of IP addresses where Veeam restore to. Attacker able retarget restore job to the IP address which owned by him/her. Xtrafil data-data thief

Probably I'm too paranoid thinking this way, but this is my job
CISO/CEH/NIS2 consultant/ISO-IEC 27001- 27005 - 22301 lead auditor and many other cybersec certs and role.

[PTide]

Hi,

Let's assume a potential attacker targeting a Veeam backup server (or even underlying Windows or .NET ), and he/she able to modify the record list ( which stored in variable) of IP addresses where Veeam restore to. Attacker able retarget restore job to the IP address which owned by him/her. Xtrafil data-data thief

Kindly submit your findings here.

Thanks!

[urbanovits]

I came to conclusions only by analyzing the debug log elements.

There is no written agreement that Veeam will entrust us/me with any type of testing. In the absence of this, according to the code of ethics, I will not conduct further investigation.

The fact is that Veeam makes errors during the collection of network information data, which is presumably the result of faulty program code. From this I derived the conclusion that there is no input validation at this point, nor will it be in several places. Consequently, the vulnerability of the program is real

[PTide]

When i disabled all not connected LAN cards on Windows ProxMox Proxy agent test succeed.
IT IS A BUG.

This one has been fixed just yesterday.

As for submitting the vulnerability report - the link that I gave you is one of the two preferred ways to submit such things.
The second method woud be to open a support case (which you did).

So, stay tuned, and thank you for bringing this up

Cheers

[duchenokelly]

i´ve been several days trying to solve this issue and its now solved.
I disabled all not connected LAN and the worker was tested sucessfully.
Thanks so much.

[Njumaen]

The error in my homelab when testing the workers deployment is

03.10.2024 16:38:22 Error Worker veeamworker test failed: unable to create VM 100 - copy failed: command '/usr/bin/qemu-img convert -p -n -O qcow2 /var/lib/vz/template/iso/PveWorker_1.0.0.439.img zeroinit:/mnt/pve/SSDINTERN/images/100/vm-100-disk-1.qcow2' failed: exit code 1

Only one active interface. Veeam B&R runs on a secondary PVE to backup the primary PVE (independent, not a cluster)

Ralf.

[PTide]

Hi,

> Only one active interface.

Where? On VBR, on Proxmox VE node which you are trying to deploy the worker to?
Also regarding primary/secondary PVE - did I get it right that you added two separate PVEs (not clustered) in VBR, and now you are trying to use worker installed on one PVE to back up VMs from the other PVE?

Thanks!

[Njumaen]

On VBR...

And yes, you got it right.

I reduced the complexity to investigate further more. Now I deployed the worker on PVE#2 (the one that ist running VBR) and this worked perfectly! \o/

Investigated further.

PveWorker_1.0.0.439.img on PVE#1 was smaller than on PVE#2... Maybe the initial upload failed!

Removed it... and hooray... worker was deployed on PVE#1 !

Maybe VBR can check the .img file size during deployment in future versions.

Happy cheers,

Ralf.

[tgx]

"Right, there are some issues with cyphers lengths in the older Windows versions (2012/2012R2).
There will be a KB issues on the topic soon."

Has there been any update on this? I am still stuck at "Failed to connect to the worker core service: Failed to connect to the backup appliance".
Appliance is there, nothing in between that should block it, it just doesn't work. Windows Server 2016 Standard.

On another note, I am unable to ping the worker at all, from any machine.
I have a static assigned IP. The IP is addressable via DNS entry both hostname only and FQDN.

I tried using the 'root login password derived from the machine name' but it doesn't work so I cannot log in to the worker
to troubleshoot.

[Gostev]

If you're on 2016 then your issue is clearly different.

[tgx]

It was unclear to me which versions may be affected. It does seem like my issue is different.
I will create a new thread.