"The response ended prematurely. (ResponseEnded)" when adding Proxmox Node

[jbender81]

Moin,
We have an issue when adding some specific Proxmox nodes to a VSA:
The SSH login works fine, also veeam@pam has sufficient rights within Proxmox and I also get the request to approve of the certificate presented on port 8006. So no immediate network issue. Still, after the approval of the certificate I get an error message "The response ended prematurely. (ResponseEnded)" .
Hence I pulled a log bundle from the VBR to investigate.

What immediate stood out to me, that the PVE plugin logged a request "https://<redacted>:8006/api2/json/nodes", but this request was never visible within the pveproxy log (/var/log/pveproxy/access.log). In fact, there was no request for the user "veeam@pam" visible, whatsoever. Also the PVE plugin logged an error message "status: error" with empty headers, cookies and body for that request.
Trying the debug log of pveproxy, which in itself was not exactly that helpful, I did some debugging by print within the pveproxy perl sourceode, what I found was, that the connection is closed by pveproxy due to a 5s timeout.
Unfortunately, that timeout is not exactly configurable. Fortunately, the source code is editable, hence I set the timeout to 60s, and the connection started to work perfectly fine.

During the same time I did some tcpdump/wireshark to the problematic connection. This showed, that after a pretty normal TCP and TLS handshake there is a 66byte big ACK packet sent from the VBR to the Proxmox node. After that there is ~5s no traffic, which is then followed by a FIN-Packet from the Proxmox node to the VBR, which is simply ACK'ed, but _not_ FIN/ACK'ed. After another 10s there is a 392byte big packet from the VBR to the Proxmox node, which is answered by a RST packet, as the Proxmox node already has closed it's side, and could not answer, even if it wanted to.
After increasing the timeout for pveproxy, the dump showed the same TCP and TLS handshake, followed by the 66byte big ACK packet from VBR to the Proxmox node. But this time there is a 392byte big packet from the VBR to the proxmox node after ~5s, which in turn is ACK'ed by the Proxmox node, followed by some bigger packets and a normal connection closure without RST.

It seems to me, that the VBR is just about to send the request, when the Proxmox node closes the connection, due to the 5s timeout, which happens to have the VBR churn for another 10s bevor actually sending the request into a "dead" connection.

<speculation>
We have another Proxmox cluster, which did not have this issue. But there is a difference:
The "new" cluster nodes are set up to use ACME to pull an additional certificate for pveproxy from an internal KMS additional to the certificate provided by the cluster internal CA, whereas the "old" cluster only uses the later. This means, that "pvenode cert info --output-format json", which is done via SSH, contains not two certificates, but three, which have to be compared by the PVE plugin to ensure, that on port 8006 is the same node as on port 22. Which in turn pushes the time between connection and first request just so slightly above 5s to have the timeout trigger.
</speculation>

Our mentioned workaround:
The timeout is set in the class PVE::APIServer:AnyEvent on line 2106 (at the time of this writing) within it's constructor called by the subclass' PVE::HTTPServer constructor on line 28. The constructor is called within PVE::Service::pveproxy on line 165 using a hash build up in said class lines 98 and following. Hence just including a "timeout => 60," within that hash sets the problematic timeout to 60s.
As this might be overwritten by any subsequent update of pveproxy, this is obviously not stable. Als I have no idea, whether the issue is only relevant for the initial setup, or whether there might connection issues afterwards, when removing the workaround. Hence it stays there for now.

The support case is #08149736 .

Cheers
Jörg