Agent backups nog longer working after renew of certificates cfr KB2806

[abwi]

We recently encountered the problem where the certificate has expired in Backup & replication 9.5 (KB ID: 2806).
We renewed the certificate as per KB2806 and for the server jobs this is ok.
But for the agent jobs we can't get them to work.
We tried re-installing the agent, without any difference.
Error that we get : 03/12/2018 16:05 :: Error: Failed to connect to remote backup service.

Anyone else seeing this?

[wishr]

Hi Wim,

Just checked it and do not see any issues with my existing agent jobs (managed by B&R) after certificate renewal on the server. It may be a separate issue.

Btw, are you experiencing the issue with Agent jobs managed by B&R or standalone Agent jobs? In any way, I would recommend you to involve our support team into the investigation since an extensive log analysis is necessary in this case. Please do not forget to post your case ID here, so we'll be able to assist as well.

Thanks,
Fedor

[ssimakov]

Try to restart Veeam Backup service manually. Under certain conditions restart initiated by the certificate wizard might not be enough.

[millardjk]

Hi,

I also ran into an issue after trying to replace the certificate: Failed to validate remote certificate.

This was when I replaced the certificate with an internally-signed cert from my Windows PKI. Did some digging, and found this page, which documents the requirements for the certificate: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Unfortunately, the requirements include certificate signing, which has the effect of adding a certificate authority to your internal chain of trust; that's unacceptable to me, so I went down the path of creating my own self-signed certificate using OpenSSL so that I could supply a longer validity period (doing it through the console results in a 1Y certificate). Although I was able to create the cert with all the necessary key use policies (basically the same as the console-generated one: all policies) with the desired lifetime, it won't work.

Among other things I discerned that is different from the way I'd generate a cert: The signature hash algorithm on the console-generated cert is SHA1, while the rest of the world has moved on to SHA256.

I'd like to see a little more transparency into the certificate requirements, including whether or not SHA1 is required--and if so, when it'll be updated to SHA256--as well as other requirements like CN and Subject.

[wishr]

Hi Jim,

Thank you for your comments and efforts.

We'll take a look at that. I'll share some additional details a bit later.

Thanks

[Ottrott]

Hi,

I can (partly) confirm this problem. After generating a new certificate as per KB2806 the server & the (local) jobs worked fine. But all agent jobs failed with an certificate error. I managed to get the agent backup jobs to work again by deleting all Veeam Server certificates from the certificate store of the backup server and reissued a new certificate as per KB2806. After restarting the Veeam services the agent jobs started working again.

[Dima P.]

Hi Ottrott,

Thanks for the confirmation and glad to hear that proposed solution resolved your problem. Cheers!

[wishr]

I'd like to see a little more transparency into the certificate requirements, including whether or not SHA1 is required--and if so, when it'll be updated to SHA256--as well as other requirements like CN and Subject.

Hi Jim,

As promised, I'm reverting back with some explanations. Please find them below.

Currently, we officially support only SHA-1 certificates for Agent Management communications (Veeam EM uses SHA-256, though), but we are also looking into the possibility to officially support other modern hash functions. The biggest issue with that is related to legacy OS support: our Backup Agents may be installed on quite old operating systems, and these OSs may not support modern cryptographic hash functions or will require to install additional packages that will potentially lead to Agent deployment difficulties. But anyway, this topic is being researched on our end.

Speaking about the Certificate Signing requirement, it's necessary because the root B&R certificate is used to sign the child Agent certificates. It will be impossible to use the Agent Management functionality without this key usage setting, unfortunately. To the best of my knowledge, it should be possible to generate certificates compliant with this requirement using OpenSSL and some other utilities.

As for the article itself, we are looking at the possibility to add more transparency to the list of requirements including supported hash functions, CN, subject, and so on.

Currently, please specify your B&R server FQDN in CN and subject - it is required for Agents communications with B&R server.

Thanks for bringing that to the table by the way, and let me know if you have any additional questions.

BR,
Fedor

[PKaufmann]

Hi,

I can (partly) confirm this problem. After generating a new certificate as per KB2806 the server & the (local) jobs worked fine. But all agent jobs failed with an certificate error. I managed to get the agent backup jobs to work again by deleting all Veeam Server certificates from the certificate store of the backup server and reissued a new certificate as per KB2806. After restarting the Veeam services the agent jobs started working again.

Thanks for this hint.
Deletion of the old veeam server certificate from the Agents' servers did the trick

[johhen1]

Hi,

We have the same problem. Agents not working after certificate renewal.

"Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.156:10005
Initializing"

Tried deleting old cert on Veeam backup server. Also tried deleting certificates at Veeam agent servers and restarting the service. Still same error.

Any suggestions?

-johhen

[Dima P.]

Hi johhen,

We observed several cases where port 10005 was not properly released after certificate renewal, so please try either restart Veeam Backup Service or reboot your backup server. That should solve the issue, but if not, please let me know. Thanks!

[Asahi]

Hi,

I also encountered the exact same event, Agent backup was failing.

I looked at the topic here and restarted Veeam Backup Service and the agent backup was successful.

Thanks！

Asahi,

[Dima P.]

Hello Asahi,

Glad it works and thanks for the confirmation. Cheers!

[johhen1]

Hi,

It worked for us as well, the problem with 10005 not listening was spot on, which we discovered before, but didnt want to restart the service while backup jobs were running. So had to wait for a Windows when no jobs were running.

All good now.

Might be a good idea to put a warning in the cert renewal KB that service need to be restarted twice if 10005 is not listening.

-johhen

[Dima P.]

Henrik,

Thank you for sharing. I'll discuss it with our support team!