Automating certificate renewal for web UI (TCP 443) on Veeam Software Appliance?

[Loosus456]

We use Let's Encrypt along with the DNS-01 challenge to renew our wildcard certificate (*[.]example[.]com) every 45 days. We have a virtual machine that takes care of the wildcard renewal, retrieves the new wildcard files and private keys, stores the new wildcard files, and runs an SFTP server that serves the certificate files to other servers that need them. For security purposes, servers that need the certificate files reach out to the certificates VM rather than the certificates VM reaching out to them.

We need to rotate the web UI certificate every 45 days (or more often). We need to automate rotation because manual replacement obviously isn't really a thing these days.

1. Can I store a simple bash script on the Veeam Software Appliance to retrieve the wildcard certificate from our certificates VM? If so, is there a preferred directory path where I should store it?
2. Can I run a cron job on VSA under the root account?
3. Will VSA updates/upgrades eventually blow away our bash script and cron job? If so, is there a way to prevent that from happening?
4. Assuming I can put a bash script and associated cron job on the VSA server, which certificate file and private key file do I need to have the bash script replace/rotate? What is the directory path to those files? After replacement, do any services/daemons need to be restarted for the replacement to take effect?

[Mildur]

Hi Jason,

Running scripts or applying custom settings on our appliances is not supported: KB 4772.

This includes running custom scripts or changing cron jobs.

We have a PowerShell cmdlet (Add-VBRBackupServerCertificate) to replace the backup server certificate, but it still requires a backup service reboot, which cannot be done for the appliance over PowerShell or REST API.

Right now, there is no supported solution for your use case, but we are thinking about options to allow automated certificate installation.

Until we deliver such a feature, either stay on a Windows-based backup server, where you can script almost anything, or use VSA and manually replace the certificate whenever needed.

Best,
Fabian

[Loosus456]

Fabian, thanks for the response.

1. Let's say we used Add-VBRBackupServerCertificate: would that impact anything other than the web UI? We want to make sure we are impacting only the web UI and not the backend components. We are okay with the backend certificates being long-lived; it's only the web UI where we have a requirement for a 45-day rotation. In other words, I want to make sure we are not potentially breaking backend components by rotating the web UI certificates.

2. If we used Add-VBRBackupServerCertificate at a certain date/time (for example, Sundays at 5:00 PM), could we get vCenter to automatically restart the VSA virtual machine on Sundays at 5:05 PM? Would that do the trick?

Just to give you some background on our cybersecurity requirements: it would be better in our case if VSA would reach out to another server to get its certificate files, rather than another server reaching out to the VSA server. In other words, this direction of initial connection:

VSA --> Certificate Server

Is better than this direction of initial connection:

VSA <-- Certificate Server

In our environment, VSA is considered a "High Risk" server, which means that any time we have to poke holes in our firewall to communicate with it, it presents an audit risk. However, our Certificate Server is considered "Medium Risk," so it's not as big a deal to open a firewall port to that server.

[Mildur]

Hi Jason,

would that impact anything other than the web UI?

It will impact backup server and Web UI, since the certificates are not split yet. We have plans for this, but I cannot share any ETA.
I will add your vote (+1) to our internal feature request list.

If we used Add-VBRBackupServerCertificate at a certain date/time (for example, Sundays at 5:00 PM), could we get vCenter to automatically restart the VSA virtual machine on Sundays at 5:05 PM? Would that do the trick?

I expect so, but I recommend giving it a test. Replace the certificate and reboot your server.

it would be better in our case if VSA would reach out to another server to get its certificate files, rather than another server reaching out to the VSA server. In other words, this direction of initial connection

Thanks for sharing the use case. That would mean we need to build a tool for our appliances that can retrieve certificates, install them, and reboot services. It would also logically require an industry standard to request and retrieve certificates. I will need to research whether something like that exists today.

We can note it as a feature request, but just to be clear, an API to replace certificates has a higher chance of being released first.

The only supported option you have today is using a Windows-based backup server. Today, with VSA, you will not be able to fulfill your cybersecurity requirements.

Best,
Fabian

[eider]

It will impact backup server and Web UI, since the certificates are not split yet. We have plans for this, but I cannot share any ETA.

Please add another vote. I was evaluating VSA by moving my homelab from Windows v12 to VSA (no migration, just fresh install) and ran into pretty big issue when trying to provision VSA with internal CA certificate - the docs say that backup server certificate must be CA, but the same certificate is used for Web UI. After provisioning certificate I was unable to log in to Web UI anymore from Firefox due to MOZILLA_PKIX_ERROR_CA_CERT_USED_AS_END_ENTITY.

[Loosus456]

Thanks for sharing the use case. That would mean we need to build a tool for our appliances that can retrieve certificates, install them, and reboot services. It would also logically require an industry standard to request and retrieve certificates. I will need to research whether something like that exists today.

The only one -- and it isn't a released standard yet -- would be DNS-PERSIST-01. DNS-01 would also work, but it requires a lot more plumbing to continually update DNS records, and some DNS hosts do not support API access. HTTP-01 is not a good candidate because most enterprises cannot and will not allow inbound traffic from the public internet to their private/internal servers.

In any case, you need to separate the web UI certificate from the other components. That should be your higher priority because that is just bad design, quite frankly. That means one private-key leak would endanger the security of the whole Veeam stack.