Cloud Director certificate renewal

[Amarokada]

Hi

Today I did two things.

1. Updated the internal management and http services certificate on VCD 10.6.0.1. This was required to facilitate the upgrade to 10.6.1 due to extra SANs needed in the cert for the Postgresql DB upgrade stage.

2. Upgraded VCD to 10.6.1

Since doing this Veeam will not communicate with the VCD server. Normally in this situation we just run through the properties of the VCD entry in Veeam and when it gets to the credentials screen it would pop up asking us to accept the new cert. Now though when it gets to the "Checking Certificate" part it takes a while and then comes back with the error "Failed to validate the VMware Cloud Director cell certificate. This may be caused by the presence of a load balancer. Skip the certificate verification?"

If I skip it then backups will not work, and a rescan of the VCD entry will succeed for the discovered vCenter and ESXi hosts (disks and volumes), but fails for the VCD entry itself and "Host discovery".

Now we do have the NSX-T normal load balancer in front of the cells and this is only a layer 4 LB so doesn't do anything at HTTPs level, it just balances the cell IPs to the LB IP. When opening the portal page on the Veeam server in a browser it opens fine and we see the new valid certificate, however Veeam seems to be trying to do something else.

Is it possible to find a log on what it's trying to do during this credentials check page? I need to get this working again asap.

[Amarokada]

It seems Veeam has changed the way it is updating certs from VCD. Instead of reading the cert from the shared portal page, it instead connects to that, resolves the north facing IPs of the cells and then tries to connect to the cells directly. This isn't possible in a service provider VCD setup as the north facing IPs are behind a L4 load balancer and the cells themselves have no gateway for routing without using the LB. The cells have secondary vNICs with IPs on the backend management network for management tasks. This is the part of the logs I found showing this.

[07.02.2025 07:23:44.874] <36> Info (3) [Vcd] Resolving vCD cells information from vCD: 'https://portal.xxx.yyy:443'
[07.02.2025 07:23:44.874] <36> Info (3) [Vcd] Executing QueryResultCellRecordType
[07.02.2025 07:23:44.905] <36> Info (3) vCD cell: 'Name: 'vcdcell01.xxx.internal', Version: '10.6.1.24532667', PrimaryIp: '192.168.14.161', BuildDate: '29/01/2025 13:19:21', IsActive: 'True''
[07.02.2025 07:23:44.905] <36> Info (3) vCD cell: 'Name: 'vcdcell03.xxx.internal', Version: '10.6.1.24532667', PrimaryIp: '192.168.14.163', BuildDate: '29/01/2025 13:19:21', IsActive: 'True''
[07.02.2025 07:23:44.905] <36> Info (3) vCD cell: 'Name: 'vcdcell02.xxx.internal', Version: '10.6.1.24532667', PrimaryIp: '192.168.14.162', BuildDate: '29/01/2025 13:19:21', IsActive: 'True''

[07.02.2025 07:24:06.036] <01> Error (3) Failed to get certificate from: https://192.168.14.161:443 (System.Exception)

Veeam never has to talk to VCD cells directly, in all backup tasks it just goes to the shared public portal page, so I'm not sure why this behaviour has changed. If it's just when it tries to update the certificate for the Veeam database then I need another way to do this because right now everything is broken.

[Amarokada]

I have a ticket open with Veeam support on this now too.

Case #07596435

[Amarokada]

Still no answer to this. I'm sure this is a design bug and won't get fixed quickly, so I'm looking for an alternative way to update the certificate Veeam has for the VCD entry in its database.

[Mildur]

Hi Rob

I see that you have an open discussion with our support team. Personally I'm not aware of any changes, and I also only see your case so far. But I will keep an eye on the case and may reach out to our QA team next week if necessary.

If you want to escalate the ticket to the next tier, then I suggest to start an escalation of the case with the "Talk to a Manager" option. This will also make sure that the case gets escalated to our QA team if required.

Best,
Fabian

[Amarokada]

Hi

We got this fixed yesterday. After some back and forth with support they crafted a SQL update query that entered a new certificate thumbprint against the VCD host entry in the Veeam SQL database.

It does seem that the certificate update logic unnecessarily tries to communicate with the VCD cells to verify the new cert and if it can't reach one it won't update the cert. This is different to when you add a new VCD to Veeam as that will take the cert from the master portal and you can cancel the bit where it attempts to validate it with a cell and it still records the cert thumbprint in the DB.

Once I had the method to inject the fix manually I was able to work out what was needed in the SQL code for our other environment which is setup the same (just different URLs/IDs etc, and cert)

I believe Veeam will take a look at this cert update logic and look to avoid connecting to the cells for verification.

[Mildur]

Hi Rob

Thank you for providing the feedback. I'm glad support was able to provide you with a solution.
We will review your case together with QA and make adjustments to our product where needed.

Best,
Fabian