After 12.2 upgrade VBR can no longer login to vCloud Director

[DanielJ]

Directly after upgrading to 12.2, VBR started claiming that the username or password to one of our vCD platforms was incorrect (vCD version 10.4.2.21954589). When we go to the /provider/login page and use the same credentials, it works, and we can see that the account is a system administrator. VBR can login to another vCD platform which is on 10.3 without problems. I don't have any servers left on 12.1, but I have one on 11.0.1.1261 P20230227, and when I try from there, I get the expected error message about SDK versions (since we haven't installed https://www.veeam.com/kb4352). So there seems to be a specific problem with VBR 12.2 and vCD 10.4. One of my admins raised a ticket for this but we haven't got much attention. I wanted to check here if someone else has had the same problem after upgrading to 12.2.

Ticket: 07410151

[mjr.epicfail]

Did you try to got to Veeam / VCD / properties/ next next finish? Most of the times this helps (its the default solution for certificate changes).

[DanielJ]

There has been no certificate change. The only thing that changed is that VBR was upgraded to 12.2. When I try to click through the properties I can't get past the credential stage because of "wrong password".

[mjr.epicfail]

Yes, I understand this is not a certificate change, was just mentioning that this process also fixes that.
Could be an underlying issue, but if you dont restore this connection, backups will fail. imho that constitutes as a P1 case.

Did you by any chance check the debug logs of VCD?

[MarkBoothmaa]

Did you upgrade the plugin in vCD? I've had no issues with 12.2 and our vCD environment.

[DanielJ]

We don't use the vCD plugin. What version is your vCD?

[tyler.jurgens]

Our VCD is on 10.4.2.21954589 and Veeam 12.2 - no VCD plugin either. No alarms we can see referencing any errors.

[DanielJ]

Thank you. That's very surprising. Whenever we test that particular combination it's a dead stop (same for 10.5 which we also have tested now).

[mjr.epicfail]

Dumb question, did you already tried to change password to a much simpler one, to rule out any special characters.

[JTT]

Directly after upgrading to 12.2, VBR started claiming that the username or password to one of our vCD platforms was incorrect (vCD version 10.4.2.21954589). When we go to the /provider/login page and use the same credentials, it works, and we can see that the account is a system administrator. VBR can login to another vCD platform which is on 10.3 without problems. I don't have any servers left on 12.1, but I have one on 11.0.1.1261 P20230227, and when I try from there, I get the expected error message about SDK versions (since we haven't installed https://www.veeam.com/kb4352). So there seems to be a specific problem with VBR 12.2 and vCD 10.4. One of my admins raised a ticket for this but we haven't got much attention. I wanted to check here if someone else has had the same problem after upgrading to 12.2.

Ticket: 07410151

We had the same issue in our environment. Veeam support didnt help. Tried changing the password, also no affect. Our own coworkers helpd to solve the issue. Turns out, that our vCD (v 10.5)) load balancer didnt have our Veeam subnet allowed for provider API but somehow for older Veeam versions this was not needed.
Can You get to the /provider/login vCD webpage from Veeam server, does the login come up?
With our case, it just gave an error message, but after allowing Veeam subnet in load balancer, it started working and also no more Veeam failed login error.

[IvanK]

Hello,

In version 12.2, we introduced support for VCD 10.6, one notable change in VCD was that it doesn't support VCD API versions older than 36.0. Because of that a new VCD login method was added, which uses a different endpoint for authentication: cloudapi/1.0.0/sessions/provider

As you've correctly pointed out, this endpoint is currently blocked by your WAF to be accessed by VBR, resulting in authentication errors for VCD integration.

This change also affects VCD versions 10.4 and 10.5 because the new login method is now applied to all VCD integrations that support API version 36.0 and newer (i.e., from version 10.3 onwards).

Please let me know if you have any questions on that.

[DanielJ]

Yes, it turned out to be the same problem with the load balancer here too. Was this change documented somewhere? This is exactly what I would like to find in the "What's new" document. It would have saved us hours of headaches.

And yes @JTT, the /provider/login page worked all the time.

[DanielJ]

This change also affects VCD versions 10.4 and 10.5 because the new login method is now applied to all VCD integrations that support API version 36.0 and newer (i.e., from version 10.3 onwards).

Why not let installations with 10.3, 10.4 and 10.5 fall back to using the older method as long as it is still supported by VMware? What was the great hurry in dropping support for it?

[JTT]

Hello,

In version 12.2, we introduced support for VCD 10.6, one notable change in VCD was that it doesn't support VCD API versions older than 36.0. Because of that a new VCD login method was added, which uses a different endpoint for authentication: cloudapi/1.0.0/sessions/provider

As you've correctly pointed out, this endpoint is currently blocked by your WAF to be accessed by VBR, resulting in authentication errors for VCD integration.

This change also affects VCD versions 10.4 and 10.5 because the new login method is now applied to all VCD integrations that support API version 36.0 and newer (i.e., from version 10.3 onwards).

Please let me know if you have any questions on that.

Where was this documented? With our support ticket, even Veeam didnt know what changed:

Hello ,
thank you for the update. I've checked the moment regarding vCD changes in v12.2, but do not see any changes in communication apart from introduction of vCD 10.6 support (https://www.veeam.com/veeam_backup_12_2 ... new_wn.pdf). With that client version was updated:
I could suspect a possible issue with TLS certificate fingerprints refresh, as this is updated during the Veeam upgrade:
- https://helpcenter.veeam.com/docs/backu ... ml?ver=120
- https://helpcenter.veeam.com/docs/backu ... ml?ver=120
Could you please let me know what exact changes were made to the network, that helped to fix the issue? Did you have to enable all traffic, disable FW, etc.?
Was this change reverted after the issue was fixed? If no, I'd suggest trying to return back to initial network configuration to see if there any changes. As I'd suspect that it should be fine now.

[mjr.epicfail]

Why not let installations with 10.3, 10.4 and 10.5 fall back to using the older method as long as it is still supported by VMware? What was the great hurry in dropping support for it?

10.3 works, I think you meant 10.4 and later.

[IvanK]

Yes, it turned out to be the same problem with the load balancer here too. Was this change documented somewhere? This is exactly what I would like to find in the "What's new" document. It would have saved us hours of headaches.

And yes JTT, the /provider/login page worked all the time.

All required connections from VBR server to VCD should be documented here. Blocking access to the particular VCD endpoints is something outside of our control or documented recommendations.

Regarding this change in general, each release brings thousands of tweaks and updates, if we publish them all in 'What's new', this document would quickly become unreadable.

Why not let installations with 10.3, 10.4 and 10.5 fall back to using the older method as long as it is still supported by VMware? What was the great hurry in dropping support for it?

Using the same login process across all versions that support API 36.0 and above reduces the combinatorics of VCD versions and login methods that need to be supported from a Dev/QA perspective.

With our support ticket, even Veeam didnt know what changed:

Can you please provide the support case number?

[DanielJ]

All required connections from VBR server to VCD should be documented here. Blocking access to the particular VCD endpoints is something outside of our control or documented recommendations.

Only in this case there was a completely new endpoint involved which you did not mention anywhere, thereby wasting time for service providers and your own support.

Regarding this change in general, each release brings thousands of tweaks and updates, if we publish them all in 'What's new', this document would quickly become unreadable.

I would very much like to read such a list. Maybe not in that brief document, but you should provide a complete list of changes somewhere, so that we (or at least those who so choose) can be fully informed on changes that have an impact on our particular environment.

[JTT]

Can you please provide the support case number?

Case # 07400264

[RubinCompServ]

Turns out, that our vCD (v 10.5)) load balancer didnt have our Veeam subnet allowed for provider API but somehow for older Veeam versions this was not needed.

What load balancer are you using?

[IvanK]

Only in this case there was a completely new endpoint involved which you did not mention anywhere, thereby wasting time for service providers and your own support.

I would very much like to read such a list. Maybe not in that brief document, but you should provide a complete list of changes somewhere, so that we (or at least those who so choose) can be fully informed on changes that have an impact on our particular environment.

Can you elaborate on your VCD setup - do you use NSX LB for VCD? And in your scenario, the VCD internal virtual service IP was was not accessible from the VBR server, correct?
If that's the case, I agree with you, we'll consider adding the mention of VBR access requirement to both internal and external virtual service IP for VCD.
A couple of references for other readers:
https://blogs.vmware.com/cloudprovider/ ... ancer.html
https://youtu.be/6rM2MxRH2FY?si=0b9um9ZQBndZTEXT

Regarding publishing all thousands of changes made to each version, unfortunately, that's not going to happen, as that would require an incredible amount of resources to proofread everything and filter out unnecessary info and duplicates before publishing it.

Case # 07400264

Thanks. Discussed this case with the support engineer.

[benthomas]

Regarding publishing all thousands of changes made to each version, unfortunately, that's not going to happen, as that would require an incredible amount of resources to proofread everything and filter out unnecessary info and duplicates before publishing it.

I don't think it's too much to ask for some notification or inclusion when the means of authenticating with a supported product are changed. Especially when it's a product that's typically published on the internet and the vendors best practice is to lockdown all endpoints access the provider APIs.
Could've been a simple note added to the VCD 10.6 support statement that says from this release VBR needs to be able to talk to the newer /cloudapi endpoint for all versions of VCD starting from 10.4. Nice and easy, and would've helped people be able to look for any changes to how VBR talks to the APIs for VCD.

[IvanK]

Hi Ben,
Agree that this change needs to be documented in the Used Ports / What's new, as this authentication method may communicate to a completely different IP (private service IP never reached by VBR before). But, before doing that, I'd like to get more specifics on the type of the VCD deployments impacted by this change from the @DanielJ

[DanielJ]

What specifics do you need? @IvanK