The support team were able point out the issue in my backup infra. Since i didnt had a DNS server in my infra, the vCenter (VCSA)was not able to resolve the VBR EM url to download the plugins from EM. So i manually added an entry in the VCSA host file to resolve the EM hostname. Then everything worked perfectly!
I had a similar issue in my second installation (actually in production). But this time i had a DNS server, and so the issue was not with the hostname resolution. This time there was a firewall in between the VCSA and VBR EM, and as per the Veeam documentation (USED PORTS section) the traffic flow happens only from EM to vCenter during the plugin installation. But after checking the firewall logs i found out that the VCSA also initiates traffic to the EM after the plugin install is initiated from VBR EM, and it's being blocked. Also there is no mention of such a traffic flow in the user guide. After allowing that port (9443) VCSA --> EM, the plugin got installed perfectly.
I'm quoting the reply from Support Team
If there is a firewall, it might just block the traffic and installation may fail. Both sides initiate connection. First of all EM connects to vCenter, then vCenter connects to EM.