Prabhash, please review the full list of ports
required for Veeam B&R components.
That said, in case VM is not available on the network, FLR via VIX is performed, so in fact direct connection to it is not required. However, in case of restoring a lot of data, it might indeed be slower.
I'm not sure about your topology and Veeam B&R version you're using, however keep in mind that v9 introduces mount server
component specifically to avoid data being transferred back and forth during FLR when backups are mounted on the backup server itself.