There will always be a bottleneck somewhere in the entire processing chain, so if you're satisfied with the overall performance and meet your backup window, there's nothing to worry about. However, if you'd still want to improve performance, in your case you indeed can offload the processing from the Hyper-V hosts by switching to off-host proxies. Please review the corresponding user guide section
for details on how to configure them.
Btw, what is the reason of the "one job per VM" type of setup?