I run a similar setup for our small sites and I've found that you just can't rely on VMware USB passthrough for any kind of performance. It works for small backups, but once you start to get a large amount of data the processing rate is just too slow to get the job done in any kind of reasonable backup window.
I've switched to using an old physical machine, with a USB 3 card, as a Veeam proxy for RDX backups. It doesn't have to be anything fancy. Just an old desktop machine will do.