Is Veeam picking the best proxy?

[pufferdude]

I have a replication job that runs every hour, that replicates two critical VMs from the SAN to local ESXi storage on one host. The Veeam server is physical and connected to the san over dedicated lan ports. However, when the job runs it always picks a *VM* proxy (which is running on the same host that has the local storage) instead of the Veeam server itself. Is this because the hotadd to the proxy VM is more efficient vs. the Veeam server pulling the VM off the SAN and (I guess?) sending it over NBD to the ESXi host's local storage?

Just trying to make sure I understand the logic. It kind of bugs me that the VM proxy gets chosen, because on a job that runs this frequently, the hotadd process on the proxy VM can take a significant amount of time to prepare the VMs vs. coming off the SAN (and thus no hotadd). Not sure why hotadd is so slow (maybe THAT is my real issue?) but on a job that runs once an hour, about 10min each hour is taken just by the hot-add and snapshot processes vs. actually transferring data, which is typically only a few gig that's changed. Maybe it's normal for the VM prep to take longer than the actual data transfer? (In this case).

[foggy]

Jim, just to make sure, are you talking about source proxy or target one being selected for hotadd? Please select the particular VM in the list in the job statistics window and look for the source and target proxy selection lines.

[pufferdude]

The same proxy VM is chosen for both source and target on this job, and the Veeam server itself (connected to the SAN, so it at least has direct access to the source VM, but not the destination local ESXi storage) doesn't get used (for this job.)

[foggy]

Correct, typically, direct SAN is preferred over hotadd. However in cases where a single hotadd proxy can serve both as source and target one, it is picked up over direct SAN. At least Veeam B&R v7 had this logic in place, I need to check whether this works in v8.

You may explicitly select the source proxy in the job settings, btw.

[pufferdude]

Great, thanks for the info. I will switch the proxy on this job and compare the runtime to using the hotadd VM proxy, and stick with whichever one is faster.

[pufferdude]

I changed the replication job to use the Veeam server as proxy, but it still chose the VM proxy for the target and the results are almost identical. Now that I've done this test, I see something I didn't notice before: The Proxy VM that is being used does NOT have the datastore for the host's local storage mounted normally, and THIS is what takes the most time on this job. It takes 2-3 minutes for the hotadd process to mount the *datastore* to this VM (vs. just mounting a disk from a datastore that it already has mounted), which to me seems like a long time for that task. Once the datastore is attached, the drives for the replication target mount quickly.

I wonder if I can configure this VM to keep that datastore attached, even though it uses no resources on that datastore (other than when doing a replication job to that storage)? That would eliminate this delay, I think. Maybe I just need to make a small disk for this VM on that storage, to keep it attached continually

[foggy]

You can also try to explicitly select target proxy as well, so that NBD mode be used that does not have this mounting overhead.

[pufferdude]

So this is really interesting. I put the replication job back on automatic proxy (vs. forcing it to be the Veeam server as source) AND then set the VM Proxy it had been choosing to NBD mode instead of automatic. When the job next ran with this config, the Veeam server was chosen as BOTH source and target proxy, presumably because the "math" changed and the VM proxy no longer being able to do hotadd was not as desirable for the algorithm.

Now the interesting stats:
Job at 9:07 (VM Proxy doing hotadd for both source and target): Data read: 534.5MB. Data Transferred: 500.6MB. Duration: 13:55
Job at 10:07 (Veeam Proxy as both source and target): Data read: 10.6GB. Data Transferred: 7.6GB. Duration: 6:21

So... the job had over 15x data to transfer (start of work day, lots of stuff happening on those two VMs) and did it in 1/2 the time. That's crazy. Part of the explanation is that the Veeam proxy was able to process both guests at the same time... so clearly my VM Proxy is underpowered and I need to add a CPU to it as it only processes one thing at a time.... but we're still talking about 15x data in that same time period, HAD they run consecutively vs. together.

I'm probably gonna leave the job as is and not dig in much further, since it's not really a problem. But it sure is curious as to why my Proxy VM is so stinking slow, particularly the process of doing hotadd. Maybe I just need to beef it up with more CPU and memory!