Serious bottleneck with gateway selection for tape jobs

[mjroberts]

We have noticed a seemingly serious flaw in the "backup to tape" gateway selection process. We have a SOBR with 6 extents and many backup to tape jobs. Because our repository does not have a built-in datamover, it relies on a gateway to get the data from the repo to the tape server.

What we are seeing is the same gateway is being selected for all backup to tape jobs and tasks (regardless of the repo) and there is seemingly no intelligence in the selection process, and no load-balancing. We have 20+ proxies that are eligible to be gateways, but the jobs always select the same one and it results in a serious bottleneck. The end result is not being able to push enough data to the tape server to be able to really take advantage of more than a couple tape drives and it is slower than it should be.

Simple example: each extent in the SOBR has 25Gb NICs, each proxy/gateway has at least one 10Gb NIC, the physical tape server has a team of 10Gb NICs. What happens is 1 gateway is used and 50% of its bandwidth is used reading from the repo and the other 50% is used sending data to the tape server. The end result, the tape server only receives 50% of what it could/should if multiple gateways were selected.

I worked this issue with support (Case #07561465) for months before finally being told there is nothing they can do except put in a feature request. I am not sure how this is just a feature request and not a serious design flaw. As I understand it, the selection process for backup to disk jobs is intelligent.

[david.domask]

Hi Mike,

Thank you for sharing the case number, and sorry to hear about the challenges.

Reviewing the case history and discussion, a few clarifying points:

- With "Automatic" gateway selection for repositories that require gateways, Veeam will try to start the datamover on the tape server. For dedicated gateway groups, the first gateway from the list is chosen
- Backup job gateway handling is indeed different from Tape handling currently
- Repositories that can host our datamovers (veeamagent.exe) do not experience this issue as there is no need for a gateway

Your use case and request are noted, as it's a somewhat common configuration for a SOBR to have performance tier extents requiring gateways; as I see it, Support was able to suggest a workaround with the current tape job selection logic to mitigate the current behavior.

[mjroberts]

Support suggested breaking our proxies/gateways into groups and only allowing certain proxies on certain SOBR extents. It sounds like a possible workaround for us (in this particular environment) because we have a lot of proxies/gateways. In smaller environments it would not be ideal, and it would probably create a bottleneck in the backup jobs. I posted to the forum for the benefit of others and the workaround provided by support may not be a good fit for others. Also, if the tape server is allowed as a gateway, is there a possibility of regular backup jobs selecting it as the gateway (also not ideal)?

[mjroberts]

Hi Mike,

Thank you for sharing the case number, and sorry to hear about the challenges.

Reviewing the case history and discussion, a few clarifying points:

- With "Automatic" gateway selection for repositories that require gateways, Veeam will try to start the datamover on the tape server. For dedicated gateway groups, the first gateway from the list is chosen
- Backup job gateway handling is indeed different from Tape handling currently
- Repositories that can host our datamovers (veeamagent.exe) do not experience this issue as there is no need for a gateway

Your use case and request are noted, as it's a somewhat common configuration for a SOBR to have performance tier extents requiring gateways; as I see it, Support was able to suggest a workaround with the current tape job selection logic to mitigate the current behavior.

Thanks for the reply, David.

We have stretched layer-2 networks and if we set selection to automatic, does VBR have solid logic to prevent the selection of a gateway in the wrong site for regular backup jobs? The storage is not all shared between sites but much of the networking is stretched. Also, not knowing which gateway the tape job considers "first in the list", what would keep tape jobs from selecting one in the wrong site, if it didn't select itself?

How does one know which gateway is considered first in the list, because it is not the first in the GUI in our case?

[david.domask]

Hi Mike,

You're very welcome, though a quick correction on the above as your question seems to merge the logic for automatic and dedicated selection.

With Automatic, Veeam will look to put the datamover on the tape server itself when possible; if you set dedicated gateways per resource that needs a gateway (e.g., SMB shares), then the first gateway in that list will be chosen.

So unfortunately it would require some planning as per Support's suggestion on configuring this to avoid needless cross-site traffic.