Yes, that's expected behavior and, unfortunately, has nothing to do with Veeam required permissions. The document that you've used, refers to global granular permissions, these permissions should be assigned to a Datacenter or a vCenter Server level. I have also tried to assign it to particular objects (as you did), and it didn't work, as vSphere API requires access to the entire infrastructure tree (based on the feedback from VMware team).
In order to solve your case, I believe vCloud Director should be used, as it has multi-tenant feature built-in. Other than that, I cannot find any other feasible solution right now.
Hope it helps!