How does Veeam choose the proxy for backup?

[JasonPGTF]

I want to know, when a backup was executed how's the behavior of Veeam in selecting which proxy server to use.
On my site, I actually have 2 VM Proxy servers & 1 Physical proxy server. I notice by checking on most of my backup jobs, all of them are using the physical proxy server only. Anyone know why is it such way?

[Egor Yakovlev]

Hi Jason!
"For each VM in the job, backup server classifies all backup proxies available to the job into 3 groups by access type they have to the VM's disks: SAN (1), hot add (2) and network (3). Proxy with least tasks currently assigned is picked from group 1, and if responding, VM processing task is assigned to this proxy. Otherwise, the next least busy proxy is selected from group 1. If all available proxies group 1 are already running max number of concurrent tasks, selection process switches to group 2 and repeats. Group 3 is only used if groups 1 and 2 have no responding proxies. For group 3, proxy subnet is also considered when picking the best proxy."
Veeam Forums - VMware : [FAQ] Frequently Asked Questions
/Cheers!

[foggy]

Hi Jason, do you mean that all the slots on the physical proxy are occupied and VMs are waiting in the queue but still don't use available VM proxies? I'd check whether your virtual proxies are actually capable of hotadd as this transport mode has a number of limitations.

[JasonPGTF]

Hi Foggy,

Yes, even though all the slots for physical proxy are full, the VM's are still waiting in the queue & not utilizing VM proxies. I had several backup jobs run at the same time slot and all of those jobs are queuing for the physical proxy.
How do I check if my virtual proxies are capable of hotadd?

Just to add, this physical proxy was just added recently, previously I only had 2 virtual proxy and the VM's are utilizing this 2 Vproxy. Once I've added the physical proxy, Veeam basically abandon those 2 virtual proxies.

[foggy]

Could you please check what transport mode was effectively used by the jobs before adding the physical proxy? You can open the statistics for any job session under the History tab, select some VM in the list to the left there and look up for the record containing proxy server name. You will see either [hotadd] or [nbd] tag next to it.

To check if the proxy is capable of hotadd, just review the limitations listed in the mentioned user guide section. Unless one of those applies, any virtual proxy can use hotadd.

[JasonPGTF]

Hi Foggy, where do you find the history tab at statistics? When I open statistics, I don't see any history tab there.
When i click Report, the generated report doesn't show the details such as which proxy server it uses.

[Egor Yakovlev]

History View can be found in bottom left corner, alongside "Home", "Inventory", "Backup Infrastructure" etc.
There, you can find your job sessions history - select Jobs(Backup) node, filter by job name and get any session that was ran before new physical proxy introduction date. You will see something like below with transport mode virtual proxies used in brackets at each virtual disk: [nbd] means network, [hotadd] means virtual appliance.

/Cheers!

[JasonPGTF]

Thanks a lot Egor! Manage to find the tiny little history button haha.
Here's the backup job before the physical proxy was added, it is using hotadd as you can see.
This is the VM proxy, using hotadd


This is the physical proxy, it's using nfs


There's also a quite a big differences in the processing rate between the 2 proxies as well.

[Egor Yakovlev]

Hi Jason.
1. It is a good practice to remove sensitive corporate data from screenshots when shared in public(hostnames, domains, IP addresses, LUN IDs etc). I have edited your pictures.
2. Hotadd and NFS datastores has a long history. Low hotadd performance in your case could be quite expected.
3. Direct NFS Transport mode is considered better(due to the fact data is read from NFS datastore directly, avoiding unnecessary VM snapshot operations linked with hotadd mode), thus job always picks that proxy on "Automatic" proxy selection path. That is 100% expected and good for your backup infrastructure. Until Physical Proxy can process VM data, it will be picked first!
4. In the screenshots I only see a single job with a single VM - to actually evaluate parallel processing and usage of multiple proxies, make sure to run several jobs at a time or combine more VMs to be processed by a single job(that will also give you better deduplication rate!).
/Cheers!

[JasonPGTF]

Thanks Egor for the explanation, it makes everything clearer now

1 last thing, I have a large exchange server VM, about 1TB in size. It's taking about 8hours just to complete the whole backup cycle plus few hours of VM stunning during the VM disk consolidation. Bear in mind it happen on incremental backups as well & often lots of users screaming that their emails can't get through. Due to this I had to disable the backup for this VM, now that I'm seeing positive improvement from a physical proxy, I might try to get a few more old servers and set it up as proxy as well. Do you think it'll be able to help?

Veeam support guys mention this is expected & advice us to use storage snapshot feature, that's why this physical proxy was created in order to test out storage snapshot & surprisingly it give positive impact on other VMs as well.

However now I'm having another headache, my storage guys are doing tech refresh on my storage appliances & they're going for a storage which not certified to integrate with Veeam yet, so I won't be able to configure storage snapshot for it once they migrated over. Which is why I'm thinking if having several physical proxies would help or not.

[Egor Yakovlev]

Having additional proxies will not help if existing one is not fully utilized - as I can see from provided screenshots Bottleneck is determined to be Source, that means thin part is Read operation from production. With Direct NFS Transport mode that is speed of NFS read Proxy<->NFS server - I would check that in the first place, before trying to add more proxies. 73 MB\s processing speed is quite slow in my opinion, as it is half of 1g link bandwidth.

Surely, changing transport mode(and gain performance boost) will change(reduce) overall backup times, leading to smaller VM snapshot size and ease of commit operation at the end.

Storage Snapshots would be of great help with big VMs and overall performance too, consider it for the future. Meanwhile performance investigation is going, to avoid your Exchange server being un-protected, you can use in-guest Agent-based backup with Veeam Agent for Windows - it has no links to VMware, so backup happens without VM snapshot, thus there is no stun at the end!

So summarizing the plan:
- investigate current physical proxy performance(check NICs, QoS, perform out-of-Veeam NFS read tests, load it with more simultaneously protected VMs)
- unless you have very strict and clear reason to keep "1 VM per Job" you should group VMs together within same Job. You can start with smaller packs(say 4-6VMs per Job) to load your proxy more and observe results. Ideally you can go up to 30 VMs per Job easily with all-default settings.
- Proxy Server processes 1xVMDK disk per 1xCPU Core it has - that is, if your physical proxy server has 16 cores, you can potentially read 16 VMDK disks simultaneously no matter which VM they belong to! Let's put your existing proxy server to the top of performance!
- Don't leave Exchange unprotected. Backup with Veeam Agent for Windows.

/Cheers!

[JasonPGTF]

Thanks again Egor!

You're a saviour
Will take note of each of your suggestion!

[Matts N]

Hello,
Thought I'd jump into this thread, since I have a different challenge compared to OP, but it is still related to proxy selection.

In my case, it's a matter of using virtual proxy using NFS Direct to backup VM's residing on a Nutanix system. This is in accordance with Nutanix best practice for using Veeam, at least based on information found on Nutanix website. The idea is to use the proxy to only back up VM's residing on the same host as the proxy, to avoid any unnecessary network traffic between Nutanix nodes. Problem is, with the selection method described above, that won't work, there is no way to limit a proxy to only back up VM's on same host. Yes, I can create a job that only backs up a certain host, but that would mean a vMotion moves a VM to a different job, creating a new backup chain.

Am I missing some nifty settng in Veeam, a setting that would limit proxy selection to only use proxy on same host as VM?

[Egor Yakovlev]

Hi Matts!

We have a special registry key specifically made for such cases!

Code: Select all

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Key: EnableSameHostDirectNFSMode
Type: REG_DWORD
Default value: 0 (disabled)

Value = 1 – when proxy A is available on the same host, Veeam Backup & Replication will leverage it. If proxy A is busy, Veeam Backup & Replication will wait for its availability; if it becomes unavailable, another directNFS proxy (proxy B) will be used.
Value = 2 - when proxy A is available on the same host, Veeam Backup & Replication will leverage it. If proxy A is busy, Veeam Backup & Replication will wait for its availability; if it becomes unavailable, Veeam Backup & Replication will switch to NBD mode.

/Cheers!

[JasonPGTF]

Hi Egor,

I just tested to use agent to backup my large exchange. Unfortunately agent too causes performance impact to the VM, because it's hogging up the disk i/o & network bandwidth throughout the backup process. I notice during the backup period all my emails are at least 10mins delayed, issue went away once the job is stopped. I pull a report from my VCenter & indeed there's a big spike in the network & disk I/O during the backup process.

[Egor Yakovlev]

Is it a clustered Exchange DAG infrastructure? If so, You can backup a passive node and have all Veeam "magic" from it - from full node down to a single email recovery with no impact on Active production exchange node.
/Cheers!

[Matts N]

Hi Matts!

We have a special registry key specifically made for such cases!

Thanks Egor! I'm not surprised there was a registry key to help with this.