What's the logic for selecting proxy?

[pkelly_sts]

I have a 24-core physical proxy/repository server with direct SAN access plus 3 x 6 core proxy appliance VMs.

I'm fine tuning a job to get maximum throughput for active fulls to a scale-out repository via per-vm backups and want to start the largest VM first so that can be processing whilst all the smaller VMs get processed along side it.

Looking at last nights first run of this job (before I specified any ordering) I notice that the first few VMs were assigned to the proxy appliance VMs and it seems only after those proxies were assigned that it started using the Direct-attach proxy.

For obvious reasons I'd REALLY like to be backing up the largest VMs via DA proxy with multiple smaller proxies using up the network bandwidth, so what's the logic for deciding which proxy does which VMs? I guess I always assumed that the DA proxy would be used first by default...

[dellock6]

It may sound silly, but better to start from the beginning: can you confirm that VM's processed by the physical process are all processed using [san] mode?

[pkelly_sts]

No such thing as a silly question

It has actually proved very interesting! Yes, I can confirm that, where it was used, the physical proxy ONLY used [san] mode, so no network failover etc.

However, whilst going through each of the VMs to check how they were processed, I actually found that very little was processed by the physical server at all, majority were via proxy VMs.

Hopefully the below table survives formatting but basically I've numbered the proxy VMs 1, 2 & 3 and the physical "P". First column is the start order (as shown in job status) second column is number of disks on that VM, third is the proxies that were used to process it.

In total I have 26 concurrent tasks available: 3 x 6 & 1 x 8 (SAN attached has but in looking at the layout below, the physical was only used for 5 disks total!

OrderDisks Proxies
1 1 3
2 1 3
3 2 1, 2
4 2 3, 1
5 2 2, 3
6 2 1,2
7 3 3, 3, 1
8 2 1, 2
9 2 3, 1
10 2 2, 3
11 3 1, 2, P
12 1 3
13 3 1, 2, P
All the above started processing at the same time, all the below started at various times as the concurrent tasks were free'd up.
14 3 P, 2, 2
15 1 3
16 1 1
17 3 3, 3, 1
18 3 P, 1, 2
19 4 2, 3, 1, 2
20 2 3, 2
21 3 1, 1, P
22 3 3, 2, 3 (this is our largest VM with a 1Tb disk 80% used!)
23 2 1, 3

So, my question is even more relevant now, I can't see why the physical is hardly used at all! :-/

[pkelly_sts]

ARGH! formatting failed.
First column (single digit) is processing order
Second column (single digit) is number of configured disks
Remainder is ID's of the proxies used

[foggy]

What is the max concurrent tasks number configured for the physical one?

[pkelly_sts]

8 for physical & 6 each for VMs.

[Gostev]

I actually found that very little was processed by the physical server at all, majority were via proxy VMs.

Perhaps some LUNs are not zoned to the physical proxy. Honestly, you should just open a support case and let them review the setup and debug logs, as hot add proxies should not be used as long a direct SAN proxy capable of processing the given VM is available.

[Vitaliy S.]

Perhaps some LUNs are not zoned to the physical proxy.

...and in order to check that, you can try to run this report > Unmapped Datastore LUNs

[pkelly_sts]

I can confirm that all LUNS are definitely mapped, I've been backing up these VMs with another job for a long time (but have other issues with that job which I suspect will require us to start the job from scratch so, whilst I have the space & time window, I'm pre-empting that ATM anyway).

One thing it /could/ be though is that this (physical) proxy is also a destination proxy for incoming backup copies from our other site but having checked the logs I'm pretty sure that job hadn't kicked off yet.

I also checked the original job (as described above) and that's using the san mode first exactly as I'd expect.

This new job is currently chained to run after the above job so I know that at least these two don't run at the same time.

What I'm going to do to ensure I rule things out is switch the timing for the jobs tonight (so the new job runs prior to the old job) and review in the morning (unless I'm sad enough to log in tonight at start time & see what it's doing live - which is highly likely )

[pkelly_sts]

Looks like it did the same again tonight & I made sure it was the only job running.
I then configured it to ONLY use the SAN attached physical proxy & re-ran it and that also ran fine (which proves san attach is working fine as I suspected).

I've now opened case 1810024 & supplied log files...

[DaveWatkins]

I've seen this same behavior before. Here I'd rather all the backups are done through the SAN attached proxy and the vVM proxies are only used for replication, but there seems to be no built in preference for that to occur. Jobs appeared to be randomly associated with any available proxy no matter it's transport capabilities. In the end I hardcoded all the backup jobs to only use the SAN attached proxy. Not ideal but it's working as expected now

[pkelly_sts]

I'm not sure we're talking about quite the same thing?

The preference I believe is built-in, but by design it will (should) use all available concurrency anyway. Bear in mind that concurrency is also limited to how many tasks can be carried out to your storage. So, in theory you can match your concurrent proxy "slots" with your concurrent repository "slots" to have an element of control. But to be honest, in my case, I configure the disk concurrency to what I think will maximise throughput without over-saturating it, then I configure MORE than enough proxy slots to meet that concurrency so with 20 disk concurrency slots available and 26 proxy concurrency slots available, I'd always expect a few proxy slots to be idle anyway, but in any case I want to maximise throughput to disk as much as possible.

The crazy thing is, this new job I have configured is to potentially replace an existing job that's been in place for a long time. The proxy selection for that job works perfectly, exactly as you would want: all the concurrent tasks are first consumed by the physical SAN-attached proxy, then the rest are distributed across the proxy VMs.

I've also confirmed to support that the 6 x VMFS volumes are already manually configured on the proxy (not auto-configured) and disabled automount is also configured (both have been this way for well over a year).