What's the logic for selecting proxy?

VMware specific discussions

What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Thu May 26, 2016 9:32 am

I have a 24-core physical proxy/repository server with direct SAN access plus 3 x 6 core proxy appliance VMs.

I'm fine tuning a job to get maximum throughput for active fulls to a scale-out repository via per-vm backups and want to start the largest VM first so that can be processing whilst all the smaller VMs get processed along side it.

Looking at last nights first run of this job (before I specified any ordering) I notice that the first few VMs were assigned to the proxy appliance VMs and it seems only after those proxies were assigned that it started using the Direct-attach proxy.

For obvious reasons I'd REALLY like to be backing up the largest VMs via DA proxy with multiple smaller proxies using up the network bandwidth, so what's the logic for deciding which proxy does which VMs? I guess I always assumed that the DA proxy would be used first by default...
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Re: What's the logic for selecting proxy?

Veeam Logoby dellock6 » Thu May 26, 2016 9:35 am

It may sound silly, but better to start from the beginning: can you confirm that VM's processed by the physical process are all processed using [san] mode?
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Thu May 26, 2016 10:19 am

No such thing as a silly question :)

It has actually proved very interesting! Yes, I can confirm that, where it was used, the physical proxy ONLY used [san] mode, so no network failover etc.

However, whilst going through each of the VMs to check how they were processed, I actually found that very little was processed by the physical server at all, majority were via proxy VMs.

Hopefully the below table survives formatting but basically I've numbered the proxy VMs 1, 2 & 3 and the physical "P". First column is the start order (as shown in job status) second column is number of disks on that VM, third is the proxies that were used to process it.

In total I have 26 concurrent tasks available: 3 x 6 & 1 x 8 (SAN attached has 8) but in looking at the layout below, the physical was only used for 5 disks total!

OrderDisks Proxies
1 1 3
2 1 3
3 2 1, 2
4 2 3, 1
5 2 2, 3
6 2 1,2
7 3 3, 3, 1
8 2 1, 2
9 2 3, 1
10 2 2, 3
11 3 1, 2, P
12 1 3
13 3 1, 2, P
All the above started processing at the same time, all the below started at various times as the concurrent tasks were free'd up.
14 3 P, 2, 2
15 1 3
16 1 1
17 3 3, 3, 1
18 3 P, 1, 2
19 4 2, 3, 1, 2
20 2 3, 2
21 3 1, 1, P
22 3 3, 2, 3 (this is our largest VM with a 1Tb disk 80% used!)
23 2 1, 3

So, my question is even more relevant now, I can't see why the physical is hardly used at all! :-/
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Re: What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Thu May 26, 2016 10:20 am

ARGH! formatting failed.
First column (single digit) is processing order
Second column (single digit) is number of configured disks
Remainder is ID's of the proxies used
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Re: What's the logic for selecting proxy?

Veeam Logoby foggy » Thu May 26, 2016 12:05 pm

What is the max concurrent tasks number configured for the physical one?
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Thu May 26, 2016 12:07 pm

8 for physical & 6 each for VMs.
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Re: What's the logic for selecting proxy?

Veeam Logoby Gostev » Thu May 26, 2016 1:20 pm

pkelly_sts wrote:I actually found that very little was processed by the physical server at all, majority were via proxy VMs.

Perhaps some LUNs are not zoned to the physical proxy. Honestly, you should just open a support case and let them review the setup and debug logs, as hot add proxies should not be used as long a direct SAN proxy capable of processing the given VM is available.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: What's the logic for selecting proxy?

Veeam Logoby Vitaliy S. » Thu May 26, 2016 1:53 pm

Gostev wrote:Perhaps some LUNs are not zoned to the physical proxy.

...and in order to check that, you can try to run this report > Unmapped Datastore LUNs
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Thu May 26, 2016 2:00 pm

I can confirm that all LUNS are definitely mapped, I've been backing up these VMs with another job for a long time (but have other issues with that job which I suspect will require us to start the job from scratch so, whilst I have the space & time window, I'm pre-empting that ATM anyway).

One thing it /could/ be though is that this (physical) proxy is also a destination proxy for incoming backup copies from our other site but having checked the logs I'm pretty sure that job hadn't kicked off yet.

I also checked the original job (as described above) and that's using the san mode first exactly as I'd expect.

This new job is currently chained to run after the above job so I know that at least these two don't run at the same time.

What I'm going to do to ensure I rule things out is switch the timing for the jobs tonight (so the new job runs prior to the old job) and review in the morning (unless I'm sad enough to log in tonight at start time & see what it's doing live - which is highly likely :) )
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Re: What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Thu May 26, 2016 7:58 pm 1 person likes this post

Looks like it did the same again tonight & I made sure it was the only job running.
I then configured it to ONLY use the SAN attached physical proxy & re-ran it and that also ran fine (which proves san attach is working fine as I suspected).

I've now opened case 1810024 & supplied log files...
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Re: What's the logic for selecting proxy?

Veeam Logoby DaveWatkins » Thu May 26, 2016 9:29 pm

I've seen this same behavior before. Here I'd rather all the backups are done through the SAN attached proxy and the vVM proxies are only used for replication, but there seems to be no built in preference for that to occur. Jobs appeared to be randomly associated with any available proxy no matter it's transport capabilities. In the end I hardcoded all the backup jobs to only use the SAN attached proxy. Not ideal but it's working as expected now
DaveWatkins
Expert
 
Posts: 252
Liked: 61 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: What's the logic for selecting proxy?

Veeam Logoby pkelly_sts » Fri May 27, 2016 8:19 am

I'm not sure we're talking about quite the same thing?

The preference I believe is built-in, but by design it will (should) use all available concurrency anyway. Bear in mind that concurrency is also limited to how many tasks can be carried out to your storage. So, in theory you can match your concurrent proxy "slots" with your concurrent repository "slots" to have an element of control. But to be honest, in my case, I configure the disk concurrency to what I think will maximise throughput without over-saturating it, then I configure MORE than enough proxy slots to meet that concurrency so with 20 disk concurrency slots available and 26 proxy concurrency slots available, I'd always expect a few proxy slots to be idle anyway, but in any case I want to maximise throughput to disk as much as possible.

The crazy thing is, this new job I have configured is to potentially replace an existing job that's been in place for a long time. The proxy selection for that job works perfectly, exactly as you would want: all the concurrent tasks are first consumed by the physical SAN-attached proxy, then the rest are distributed across the proxy VMs.

I've also confirmed to support that the 6 x VMFS volumes are already manually configured on the proxy (not auto-configured) and disabled automount is also configured (both have been this way for well over a year).
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 18 guests