Discussions specific to the VMware vSphere hypervisor
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Number of VM's per job

Post by mcrdr85 » Oct 08, 2013 7:24 pm

Hi everyone,

Currently, I have 3 jobs. 1 exchange job, one file server job and then everything else. The everything else job has 67 with 2.5 TB of raw data and it taking a backup window of about 10-12 hours. Its just taking way too long. The only positive is the incremental size. Oh and I should mention I do the jobs in order so only one job runs at a time. And we only have one proxy.

How many VM's\data per job is optimal? Is splitting the one job into 4 about right? Is one proxy good enough? VM's per proxy?

I'm just trying to make the backup more efficient and close the backup window. Any suggestions on how to make my environment better would greatly be appreciated.

Thanks Chris!

Gostev
SVP, Product Management
Posts: 24292
Liked: 3329 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Number of VM's per job

Post by Gostev » Oct 08, 2013 7:29 pm

Hi Chris, what product version are you using?

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 08, 2013 7:45 pm

7.0 enterprise

Gostev
SVP, Product Management
Posts: 24292
Liked: 3329 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Number of VM's per job

Post by Gostev » Oct 08, 2013 7:51 pm 1 person likes this post

Assuming it all goes into the same backup repository, with this amount of data I would just put everything into the single job, and enabled parallel processing.

If you are still going to be missing your backup window after that, then may be its time to add another proxy server (unless it is backup repository that cannot keep up, which it will be clear from the bottleneck analysis stats).

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 08, 2013 8:32 pm

Yes, this is all going to the same repository, a DataDomain DD630 via CIFS.

I just enabled parallel processing. so i will see how that goes!

The bottleneck statistics look like this (took from one job but im sure its the avg):

Source 95%
Proxy 29%
Network 5%
Target 22%

Gostev
SVP, Product Management
Posts: 24292
Liked: 3329 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Number of VM's per job

Post by Gostev » Oct 08, 2013 8:36 pm

Judging on this statistics, parallel processing should make a huge difference.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 08, 2013 8:40 pm

What transport mode is being used during processing (look at the [tag] next to the proxy name in the job statistics window, after selecting the particular VM to the left)?

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 08, 2013 8:55 pm

Are you talking about [hotadd]? That's the only thing i see besides that I'm using CBT

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 08, 2013 9:01 pm

Yes, I was referring to that. So let's see whether parallel processing does the job.

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 09, 2013 2:25 pm

The job just finished at it still took 11 hrs and 15 minutes. Still better than the night before at 13 hrs and 49 minutes. I only have max concurrent tasks set to 2 though. So ill bump it up to 4 and see if thats any better.

The bottleneck statistic did change though:

Source 78%
Proxy 31%
Network 24%
Target 19%

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 09, 2013 2:47 pm

Make sure you check the max concurrent tasks setting for both proxy and repository.

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 09, 2013 3:08 pm

okay.... I set them both to 4!

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 10, 2013 3:33 pm

still took 13 hrs to complete after making those changes!

Source 78%
Proxy 17%
Network 39%
Target 34%

dellock6
Veeam Software
Posts: 5680
Liked: 1600 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Number of VM's per job

Post by dellock6 » Oct 10, 2013 4:47 pm

In all your tests the bottleneck is always source. Can you give us more details about the storage you are using in production, its connections to ESXi servers, the sizing of the proxies, etc?

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 10, 2013 5:10 pm

I'm using a DataDomain DD630 for storage and it has 2 gig nics that are bonded that is hooked up to the same core of switches as the esx's. Connected to the proxy via cifs. The proxy has 8gb of ram and 2 quad core cpu's. the datastores for the esx's are hp lefthand sans connected via iSCSI to the esx's (again on the same switches).

tsightler
VP, Product Management
Posts: 5344
Liked: 2189 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Number of VM's per job

Post by tsightler » Oct 10, 2013 7:29 pm

Are you sure CBT is working? A couple of screenshots would probably help shed some light on the situation.

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 10, 2013 8:09 pm

I would attach some screenshoots but it seems like i can't! And CBT looks to be working!

tsightler
VP, Product Management
Posts: 5344
Liked: 2189 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Number of VM's per job

Post by tsightler » Oct 10, 2013 9:28 pm

OK, can you at least post some job information, like Read/Processed/Transmitted, etc.

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 11, 2013 12:39 pm

Duration 13:17:20
Processing rate: 15MB/s
Bottleneck: (see above)
Processed: 2.5 TB
Read: 258.0 GB
Transferred:77.1GB (3.3x)
Success: 67 VMs

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 11, 2013 1:16 pm

From these stats, the actual data transfer took about 5 hours only (from the overall >13). Could you please try to force network mode for this job and see how it goes?

Giacomo_N
Enthusiast
Posts: 68
Liked: 6 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Vicenza, Italy
Contact:

Re: Number of VM's per job

Post by Giacomo_N » Oct 11, 2013 3:17 pm

foggy wrote:From these stats, the actual data transfer took about 5 hours only (from the overall >13). Could you please try to force network mode for this job and see how it goes?
Did you mean in the proxy transport mode?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 11, 2013 3:19 pm

Yes. Proxy server settings > Transport mode.

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 11, 2013 3:23 pm

ok right now i have it set to automatic so i will set it to network mode and see how that goes!

Giacomo_N
Enthusiast
Posts: 68
Liked: 6 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Vicenza, Italy
Contact:

Re: Number of VM's per job

Post by Giacomo_N » Oct 11, 2013 3:25 pm

foggy wrote:Yes. Proxy server settings > Transport mode.
Alexandaer, but if i understand correclty is the worse transport method..

veremin
Product Manager
Posts: 16497
Liked: 1371 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Number of VM's per job

Post by veremin » Oct 11, 2013 3:37 pm

Typically, yes. But in some environments it can show pretty decent speed. OP’s situation looks like the case where switching proxy mode might, indeed, help. Thanks.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 11, 2013 3:45 pm

Generally, yes. However, hotadd has some overhead required to hotadd disks to the proxy server, while network mode doesn't, which could make it faster on the fast links. Here is a good topic on that.

Giacomo_N
Enthusiast
Posts: 68
Liked: 6 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Vicenza, Italy
Contact:

Re: Number of VM's per job

Post by Giacomo_N » Oct 14, 2013 12:29 pm

foggy wrote:Generally, yes. However, hotadd has some overhead required to hotadd disks to the proxy server, while network mode doesn't, which could make it faster on the fast links. Here is a good topic on that.
I've read the post and i've understand. Thank you

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 14, 2013 12:39 pm

This still did not help... its still take 11 hours to complete!!! It looks like im going to have to split these up into 3 jobs as this is taking away to long to complete!

Source 54%
Proxy 8%
Network 47%
Target 14%

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy » Oct 15, 2013 4:25 pm

Is the processing rate is comparable for all your VMs or are there any VMs that are processed much slower/longer than the others?

mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 » Oct 15, 2013 4:34 pm

its seems like its all over the board... some are as high has 9 mb/s and some are as low as 800kb/s.

I see one VM with a 20GB hard drive took an hour to process which seems WAY too long!

Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests