Host-based backup of VMware vSphere VMs.
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Number of VM's per job

Post by mcrdr85 »

Hi everyone,

Currently, I have 3 jobs. 1 exchange job, one file server job and then everything else. The everything else job has 67 with 2.5 TB of raw data and it taking a backup window of about 10-12 hours. Its just taking way too long. The only positive is the incremental size. Oh and I should mention I do the jobs in order so only one job runs at a time. And we only have one proxy.

How many VM's\data per job is optimal? Is splitting the one job into 4 about right? Is one proxy good enough? VM's per proxy?

I'm just trying to make the backup more efficient and close the backup window. Any suggestions on how to make my environment better would greatly be appreciated.

Thanks Chris!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Number of VM's per job

Post by Gostev »

Hi Chris, what product version are you using?
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

7.0 enterprise
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Number of VM's per job

Post by Gostev » 1 person likes this post

Assuming it all goes into the same backup repository, with this amount of data I would just put everything into the single job, and enabled parallel processing.

If you are still going to be missing your backup window after that, then may be its time to add another proxy server (unless it is backup repository that cannot keep up, which it will be clear from the bottleneck analysis stats).
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

Yes, this is all going to the same repository, a DataDomain DD630 via CIFS.

I just enabled parallel processing. so i will see how that goes!

The bottleneck statistics look like this (took from one job but im sure its the avg):

Source 95%
Proxy 29%
Network 5%
Target 22%
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Number of VM's per job

Post by Gostev »

Judging on this statistics, parallel processing should make a huge difference.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

What transport mode is being used during processing (look at the [tag] next to the proxy name in the job statistics window, after selecting the particular VM to the left)?
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

Are you talking about [hotadd]? That's the only thing i see besides that I'm using CBT
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

Yes, I was referring to that. So let's see whether parallel processing does the job.
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

The job just finished at it still took 11 hrs and 15 minutes. Still better than the night before at 13 hrs and 49 minutes. I only have max concurrent tasks set to 2 though. So ill bump it up to 4 and see if thats any better.

The bottleneck statistic did change though:

Source 78%
Proxy 31%
Network 24%
Target 19%
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

Make sure you check the max concurrent tasks setting for both proxy and repository.
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

okay.... I set them both to 4!
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

still took 13 hrs to complete after making those changes!

Source 78%
Proxy 17%
Network 39%
Target 34%
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Number of VM's per job

Post by dellock6 »

In all your tests the bottleneck is always source. Can you give us more details about the storage you are using in production, its connections to ESXi servers, the sizing of the proxies, etc?

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

I'm using a DataDomain DD630 for storage and it has 2 gig nics that are bonded that is hooked up to the same core of switches as the esx's. Connected to the proxy via cifs. The proxy has 8gb of ram and 2 quad core cpu's. the datastores for the esx's are hp lefthand sans connected via iSCSI to the esx's (again on the same switches).
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Number of VM's per job

Post by tsightler »

Are you sure CBT is working? A couple of screenshots would probably help shed some light on the situation.
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

I would attach some screenshoots but it seems like i can't! And CBT looks to be working!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Number of VM's per job

Post by tsightler »

OK, can you at least post some job information, like Read/Processed/Transmitted, etc.
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

Duration 13:17:20
Processing rate: 15MB/s
Bottleneck: (see above)
Processed: 2.5 TB
Read: 258.0 GB
Transferred:77.1GB (3.3x)
Success: 67 VMs
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

From these stats, the actual data transfer took about 5 hours only (from the overall >13). Could you please try to force network mode for this job and see how it goes?
Giacomo_N
Enthusiast
Posts: 92
Liked: 16 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Italy
Contact:

Re: Number of VM's per job

Post by Giacomo_N »

foggy wrote:From these stats, the actual data transfer took about 5 hours only (from the overall >13). Could you please try to force network mode for this job and see how it goes?
Did you mean in the proxy transport mode?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

Yes. Proxy server settings > Transport mode.
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

ok right now i have it set to automatic so i will set it to network mode and see how that goes!
Giacomo_N
Enthusiast
Posts: 92
Liked: 16 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Italy
Contact:

Re: Number of VM's per job

Post by Giacomo_N »

foggy wrote:Yes. Proxy server settings > Transport mode.
Alexandaer, but if i understand correclty is the worse transport method..
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Number of VM's per job

Post by veremin »

Typically, yes. But in some environments it can show pretty decent speed. OP’s situation looks like the case where switching proxy mode might, indeed, help. Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

Generally, yes. However, hotadd has some overhead required to hotadd disks to the proxy server, while network mode doesn't, which could make it faster on the fast links. Here is a good topic on that.
Giacomo_N
Enthusiast
Posts: 92
Liked: 16 times
Joined: Feb 15, 2013 1:56 pm
Full Name: Giacomo
Location: Italy
Contact:

Re: Number of VM's per job

Post by Giacomo_N »

foggy wrote:Generally, yes. However, hotadd has some overhead required to hotadd disks to the proxy server, while network mode doesn't, which could make it faster on the fast links. Here is a good topic on that.
I've read the post and i've understand. Thank you
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

This still did not help... its still take 11 hours to complete!!! It looks like im going to have to split these up into 3 jobs as this is taking away to long to complete!

Source 54%
Proxy 8%
Network 47%
Target 14%
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Number of VM's per job

Post by foggy »

Is the processing rate is comparable for all your VMs or are there any VMs that are processed much slower/longer than the others?
mcrdr85
Influencer
Posts: 17
Liked: never
Joined: Aug 15, 2013 6:51 pm
Full Name: Chris Powell
Contact:

Re: Number of VM's per job

Post by mcrdr85 »

its seems like its all over the board... some are as high has 9 mb/s and some are as low as 800kb/s.

I see one VM with a 20GB hard drive took an hour to process which seems WAY too long!
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 102 guests