Number of VM's per job

[mcrdr85]

Hi everyone,

Currently, I have 3 jobs. 1 exchange job, one file server job and then everything else. The everything else job has 67 with 2.5 TB of raw data and it taking a backup window of about 10-12 hours. Its just taking way too long. The only positive is the incremental size. Oh and I should mention I do the jobs in order so only one job runs at a time. And we only have one proxy.

How many VM's\data per job is optimal? Is splitting the one job into 4 about right? Is one proxy good enough? VM's per proxy?

I'm just trying to make the backup more efficient and close the backup window. Any suggestions on how to make my environment better would greatly be appreciated.

Thanks Chris!

[Gostev]

Hi Chris, what product version are you using?

[mcrdr85]

7.0 enterprise

[Gostev]

Assuming it all goes into the same backup repository, with this amount of data I would just put everything into the single job, and enabled parallel processing.

If you are still going to be missing your backup window after that, then may be its time to add another proxy server (unless it is backup repository that cannot keep up, which it will be clear from the bottleneck analysis stats).

[mcrdr85]

Yes, this is all going to the same repository, a DataDomain DD630 via CIFS.

I just enabled parallel processing. so i will see how that goes!

The bottleneck statistics look like this (took from one job but im sure its the avg):

Source 95%
Proxy 29%
Network 5%
Target 22%

[Gostev]

Judging on this statistics, parallel processing should make a huge difference.

[foggy]

What transport mode is being used during processing (look at the [tag] next to the proxy name in the job statistics window, after selecting the particular VM to the left)?

[mcrdr85]

Are you talking about [hotadd]? That's the only thing i see besides that I'm using CBT

[foggy]

Yes, I was referring to that. So let's see whether parallel processing does the job.

[mcrdr85]

The job just finished at it still took 11 hrs and 15 minutes. Still better than the night before at 13 hrs and 49 minutes. I only have max concurrent tasks set to 2 though. So ill bump it up to 4 and see if thats any better.

The bottleneck statistic did change though:

Source 78%
Proxy 31%
Network 24%
Target 19%

[foggy]

Make sure you check the max concurrent tasks setting for both proxy and repository.

[mcrdr85]

okay.... I set them both to 4!

[mcrdr85]

still took 13 hrs to complete after making those changes!

Source 78%
Proxy 17%
Network 39%
Target 34%

[dellock6]

In all your tests the bottleneck is always source. Can you give us more details about the storage you are using in production, its connections to ESXi servers, the sizing of the proxies, etc?

Luca.

[mcrdr85]

I'm using a DataDomain DD630 for storage and it has 2 gig nics that are bonded that is hooked up to the same core of switches as the esx's. Connected to the proxy via cifs. The proxy has 8gb of ram and 2 quad core cpu's. the datastores for the esx's are hp lefthand sans connected via iSCSI to the esx's (again on the same switches).

[tsightler]

Are you sure CBT is working? A couple of screenshots would probably help shed some light on the situation.

[mcrdr85]

I would attach some screenshoots but it seems like i can't! And CBT looks to be working!

[tsightler]

OK, can you at least post some job information, like Read/Processed/Transmitted, etc.

[mcrdr85]

Duration 13:17:20
Processing rate: 15MB/s
Bottleneck: (see above)
Processed: 2.5 TB
Read: 258.0 GB
Transferred:77.1GB (3.3x)
Success: 67 VMs

[foggy]

From these stats, the actual data transfer took about 5 hours only (from the overall >13). Could you please try to force network mode for this job and see how it goes?

[Giacomo_N]

From these stats, the actual data transfer took about 5 hours only (from the overall >13). Could you please try to force network mode for this job and see how it goes?

Did you mean in the proxy transport mode?

[foggy]

Yes. Proxy server settings > Transport mode.

[mcrdr85]

ok right now i have it set to automatic so i will set it to network mode and see how that goes!

[Giacomo_N]

Yes. Proxy server settings > Transport mode.

Alexandaer, but if i understand correclty is the worse transport method..

[veremin]

Typically, yes. But in some environments it can show pretty decent speed. OP’s situation looks like the case where switching proxy mode might, indeed, help. Thanks.

[foggy]

Generally, yes. However, hotadd has some overhead required to hotadd disks to the proxy server, while network mode doesn't, which could make it faster on the fast links. Here is a good topic on that.

[Giacomo_N]

Generally, yes. However, hotadd has some overhead required to hotadd disks to the proxy server, while network mode doesn't, which could make it faster on the fast links. Here is a good topic on that.

I've read the post and i've understand. Thank you

[mcrdr85]

This still did not help... its still take 11 hours to complete!!! It looks like im going to have to split these up into 3 jobs as this is taking away to long to complete!

Source 54%
Proxy 8%
Network 47%
Target 14%

[foggy]

Is the processing rate is comparable for all your VMs or are there any VMs that are processed much slower/longer than the others?

[mcrdr85]

its seems like its all over the board... some are as high has 9 mb/s and some are as low as 800kb/s.

I see one VM with a 20GB hard drive took an hour to process which seems WAY too long!