Comprehensive data protection for all workloads
Post Reply
markus
Novice
Posts: 4
Liked: never
Joined: Nov 26, 2014 8:41 am
Full Name: Markus Hiebaum
Contact:

CBT Job performance problem

Post by markus »

Hi all,

i've run into a performance problem with my backup job.

My infrastructure is 4 vmware hosts (esx 5.5) connected to a hp eva 6300 storage. The backup repository is on a separate physical server with a 10gbit uplink. I'm running a virtual machine (Windows server core 2012R2) as backup proxy.

My backup job contains 26 virtual machines and is using the virtual backup proxy in transport mode "virtual appliance". Using direct SAN Access is not allowed by IT policies/management decision. We are using incremental backups (CBT).

The problem is that data transfer (means reading and transferring CBT data) itself is really fast, but most of the time of the backup job is required for mounting disks, creating snapshots, removing snapshots and so on. That means I get 130mb/s processing rate in "virtual appliance" mode and 60mb/s in "network mode", but nevertheless backups take actually longer in "virtual appliance mode" because preparation of the vm takes longer.

For example, my main file server vm needs 8 minutes and 45 seconds backup time, but only 4 minutes 15 seconds of this time are used for data transfer, rest is preparation/post-processing.

Any idea how to get shorter backup times?

I've seen that the backup job processes only one VM at a time. But the backup proxy and repository are configured for 4 concurrent jobs. Is it possible to make the backup job process 4 VMs at the same time or is it useful to create more than one backup job and let several jobs run at the same time?

Thank you for your help!
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: CBT Job performance problem

Post by Shestakov »

Hi Markus,
Could you please provide bottleneck jobs` statistics for both "virtual appliance" and "network" modes?
The possible reason of "virtual appliance" working slower than "network" mode is a lack of backup proxy CPU resources.
markus wrote:I've seen that the backup job processes only one VM at a time. But the backup proxy and repository are configured for 4 concurrent jobs. Is it possible to make the backup job process 4 VMs at the same time or is it useful to create more than one backup job and let several jobs run at the same time?
Yes, you can set several concurrent tasks for backup proxies to leverage parallel data processing. Note that each task in a job requires one CPU core. Consider this value when configuring job settings and backup infrastructure components settings.
Creating more than one backup job using same backup proxy in a same time doesn`t help. You can rather add proxy servers.
Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Job performance problem

Post by Vitaliy S. »

markus wrote:For example, my main file server vm needs 8 minutes and 45 seconds backup time, but only 4 minutes 15 seconds of this time are used for data transfer, rest is preparation/post-processing.
Do you see it for all VMs using application-aware image processing? Do you have indexing enabled as well?
markus
Novice
Posts: 4
Liked: never
Joined: Nov 26, 2014 8:41 am
Full Name: Markus Hiebaum
Contact:

Re: CBT Job performance problem

Post by markus »

Application Aware processing and indexing is both disabled.
Also deduplication is off and compression is optimal.

bottleneck statistics:

Virtual Appliance: Source 99% -> Proxy 40% -> Network 6% -> Target 4%
Network Mode: Source 98% -> Proxy 17% -> Network 0% -> Target 1%

The physical Veeam Server has 12 cores installed, the virtual backup proxy uses 4 virtual cores, the vmware host has 8 physical cores availabe.

@Shestakov: The knowledge base link you posted says that parallel processing is on by default. It is off in my Installation (maybe because this was upgraded from 6.0->6.5->7.0->8.0. I will try and enable this setting and run another full backup test in my maintenance window at night.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Job performance problem

Post by Vitaliy S. »

Though it is unrelated to your problem, but is there any reason why you have disabled deduplication and application-aware image processing options? Both are recommended and considered to be best practice configuration.
markus
Novice
Posts: 4
Liked: never
Joined: Nov 26, 2014 8:41 am
Full Name: Markus Hiebaum
Contact:

Re: CBT Job performance problem

Post by markus »

Vitaliy S. wrote:Though it is unrelated to your problem, but is there any reason why you have disabled deduplication and application-aware image processing options? Both are recommended and considered to be best practice configuration.
The problem is, my backup window is very short (2h) and both options are documented as making backup slower. But I will try both to see how they work in our environment.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Job performance problem

Post by Vitaliy S. »

They will most likely increase your job duration, but in order to have application consistent backup of SQL Server/Exchange etc. it is recommended to use these options.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: CBT Job performance problem

Post by Shestakov »

Are you comparing jobs using same backup proxy? Otherwise the comparison is not correct.

Since you have Source=99% as a bottleneck, despite parallel processing is going to speed up your job, it will not become much faster. Thanks.
markus
Novice
Posts: 4
Liked: never
Joined: Nov 26, 2014 8:41 am
Full Name: Markus Hiebaum
Contact:

Re: CBT Job performance problem

Post by markus »

My dedicated Veeam server can not do "virtual appliance mode" but I can compare the backup proxies in Network mode. But I'm quite sure that the physical machine is faster (more and faster cores, more RAM), but I thought that "virtual appliance mode" is faster than network mode.
I will try all suggested changes in the next few days, one change per day (starting with activating parallel processing today).

Having source as a (severe) bottleneck was expected, EVA6300 is quite outdated, 2x 4Gbit FC uplink is not much and some of the VMs are on near line 7200rpm hard drives which are straightforward horrible painful slow.

Thank you for your help!
Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 179 guests