-
- Novice
- Posts: 4
- Liked: never
- Joined: Nov 26, 2014 8:41 am
- Full Name: Markus Hiebaum
- Contact:
CBT Job performance problem
Hi all,
i've run into a performance problem with my backup job.
My infrastructure is 4 vmware hosts (esx 5.5) connected to a hp eva 6300 storage. The backup repository is on a separate physical server with a 10gbit uplink. I'm running a virtual machine (Windows server core 2012R2) as backup proxy.
My backup job contains 26 virtual machines and is using the virtual backup proxy in transport mode "virtual appliance". Using direct SAN Access is not allowed by IT policies/management decision. We are using incremental backups (CBT).
The problem is that data transfer (means reading and transferring CBT data) itself is really fast, but most of the time of the backup job is required for mounting disks, creating snapshots, removing snapshots and so on. That means I get 130mb/s processing rate in "virtual appliance" mode and 60mb/s in "network mode", but nevertheless backups take actually longer in "virtual appliance mode" because preparation of the vm takes longer.
For example, my main file server vm needs 8 minutes and 45 seconds backup time, but only 4 minutes 15 seconds of this time are used for data transfer, rest is preparation/post-processing.
Any idea how to get shorter backup times?
I've seen that the backup job processes only one VM at a time. But the backup proxy and repository are configured for 4 concurrent jobs. Is it possible to make the backup job process 4 VMs at the same time or is it useful to create more than one backup job and let several jobs run at the same time?
Thank you for your help!
i've run into a performance problem with my backup job.
My infrastructure is 4 vmware hosts (esx 5.5) connected to a hp eva 6300 storage. The backup repository is on a separate physical server with a 10gbit uplink. I'm running a virtual machine (Windows server core 2012R2) as backup proxy.
My backup job contains 26 virtual machines and is using the virtual backup proxy in transport mode "virtual appliance". Using direct SAN Access is not allowed by IT policies/management decision. We are using incremental backups (CBT).
The problem is that data transfer (means reading and transferring CBT data) itself is really fast, but most of the time of the backup job is required for mounting disks, creating snapshots, removing snapshots and so on. That means I get 130mb/s processing rate in "virtual appliance" mode and 60mb/s in "network mode", but nevertheless backups take actually longer in "virtual appliance mode" because preparation of the vm takes longer.
For example, my main file server vm needs 8 minutes and 45 seconds backup time, but only 4 minutes 15 seconds of this time are used for data transfer, rest is preparation/post-processing.
Any idea how to get shorter backup times?
I've seen that the backup job processes only one VM at a time. But the backup proxy and repository are configured for 4 concurrent jobs. Is it possible to make the backup job process 4 VMs at the same time or is it useful to create more than one backup job and let several jobs run at the same time?
Thank you for your help!
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: CBT Job performance problem
Hi Markus,
Could you please provide bottleneck jobs` statistics for both "virtual appliance" and "network" modes?
The possible reason of "virtual appliance" working slower than "network" mode is a lack of backup proxy CPU resources.
Creating more than one backup job using same backup proxy in a same time doesn`t help. You can rather add proxy servers.
Thanks.
Could you please provide bottleneck jobs` statistics for both "virtual appliance" and "network" modes?
The possible reason of "virtual appliance" working slower than "network" mode is a lack of backup proxy CPU resources.
Yes, you can set several concurrent tasks for backup proxies to leverage parallel data processing. Note that each task in a job requires one CPU core. Consider this value when configuring job settings and backup infrastructure components settings.markus wrote:I've seen that the backup job processes only one VM at a time. But the backup proxy and repository are configured for 4 concurrent jobs. Is it possible to make the backup job process 4 VMs at the same time or is it useful to create more than one backup job and let several jobs run at the same time?
Creating more than one backup job using same backup proxy in a same time doesn`t help. You can rather add proxy servers.
Thanks.
-
- VP, Product Management
- Posts: 27196
- Liked: 2741 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: CBT Job performance problem
Do you see it for all VMs using application-aware image processing? Do you have indexing enabled as well?markus wrote:For example, my main file server vm needs 8 minutes and 45 seconds backup time, but only 4 minutes 15 seconds of this time are used for data transfer, rest is preparation/post-processing.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Nov 26, 2014 8:41 am
- Full Name: Markus Hiebaum
- Contact:
Re: CBT Job performance problem
Application Aware processing and indexing is both disabled.
Also deduplication is off and compression is optimal.
bottleneck statistics:
Virtual Appliance: Source 99% -> Proxy 40% -> Network 6% -> Target 4%
Network Mode: Source 98% -> Proxy 17% -> Network 0% -> Target 1%
The physical Veeam Server has 12 cores installed, the virtual backup proxy uses 4 virtual cores, the vmware host has 8 physical cores availabe.
@Shestakov: The knowledge base link you posted says that parallel processing is on by default. It is off in my Installation (maybe because this was upgraded from 6.0->6.5->7.0->8.0. I will try and enable this setting and run another full backup test in my maintenance window at night.
Also deduplication is off and compression is optimal.
bottleneck statistics:
Virtual Appliance: Source 99% -> Proxy 40% -> Network 6% -> Target 4%
Network Mode: Source 98% -> Proxy 17% -> Network 0% -> Target 1%
The physical Veeam Server has 12 cores installed, the virtual backup proxy uses 4 virtual cores, the vmware host has 8 physical cores availabe.
@Shestakov: The knowledge base link you posted says that parallel processing is on by default. It is off in my Installation (maybe because this was upgraded from 6.0->6.5->7.0->8.0. I will try and enable this setting and run another full backup test in my maintenance window at night.
-
- VP, Product Management
- Posts: 27196
- Liked: 2741 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: CBT Job performance problem
Though it is unrelated to your problem, but is there any reason why you have disabled deduplication and application-aware image processing options? Both are recommended and considered to be best practice configuration.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Nov 26, 2014 8:41 am
- Full Name: Markus Hiebaum
- Contact:
Re: CBT Job performance problem
The problem is, my backup window is very short (2h) and both options are documented as making backup slower. But I will try both to see how they work in our environment.Vitaliy S. wrote:Though it is unrelated to your problem, but is there any reason why you have disabled deduplication and application-aware image processing options? Both are recommended and considered to be best practice configuration.
-
- VP, Product Management
- Posts: 27196
- Liked: 2741 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: CBT Job performance problem
They will most likely increase your job duration, but in order to have application consistent backup of SQL Server/Exchange etc. it is recommended to use these options.
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: CBT Job performance problem
Are you comparing jobs using same backup proxy? Otherwise the comparison is not correct.
Since you have Source=99% as a bottleneck, despite parallel processing is going to speed up your job, it will not become much faster. Thanks.
Since you have Source=99% as a bottleneck, despite parallel processing is going to speed up your job, it will not become much faster. Thanks.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Nov 26, 2014 8:41 am
- Full Name: Markus Hiebaum
- Contact:
Re: CBT Job performance problem
My dedicated Veeam server can not do "virtual appliance mode" but I can compare the backup proxies in Network mode. But I'm quite sure that the physical machine is faster (more and faster cores, more RAM), but I thought that "virtual appliance mode" is faster than network mode.
I will try all suggested changes in the next few days, one change per day (starting with activating parallel processing today).
Having source as a (severe) bottleneck was expected, EVA6300 is quite outdated, 2x 4Gbit FC uplink is not much and some of the VMs are on near line 7200rpm hard drives which are straightforward horrible painful slow.
Thank you for your help!
I will try all suggested changes in the next few days, one change per day (starting with activating parallel processing today).
Having source as a (severe) bottleneck was expected, EVA6300 is quite outdated, 2x 4Gbit FC uplink is not much and some of the VMs are on near line 7200rpm hard drives which are straightforward horrible painful slow.
Thank you for your help!
Who is online
Users browsing this forum: Baidu [Spider], Google [Bot] and 69 guests