Comprehensive data protection for all workloads
Post Reply
pjc
Novice
Posts: 4
Liked: never
Joined: Aug 28, 2014 10:26 am
Full Name: Philip Curry
Contact:

Target Bottleneck Issue

Post by pjc »

Hello,

Sorry to bother you all here but it feels like my last resort.

I have a pretty small setup:

One 12 core server that also has a 25TB array on it. I use this as a proxy and the main repository for backup jobs. Backup jobs just come out of vCenter, my SAN doesn't support pulling the backups directly out of storage.
Two Proxy VM's with a couple of cores each.
A Thecus N12000 Pro enterprise NAS box that is connected via iSCSI to and is used for Replica jobs.

Everything, including the iSCSI is 1Gbit and the total amount of data is minimal. With 28 days of daily reverse incremental backups there is only about 15TB of Data on the main repository.

The problem is that it literally takes 8-9 hours to complete ONLY the backup jobs meaning that I don't even have time to run daily replicas.

On all the jobs, regardless of if it's a Backup or Replica job it lists the Target as the bottleneck and the transfer speed is about 5-8MB/sec. This seems incredibly slow, even if I am able to run a lot of simultaneous jobs due to having the 3 proxies.

I am reasonably new to Veeam so I am sure there is going to be some stuff that I am missing. If you need me to provide any information to help diagnose I am happy to do so.

Another thing is that I can't run backups during the day because if I do it makes the server in question run incredibly slow, for example outlook locks up all the time if I am backing up Exchange.

I am running Veeam B&R 7 (Build 871) on Windows Server 2008 R2.

All help much appreciated and I do have a valid support contract.

"Your support case number is 00665768"

Regards

//PJC
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Target Bottleneck Issue

Post by Shestakov »

Hello PJC and welcome to the forum!

Could you provide the job`s statistics? I`m interested in "Load: Source _% > Proxy _% > Network _% > Target _%"
In general reversed incremental takes 3 times as much I/O as forward incremental backup, so switching to that mode can be useful. It`s pretty common to have target as a bottleneck using reversed incremental backups with not high-end backup repository.
PJC wrote:Another thing is that I can't run backups during the day because if I do it makes the server in question run incredibly slow, for example outlook locks up all the time if I am backing up Exchange.
It`s probably caused by snapshot taking/removal mechanism. We have a big related topic describing the issue.
Thanks.
pjc
Novice
Posts: 4
Liked: never
Joined: Aug 28, 2014 10:26 am
Full Name: Philip Curry
Contact:

Re: Target Bottleneck Issue

Post by pjc »

Yeah I just got the same answer from Veeam support. Since they are running on Raid5 and Raid50 (the storage respectively). It will run poorly because it's doing reverse incremental.

What's the advantage of reverse incremental? What's the implication if I just change that option on a job?

Finally is this even an option on a replica job?

Thanks

//PJC

*edit*

28/10/2014 06:42:05 :: Load: Source 8% > Proxy 1% > Network 43% > Target 82%
28/10/2014 03:07:28 :: Load: Source 13% > Proxy 4% > Network 2% > Target 98%
28/10/2014 00:59:37 :: Load: Source 7% > Proxy 2% > Network 28% > Target 66%


Couple of examples
pjc
Novice
Posts: 4
Liked: never
Joined: Aug 28, 2014 10:26 am
Full Name: Philip Curry
Contact:

Re: Target Bottleneck Issue

Post by pjc »

Oh and I don't even mean during snapshots, I just mean while the backup is running, before the snapshot is even trying to be removed, the performance is heavily reduced (300-400ms ping spikes and stuff like file shares dropping for a period of time or outlook becoming unresponsive). Makes it almost impossible to do backups during the day.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Target Bottleneck Issue

Post by Shestakov »

pjc wrote:What's the advantage of reverse incremental? What's the implication if I just change that option on a job?
Reversed incremental job keeps only 1 Full backup at a time, while Forward incremental requires a bit more repository space, but can increase the backup process.
You don`t need any special preparations for switching from reversed to forward incremental.
Another valuable tip is to restrict number of concurrent writing tasks for the same repository.
pjc wrote:Finally is this even an option on a replica job?
No, it`s not. Roughly speaking replication job works like forward incremental backup job.
pjc wrote:Oh and I don't even mean during snapshots, I just mean while the backup is running, before the snapshot is even trying to be removed, the performance is heavily reduced (300-400ms ping spikes and stuff like file shares dropping for a period of time or outlook becoming unresponsive). Makes it almost impossible to do backups during the day.
Are those interrupts current during the backup process or occur while some particular operations are processing? Have you reviewed logs with Veeam support team to investigate the problem?
pjc
Novice
Posts: 4
Liked: never
Joined: Aug 28, 2014 10:26 am
Full Name: Philip Curry
Contact:

Re: Target Bottleneck Issue

Post by pjc »

I have sent some logs to Veeam support but probably not everything. I have arranged a session to have support look at my system so I will just go from there. You can delete this thread if you like.

Regards
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Target Bottleneck Issue

Post by Shestakov »

Philip,

We don`t delete threads, since it can be useful for other who experience same issues.
However, it would be nice of you to post here a cause of the problem with your Exchange server once you figure it out with the support team.
Thank you!
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 221 guests