Target Bottleneck Issue

[pjc]

Hello,

Sorry to bother you all here but it feels like my last resort.

I have a pretty small setup:

One 12 core server that also has a 25TB array on it. I use this as a proxy and the main repository for backup jobs. Backup jobs just come out of vCenter, my SAN doesn't support pulling the backups directly out of storage.
Two Proxy VM's with a couple of cores each.
A Thecus N12000 Pro enterprise NAS box that is connected via iSCSI to and is used for Replica jobs.

Everything, including the iSCSI is 1Gbit and the total amount of data is minimal. With 28 days of daily reverse incremental backups there is only about 15TB of Data on the main repository.

The problem is that it literally takes 8-9 hours to complete ONLY the backup jobs meaning that I don't even have time to run daily replicas.

On all the jobs, regardless of if it's a Backup or Replica job it lists the Target as the bottleneck and the transfer speed is about 5-8MB/sec. This seems incredibly slow, even if I am able to run a lot of simultaneous jobs due to having the 3 proxies.

I am reasonably new to Veeam so I am sure there is going to be some stuff that I am missing. If you need me to provide any information to help diagnose I am happy to do so.

Another thing is that I can't run backups during the day because if I do it makes the server in question run incredibly slow, for example outlook locks up all the time if I am backing up Exchange.

I am running Veeam B&R 7 (Build 871) on Windows Server 2008 R2.

All help much appreciated and I do have a valid support contract.

"Your support case number is 00665768"

Regards

//PJC

[Shestakov]

Hello PJC and welcome to the forum!

Could you provide the job`s statistics? I`m interested in "Load: Source _% > Proxy _% > Network _% > Target _%"
In general reversed incremental takes 3 times as much I/O as forward incremental backup, so switching to that mode can be useful. It`s pretty common to have target as a bottleneck using reversed incremental backups with not high-end backup repository.

Another thing is that I can't run backups during the day because if I do it makes the server in question run incredibly slow, for example outlook locks up all the time if I am backing up Exchange.

It`s probably caused by snapshot taking/removal mechanism. We have a big related topic describing the issue.
Thanks.

[pjc]

Yeah I just got the same answer from Veeam support. Since they are running on Raid5 and Raid50 (the storage respectively). It will run poorly because it's doing reverse incremental.

What's the advantage of reverse incremental? What's the implication if I just change that option on a job?

Finally is this even an option on a replica job?

Thanks

//PJC

*edit*

28/10/2014 06:42:05 :: Load: Source 8% > Proxy 1% > Network 43% > Target 82%
28/10/2014 03:07:28 :: Load: Source 13% > Proxy 4% > Network 2% > Target 98%
28/10/2014 00:59:37 :: Load: Source 7% > Proxy 2% > Network 28% > Target 66%


Couple of examples

[pjc]

Oh and I don't even mean during snapshots, I just mean while the backup is running, before the snapshot is even trying to be removed, the performance is heavily reduced (300-400ms ping spikes and stuff like file shares dropping for a period of time or outlook becoming unresponsive). Makes it almost impossible to do backups during the day.

[Shestakov]

What's the advantage of reverse incremental? What's the implication if I just change that option on a job?

Reversed incremental job keeps only 1 Full backup at a time, while Forward incremental requires a bit more repository space, but can increase the backup process.
You don`t need any special preparations for switching from reversed to forward incremental.
Another valuable tip is to restrict number of concurrent writing tasks for the same repository.

Finally is this even an option on a replica job?

No, it`s not. Roughly speaking replication job works like forward incremental backup job.

Oh and I don't even mean during snapshots, I just mean while the backup is running, before the snapshot is even trying to be removed, the performance is heavily reduced (300-400ms ping spikes and stuff like file shares dropping for a period of time or outlook becoming unresponsive). Makes it almost impossible to do backups during the day.

Are those interrupts current during the backup process or occur while some particular operations are processing? Have you reviewed logs with Veeam support team to investigate the problem?

[pjc]

I have sent some logs to Veeam support but probably not everything. I have arranged a session to have support look at my system so I will just go from there. You can delete this thread if you like.

Regards

[Shestakov]

Philip,

We don`t delete threads, since it can be useful for other who experience same issues.
However, it would be nice of you to post here a cause of the problem with your Exchange server once you figure it out with the support team.
Thank you!