238 KB/s Processing Rate -> Bottleneck: Target???

[KristiEva]

Hey everyone I'm having the hardest time figuring out why my processing rates are so incredibly low. It's taking 4-8 hours to run a replication job depending on the VM size which essentially makes it impossible to run all my replication jobs outside of business hours.

The frusterating part is that the issue is intermittant. Most of the time the jobs will process at less than 300 KB/s with a bottleneck at the Target but occasionally my jobs will process at 2 MB/s with a bottleneck at the Network (which makes sense because the jobs are going over a WAN with a 10mbps pipe).

The target proxy server is my Veeam server and is a VM running on my DR Host that has an iSCSI connection to a Drobo 800i for storage.

The only obvious thing that doesn't seem to be working properly is the Veeam server itself. When processing a job the guest OS/VM gets so bogged down that the server becomes unresponsive (takes 1-2 minutes to open task manager, 1-2 minutes to bounce around menu items in the Veeam console, 1-2 minutes to open Windows Explorer etc). Nothing seems to be using an extraodinary amount of systems resources though.

I'm completely at a loss...if anyone has some ideas as to what could be wrong I would love to hear from you

[foggy]

Kristin, what transport mode is being used to populate the target datastore? Also, I would note that Drobo is not among the fastest storages and you should keep in mind the fact that you also run proxy/backup server from the same storage - the VM that requires resources during replication process.

[KristiEva]

We will be replacing the Drobo with a much higher performance SAN in the upcoming year but due to budget constraints we elected to go with the Drobo in the meantime.

I have the proxies set to automatically chose a transport mode (I believe they are chosing Hot-Add). Below is the log.

Code: Select all

[code]
11/16/2014 7:00:31 PM :: Queued for processing at 11/16/2014 7:00:31 PM
11/16/2014 7:00:31 PM :: Required backup infrastructure resources have been assigned
11/16/2014 7:00:33 PM :: VM processing started at 11/16/2014 7:00:33 PM
11/16/2014 7:00:33 PM :: VM size: 465.8 GB
11/16/2014 7:00:36 PM :: Discovering replica VM
11/16/2014 7:00:51 PM :: Inventoring guest system
11/16/2014 7:00:53 PM :: Preparing guest for hot backup
11/16/2014 7:00:58 PM :: Creating snapshot
11/16/2014 7:01:00 PM :: Releasing guest
11/16/2014 7:01:06 PM :: Preparing replica VM
11/16/2014 7:01:28 PM :: Processing configuration
11/16/2014 7:01:52 PM :: Creating helper snapshot
11/16/2014 7:02:22 PM :: Using backup proxy 10.3.1.17 for disk Hard disk 1 [hotadd]
11/16/2014 7:02:22 PM :: Using target proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]
11/16/2014 7:02:50 PM :: Hard disk 1 (465.8 GB)
11/17/2014 2:36:33 AM :: Removing VM snapshot
11/17/2014 2:38:15 AM :: Deleting helper snapshot
11/17/2014 2:38:43 AM :: Finalizing
11/17/2014 2:38:47 AM :: Busy: Source 0% > Proxy 11% > Network 88% > Target 99%
11/17/2014 2:38:47 AM :: Primary bottleneck: Target
11/17/2014 2:38:47 AM :: Network traffic verification detected no corrupted blocks
11/17/2014 2:38:47 AM :: Processing finished at 11/17/2014 2:38:47 AM

Duration: 7:39:08
Processing rate: 238 KB/s
Bottleneck: Target

Processed: 465.8GB
Read: 6.2 GB
Transferred: 2.7DB (2.3x)

[KristiEva]

Since my VM that is hosting the Veeam server is running from the same datastore as the VM replicas are being written to would it be beneficial to move the target proxy server to a physical server on that network? It would then be transferring data over the network instead but perhaps at a faster rate than what I'm currently getting?

[KristiEva]

Also, the most powerful physical server I have at the DR location is also a domain controller. Would it be against best practices to utilize that server as a proxy?

Thanks in advance for the help by the way

[foggy]

Since my VM that is hosting the Veeam server is running from the same datastore as the VM replicas are being written to would it be beneficial to move the target proxy server to a physical server on that network? It would then be transferring data over the network instead but perhaps at a faster rate than what I'm currently getting?

You can try that and see whether it helps.

Also, the most powerful physical server I have at the DR location is also a domain controller. Would it be against best practices to utilize that server as a proxy?

Yes, against best practices, however not completely prohibited.

[KristiEva]

I moved the target proxy to a physical server and that seemed to resolve the issue.

I suppose the Drobo & its drives just didn't have high enough IOPS to host the Veeam server and Veeam proxy while also being the target datastore for the replicated VMs.

Thanks for your help.

[foggy]

I suppose the Drobo & its drives just didn't have high enough IOPS to host the Veeam server and Veeam proxy while also being the target datastore for the replicated VMs.

Yes, that's what I was initially trying to tell. Anyway, I'm glad you were able to get an improvement, thanks for getting back to us.