Slow Processing Rate

[jorgene]

Hi,

Just set up a new W2012 R2 Hyper-V server on brand new HP DL360 G9 with mirrored 10K RPM disks.
Migrated 3 VM's from an W2008 Hyper-V to it.
Installed Veeam B&R on the old Hyper-V server (W2008).

Set up BackupJob and now when first Job is Running Processing rate is extremely slow (11MB/s).

Bottleneck: Target

Disk read (live data, since job is running now) are ~4-5MB/s per/Harddisk. (Currently backing up 2 VMS)
I had no problem pushing 200GB data between the servers when i copied the VM files between them.
So i see no bottleneck in the Network or Disks.
On the Veeam B&R (the W2008 server) i have a ISCSI Connection on a dedicated NIC to a LUN where backup Respository is located.
I tried copy 10GB data from the server to the LUN and those times are normal.

I see no abnormal load on W2008 server or on the new Hyper-V server (W2012 R2) CPU/Network or Disk Queue Lengt on disks.

What am i missing here? Why is it so slow?

[Vitaliy S.]

Hi Jorgen,

What backup mode are you using (forever incremental or reversed incremental)? Also what is the full bottleneck stats (including all components)?

Thanks!

[jorgene]

Hi,

(Incremental with syntectic full on Fridays)

Now first Job finished successfully (Active Full).

Duration 11:03:24.
Processing Rate 9MB/s
Processed: 342,5 GB (100%)
Read: 336,9 GB
Transferred: 227,6 GB (1.5x)

First VM
Busy: Source 2% > Proxy 1% > Network 44% > Target 57%
Hard disk 1 (127,0 GB) 94,0 GB read at 3 MB/s
Hard disk 2 (390,0 GB) 169,8 GB read at 4 MB/s

Second VM
Busy: Source 1% > Proxy 1% > Network 3% > Target 97%
Hard disk 1 (80,0 GB) 73,1 GB read at 3 MB/s


Now Second Job Run (same Job)(Incremental)

Duration: 0:08:56
Processing Rate: 53 MB/s
Bottleneck: Target

Processed: 342,8 GB (100%)
Read: 18,4 GB
Transferred: 2,8 GB (6.5x)

First VM
Busy: Source 70% > Proxy 5% > Network 45% > Target 82%
Hard disk 1 (127,0 GB) 8,4 GB read at 42 MB/s [CBT]
Hard disk 2 (390,0 GB) 165,0 MB read at 23 MB/s [CBT]

Second VM
Busy: Source 22% > Proxy 3% > Network 50% > Target 49%
Hard disk 1 (80,0 GB) 9,8 GB read at 30 MB/s [CBT]

[Vitaliy S.]

Hmm...Seems like target is indeed the issue. Can you create any temp backup job and point to another repository? For example, the same local Hyper-v server 2012R2 can be used for that. BTW, are you using software or hardware VSS provider?

[jorgene]

Hi, i can do later on and test that, but the battery in the raid controller is bad (on W2008 server) so cache is disabled on that one so i will get bad performance on local disk util it's replaced.
But here are some results on other Jobs:

Backup Job Disk Before Tape (Reversed Inremental) (First Run).
To Respository (New LUN on same NAS device (ISCSI))
Duration 13:59:50
Processing Rate: 7MB/s
Bottleneck: Target

Processed: 357,5 GB (100%)
Read: 351,8 GB
Transferred: 233,4 GB (1.5x)

(3 VMs in total)

Backup To Tape Job (Backing up data from previous Backup Job to Tape)

Duration 1:05:28
Processing Rate: 65 MB/s
Bottleneck: Source

Processed: 232,8 GB
Read: 232,8 GB
Transferred: 232,8 GB

So pushing the same data from the LUN to the Tape device goes normal rates as it should. Also i have noticed no problems pushing data to the LUNs by normal windows copy.

No Hardware VSS Provider used.

So is it really a Target issue? The NAS target has 4x7200rpm HDD's set up as Raid 10.

I got an idea though, could it be that CBT data is written to local disks on the W2008 server? and that the failing battery on the raid controller resulting in bad write performance is responsible for low performance?
As i understand those CBT data should be written to the Hyper-V host on the W2012 R2 Hyper-V server.

[Vitaliy S.]

Yes, it seems like write rate is the bottleneck here. CBT data is stored on the source hosts and should not affect your job run.

[jorgene]

Found the problem i Think, it was "Always sync writes" setting on the LUN on the Netgear Readynas where backup respository resides.
When this was disabled performance went up !

[Vitaliy S.]

What's the new bottleneck? Try installing hardware VSS and using off-host proxy server to optimize VM data retrieval process.

[jorgene]

Performance is now acceptable, between 50-60MB/s on reversed incremental and 160MB/s on incremental runs.
Bottleneck is still target, and that's normal since it has slower disks then the source.

Apperently firmware 6.2.2 of Netgear (probably from 6.2.0 or even earlier) there is a new setting on the details tab of an ISCSI LUN.
Sync Writes: that you can set to Allow, Force or Disable. It is default to Allow. When disabled it was a big performance increase on the device.

[Vitaliy S.]

Thanks for letting us know the settings required to optimize job performance with Netgear box.