Host-based backup of Microsoft Hyper-V VMs.
Post Reply
jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Slow Processing Rate

Post by jorgene »

Hi,

Just set up a new W2012 R2 Hyper-V server on brand new HP DL360 G9 with mirrored 10K RPM disks.
Migrated 3 VM's from an W2008 Hyper-V to it.
Installed Veeam B&R on the old Hyper-V server (W2008).

Set up BackupJob and now when first Job is Running Processing rate is extremely slow (11MB/s).

Bottleneck: Target

Disk read (live data, since job is running now) are ~4-5MB/s per/Harddisk. (Currently backing up 2 VMS)
I had no problem pushing 200GB data between the servers when i copied the VM files between them.
So i see no bottleneck in the Network or Disks.
On the Veeam B&R (the W2008 server) i have a ISCSI Connection on a dedicated NIC to a LUN where backup Respository is located.
I tried copy 10GB data from the server to the LUN and those times are normal.

I see no abnormal load on W2008 server or on the new Hyper-V server (W2012 R2) CPU/Network or Disk Queue Lengt on disks.

What am i missing here? Why is it so slow?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Processing Rate

Post by Vitaliy S. »

Hi Jorgen,

What backup mode are you using (forever incremental or reversed incremental)? Also what is the full bottleneck stats (including all components)?

Thanks!
jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: Slow Processing Rate

Post by jorgene »

Hi,

(Incremental with syntectic full on Fridays)

Now first Job finished successfully (Active Full).

Duration 11:03:24.
Processing Rate 9MB/s
Processed: 342,5 GB (100%)
Read: 336,9 GB
Transferred: 227,6 GB (1.5x)

First VM
Busy: Source 2% > Proxy 1% > Network 44% > Target 57%
Hard disk 1 (127,0 GB) 94,0 GB read at 3 MB/s
Hard disk 2 (390,0 GB) 169,8 GB read at 4 MB/s

Second VM
Busy: Source 1% > Proxy 1% > Network 3% > Target 97%
Hard disk 1 (80,0 GB) 73,1 GB read at 3 MB/s


Now Second Job Run (same Job)(Incremental
)

Duration: 0:08:56
Processing Rate: 53 MB/s
Bottleneck: Target

Processed: 342,8 GB (100%)
Read: 18,4 GB
Transferred: 2,8 GB (6.5x)

First VM
Busy: Source 70% > Proxy 5% > Network 45% > Target 82%
Hard disk 1 (127,0 GB) 8,4 GB read at 42 MB/s [CBT]
Hard disk 2 (390,0 GB) 165,0 MB read at 23 MB/s [CBT]

Second VM
Busy: Source 22% > Proxy 3% > Network 50% > Target 49%
Hard disk 1 (80,0 GB) 9,8 GB read at 30 MB/s [CBT]
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Processing Rate

Post by Vitaliy S. »

Hmm...Seems like target is indeed the issue. Can you create any temp backup job and point to another repository? For example, the same local Hyper-v server 2012R2 can be used for that. BTW, are you using software or hardware VSS provider?
jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: Slow Processing Rate

Post by jorgene »

Hi, i can do later on and test that, but the battery in the raid controller is bad (on W2008 server) so cache is disabled on that one so i will get bad performance on local disk util it's replaced.
But here are some results on other Jobs:

Backup Job Disk Before Tape (Reversed Inremental) (First Run).
To Respository (New LUN on same NAS device (ISCSI))
Duration 13:59:50
Processing Rate: 7MB/s
Bottleneck: Target

Processed: 357,5 GB (100%)
Read: 351,8 GB
Transferred: 233,4 GB (1.5x)

(3 VMs in total)

Backup To Tape Job (Backing up data from previous Backup Job to Tape)

Duration 1:05:28
Processing Rate: 65 MB/s
Bottleneck: Source

Processed: 232,8 GB
Read: 232,8 GB
Transferred: 232,8 GB

So pushing the same data from the LUN to the Tape device goes normal rates as it should. Also i have noticed no problems pushing data to the LUNs by normal windows copy.

No Hardware VSS Provider used.

So is it really a Target issue? The NAS target has 4x7200rpm HDD's set up as Raid 10.

I got an idea though, could it be that CBT data is written to local disks on the W2008 server? and that the failing battery on the raid controller resulting in bad write performance is responsible for low performance?
As i understand those CBT data should be written to the Hyper-V host on the W2012 R2 Hyper-V server.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Processing Rate

Post by Vitaliy S. »

Yes, it seems like write rate is the bottleneck here. CBT data is stored on the source hosts and should not affect your job run.
jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: Slow Processing Rate

Post by jorgene » 2 people like this post

Found the problem i Think, it was "Always sync writes" setting on the LUN on the Netgear Readynas where backup respository resides.
When this was disabled performance went up !
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Processing Rate

Post by Vitaliy S. »

What's the new bottleneck? Try installing hardware VSS and using off-host proxy server to optimize VM data retrieval process.
jorgene
Enthusiast
Posts: 76
Liked: 16 times
Joined: Dec 22, 2011 1:39 pm
Full Name: Jorgen Eriksson
Contact:

Re: Slow Processing Rate

Post by jorgene » 1 person likes this post

Performance is now acceptable, between 50-60MB/s on reversed incremental and 160MB/s on incremental runs.
Bottleneck is still target, and that's normal since it has slower disks then the source.

Apperently firmware 6.2.2 of Netgear (probably from 6.2.0 or even earlier) there is a new setting on the details tab of an ISCSI LUN.
Sync Writes: that you can set to Allow, Force or Disable. It is default to Allow. When disabled it was a big performance increase on the device.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Processing Rate

Post by Vitaliy S. »

Thanks for letting us know the settings required to optimize job performance with Netgear box.
Post Reply

Who is online

Users browsing this forum: No registered users and 17 guests