Host-based backup of VMware vSphere VMs.
Post Reply
Kawa
Novice
Posts: 6
Liked: 1 time
Joined: Mar 19, 2013 12:23 pm
Full Name: Kawa Farid
Contact:

VERY Slow Replication

Post by Kawa »

I'm Currently evaluating Veeam B&R 6.5 . Backing up locally to a Windows 2012 Veeam Server with Direct Attached Disks (MD1220) and Direct SAN access is giving me good results BUT when i replicate to our DR site the replication is really slow, even the incrementals is taking hours.
The bottleneck is always displayed as target Datastores !

Using IOmeter and DD command , 1 MBps is the maximum speed we can get when writing data to our Target datastore @ the moment on a 100M Link. The write latency on our Target [DR] datastores seems to very high and the speed that Veeam replication job gives me actually corresponds to the performance rates I see in vClient when writing data manually to the datastore with dd tool. write latency is around 100ms

I have updated all the Firmware and BIOS to the latest versions, The only question i have is should I Disable the write back Cache on the Raid Controllers (PowerVault MD3620i)

Read cache: Enabled
Write cache: Enabled
Write cache without batteries: Disabled
Write cache with mirroring: Enabled
Flush write cache after (in seconds): 10.00
Dynamic cache read prefetch: Disabled

Appreciate your help !

Kawa
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VERY Slow Replication

Post by Dima P. »

Hello Kawa,
Thank you for the detailed explanation of you issues - it is always highly appreciated!

According to the raid configuration best practices the write back Cache provides better IO performance: "...provides better performance in most cases. In Write‐Back mode, the RAID controller acknowledges write I/O requests immediately after the data loads into the controller cache. The application can continue working without waiting for the data to be physically written to the hard drives." so in case resolving IO issue it would be a good option to use.

However if you using the write back cached mode - in case of power outage the data stored at the moment in cache would be completely loss so using UPS is highly recommended.

Hope that would answer your question,
Cheers!
Kawa
Novice
Posts: 6
Liked: 1 time
Joined: Mar 19, 2013 12:23 pm
Full Name: Kawa Farid
Contact:

Re: VERY Slow Replication

Post by Kawa »

Thanks for your response Dmitry.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VERY Slow Replication

Post by Dima P. »

Kawa,

You are welcome; please let us know if you need any further assistance!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VERY Slow Replication

Post by foggy »

Kawa, do you have proxy server deployed in your DR? If yes, what transport mode is being used to populate replica disks on target datastore and where replica metadata is stored?
Kawa
Novice
Posts: 6
Liked: 1 time
Joined: Mar 19, 2013 12:23 pm
Full Name: Kawa Farid
Contact:

Re: VERY Slow Replication

Post by Kawa »

yes we have 2 Proxy servers in DR , 1 physical server (nbd) with 2 X 10gb nics teamed (20GB ) and 1 Virtual Server (hotadd)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VERY Slow Replication

Post by foggy »

Kawa wrote:Using IOmeter and DD command , 1 MBps is the maximum speed we can get when writing data to our Target datastore @ the moment on a 100M Link. The write latency on our Target [DR] datastores seems to very high and the speed that Veeam replication job gives me actually corresponds to the performance rates I see in vClient when writing data manually to the datastore with dd tool. write latency is around 100ms
Looks like the issue is with the storage itself. Have you contacted target storage vendor support?

Btw, what RAID type do you use?
Kawa
Novice
Posts: 6
Liked: 1 time
Joined: Mar 19, 2013 12:23 pm
Full Name: Kawa Farid
Contact:

Re: VERY Slow Replication

Post by Kawa »

Using Raid 10, Yes I have logged a case with Dell Support , they are investigating :-)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VERY Slow Replication

Post by foggy »

Would be great if you could update the thread with the results. Thanks.
Kawa
Novice
Posts: 6
Liked: 1 time
Joined: Mar 19, 2013 12:23 pm
Full Name: Kawa Farid
Contact:

Re: VERY Slow Replication

Post by Kawa » 1 person likes this post

Dell made the following changes in our infrastructure in our DR site

1. Updated DR PowerConnect 8024 stack to version 5.1.0.1
2. Upgraded the Intel 10G NIC driver to 3.14.3 on 2nd host to bring it in line with host 1
3. Upgraded MD3620i firmware to latest version 07.84.44.60

After the changes were made, below are 2 replication i attempted and below the results looks promising, Since we only have 100 Mbit/s link which gives you the maximum transfer speed of 10 MB/s. The performance we're getting is expected.

1. Print_Replication (initial Rep)VM HMCA-PRINT-SVR01, 14.4 GB was transferred in 33 min 18 sec, that gives us 14745.6 MB / 1998 sec ~ 7.4 MB/s

2. Email_Replication (initial Rep)
VM HMCA-EX01, 571.8 GB was transferred in 15 h 4 min, that gives us 571.8 GB / 15 h 5 min =585523 MB / 54000 sec ~ 10 MBps
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VERY Slow Replication

Post by foggy »

Glad that you have such an improvement! Thanks for providing the results, much appreciated, indeed.
Kawa
Novice
Posts: 6
Liked: 1 time
Joined: Mar 19, 2013 12:23 pm
Full Name: Kawa Farid
Contact:

Re: VERY Slow Replication

Post by Kawa »

Very happy with the product, getting quotes now and looking forward to V7
Post Reply

Who is online

Users browsing this forum: No registered users and 73 guests