VERY Slow Replication

[Kawa]

I'm Currently evaluating Veeam B&R 6.5 . Backing up locally to a Windows 2012 Veeam Server with Direct Attached Disks (MD1220) and Direct SAN access is giving me good results BUT when i replicate to our DR site the replication is really slow, even the incrementals is taking hours.
The bottleneck is always displayed as target Datastores !

Using IOmeter and DD command , 1 MBps is the maximum speed we can get when writing data to our Target datastore @ the moment on a 100M Link. The write latency on our Target [DR] datastores seems to very high and the speed that Veeam replication job gives me actually corresponds to the performance rates I see in vClient when writing data manually to the datastore with dd tool. write latency is around 100ms

I have updated all the Firmware and BIOS to the latest versions, The only question i have is should I Disable the write back Cache on the Raid Controllers (PowerVault MD3620i)

Read cache: Enabled
Write cache: Enabled
Write cache without batteries: Disabled
Write cache with mirroring: Enabled
Flush write cache after (in seconds): 10.00
Dynamic cache read prefetch: Disabled

Appreciate your help !

Kawa

[Dima P.]

Hello Kawa,
Thank you for the detailed explanation of you issues - it is always highly appreciated!

According to the raid configuration best practices the write back Cache provides better IO performance: "...provides better performance in most cases. In Write‐Back mode, the RAID controller acknowledges write I/O requests immediately after the data loads into the controller cache. The application can continue working without waiting for the data to be physically written to the hard drives." so in case resolving IO issue it would be a good option to use.

However if you using the write back cached mode - in case of power outage the data stored at the moment in cache would be completely loss so using UPS is highly recommended.

Hope that would answer your question,
Cheers!

[Kawa]

Thanks for your response Dmitry.

[Dima P.]

Kawa,

You are welcome; please let us know if you need any further assistance!

[foggy]

Kawa, do you have proxy server deployed in your DR? If yes, what transport mode is being used to populate replica disks on target datastore and where replica metadata is stored?

[Kawa]

yes we have 2 Proxy servers in DR , 1 physical server (nbd) with 2 X 10gb nics teamed (20GB ) and 1 Virtual Server (hotadd)

[foggy]

Using IOmeter and DD command , 1 MBps is the maximum speed we can get when writing data to our Target datastore @ the moment on a 100M Link. The write latency on our Target [DR] datastores seems to very high and the speed that Veeam replication job gives me actually corresponds to the performance rates I see in vClient when writing data manually to the datastore with dd tool. write latency is around 100ms

Looks like the issue is with the storage itself. Have you contacted target storage vendor support?

Btw, what RAID type do you use?

[Kawa]

Using Raid 10, Yes I have logged a case with Dell Support , they are investigating

[foggy]

Would be great if you could update the thread with the results. Thanks.

[Kawa]

Dell made the following changes in our infrastructure in our DR site

1. Updated DR PowerConnect 8024 stack to version 5.1.0.1
2. Upgraded the Intel 10G NIC driver to 3.14.3 on 2nd host to bring it in line with host 1
3. Upgraded MD3620i firmware to latest version 07.84.44.60

After the changes were made, below are 2 replication i attempted and below the results looks promising, Since we only have 100 Mbit/s link which gives you the maximum transfer speed of 10 MB/s. The performance we're getting is expected.

1. Print_Replication (initial Rep)VM HMCA-PRINT-SVR01, 14.4 GB was transferred in 33 min 18 sec, that gives us 14745.6 MB / 1998 sec ~ 7.4 MB/s

2. Email_Replication (initial Rep)
VM HMCA-EX01, 571.8 GB was transferred in 15 h 4 min, that gives us 571.8 GB / 15 h 5 min =585523 MB / 54000 sec ~ 10 MBps

[foggy]

Glad that you have such an improvement! Thanks for providing the results, much appreciated, indeed.

[Kawa]

Very happy with the product, getting quotes now and looking forward to V7