New Dell Compellent = slower replication?

[B.F.]

We recently installed new Dell Compellent disk storage systems in both our main and secondary sites. Prior to Compellent, we have been using an old HP MSA2000 where each of the ESXi hosts were directly connected via SAS cables. Now with Compellent, we have a dedicated 10gb switch at each site where all the local ESXi hosts communicate with the site's new Dell storage via iSCSI. We still have the HP MSA installed as well for the time being.

One of our main VM's that we replicate from the main to secondary site is our file server. The file server is around 3tb total. Shortly after the Compellent installation, we migrated the file server's replica in the secondary site from the HP MSA to the Dell storage.

Ever since we have done this, we have noticed that the replication time to complete has increased. Looking at the bottleneck history, I can see that the target percentage has increase 13 - 25 percentage points. All the other pieces of the bottleneck chain remains relatively the same. The communication between the sites has not changed and is reflected by the consistent 27% (+-2%) in the bottleneck logs. The performance rate logs from EM show that before Dell, it was around 13 MB/s. Now we are getting around 6 MB/s

We have confirmed that the replication does not overlap with the Compellent Data Progression or it's built in Snap Shot schedule. I did discover that the Server OS preference setting on each of the Dell systems was set to Other Single Path. I have since changed them to VMware ESXi 6.0. No change. Compellent does not have deduplication as far as I'm aware either.

Please advice and thanks!

[foggy]

Am I getting right, that the primary bottleneck for this replication job is target? What transport modes are used by the source and target proxy servers?

[B.F.]

...the primary bottleneck for this replication job is target?

Actually the bottleneck seems to ping pong between the source and the target now. The source has always been 77% (+- of course) and the target was 50% - 55%. Now the target is 68% - 80%.

What transport modes are used by the source and target proxy servers?

Not certain I understand what you are asking.
Compression level = Optimal (recommended)
Exclude swap file blocks = checked
Enable VMware Tools quiescence = unchecked
Use changed block tracking data = checked
Enable CBT for all protected VMs automatically = checked
Data Transfer = Direct (only option available)
Target Proxy = Automatic selection (we do have a proxy setup at each site)

Does the above answer your question?

Thanks

[DaveWatkins]

You talk about pathing, have you set the ESX hosts to use round robin for the new iSCSI LUN's?

Jumbo frames setup for the iSCSI traffic?

[foggy]

I was talking about the transport mode used to populate the target datastore. You can look it up in the job session window, if you select the particular VM in the left pane and locate the proxy server selected for processing to the right.

[B.F.]

You talk about pathing, have you set the ESX hosts to use round robin for the new iSCSI LUN's?

Jumbo frames setup for the iSCSI traffic?

Looking at the "Path Selection Policy", it looks like they are set for "Most Recently Used" and not "Round Robin".

I'm also seeing that the vSwitches are set for 9000 MTU. However, we must of forgotten to adjust the NIC's themselves since I see they are still set to 1500 MTU.

Is it ok to make these type of changes on a live system or do I need to vmotion VM's off, reboot hosts, etc?

Thanks


PS: We are still pretty new to iSCSI come from a direct SAS connection environment.

[B.F.]

I was talking about the transport mode used to populate the target datastore. You can look it up in the job session window, if you select the particular VM in the left pane and locate the proxy server selected for processing to the right.

The Transport Mode for the Proxy is "Automatic Selection"

Should it be something else?

Thanks

[foggy]

I mean the transport mode effectively selected by the proxy server during VM processing. You can look it up in the job session log, as I've described above.

[B.F.]

Sorry foggy, I'm not sure I'm finding what you are asking.

Here's where I'm going

• Backup and Replication
  Jobs
  Then select the job on the right
  I then select the VM on the left, as described, in the bottom pane.

I then see the following Actions show up

• Replicating restore point....
  Queued for processing....
  Required Backup infrastructure resources have been assigned
  VM processing started...
  VM size...
  Discovering replica VM
  Preparing replica VM
  Processing configuration
  Creating helper snapshot
  Using target proxy <name> for disk Hard disk 2 [hotadd]
  Hard disk 2 ...read at 5 MB/s
  Using target proxy <name> for disk Hard disk 1 [hotadd]
  --- Continues this for all the disks for the VM ---
  Deleting helper snapshot
  Finalizing
  Busy: Source 76% > Proxy.....
  Primary bottle neck....
  Network traffic verification detected no corrupted blocks
  Process finished...

Not seeing a Transport method so I must be in the wrong place.

[foggy]

Hotadd is the transport method in this case. In case target is the primary bottleneck, looks like the target storage is the issue.

[B.F.]

I have a couple followup questions on this topic.

1. What does "Processing rate" actually mean? I see that the Throughput Speed can be quite a bit faster than the processing rate.

2. When there is an incremental replication going on, is there a lot of random searches and writes at the destination? Mostly curious if there is a lot of comparing going on between the latest source and what's at the destination.

3. What is the path of the data during offsite replication? Does it go through vCenter at all? My assumption is Source -> local Veeam Server -> WAN -> Offsite Proxy -> Destination.

Thanks
Brendan

[DGrinev]

Hi,

1. I'd recommend you review a thread called "Interpreting real-time statistics" you'll find detailed descriptions of every stat used by jobs.
2. All data changes since the last job run are written to the snapshot delta file, and the snapshot delta file acts as a restore point. The more details you'll find in "Replication chain".
3. The Off-site replication data flow looks like: Source -> Source Proxy -> WAN -> Offsite Proxy -> Destination. Also, you can see it in the UG article "Replication Scenarios". Thanks!

[B.F.]

After a long process of speaking with Dell and Veeam on this issue, we have FINALLY got the problem resolved.

The target proxy transport mode was set to "Automatic". When the replication ran, it would choose to use HotAdd. If we changed the transport mode to "Network", it then would show NBD instead of HotAdd. Result? HotAdd was getting between 4 -10 MB/s. NBD is getting 34 - 50 MB/s! Huge improvement

Thought I'd share our findings in case others are noticing similar throughput issues with replication.

Thanks

[DGrinev]

Hi B.F.,

Thank you for following up, that's could be useful for further readers.
Also, I would recommend you to read this post by Tom, that explains in depth the difference between Hotadd and Network mode processes. Thanks!

[Mawdo@LMH]

Looking at the "Path Selection Policy", it looks like they are set for "Most Recently Used" and not "Round Robin".

Just a note (and an apology for resurrecting a thread...)

We have experienced some VERY serious outages with the MPIO set to "Most Recently Used" as is default. I would recommend, if not already done, you set this to "Round Robin" ASAP. We have ... NOW Please contact Dell support if unsure.

It was a very annoying day off for me when an MRU pathed volume went offline with ~50% of our servers on it... Took 3 days to get SharePoint behaving again.

[pidthepiper]

As far as i am aware the PSP for compellent is RR not MRU. The newer versions of SCOS show as SATP_ALUA in ESXi and this has a default PSP of MRU. Compellent support and in their docs recommend that you change the PSP for ALUA to RR to be fully supported.