Virtual appliance/hotadd poor performance ! ?

[f.daoulas]

Hello,

(first of all, I have to apologize fo my bad english)

When I use [Transport mode=Virtual appliance] instead of [Transport mode=Network] the replication 'performances' decrease a lot .

-Transport mode=Network : hard disc read 15<->22 MB/s [CBT]
-Transport mode=Virtual appliance : hard disc read 2<->8 MB/s [CBT] and réplication jobs take 3X longer

I don't know where/what to search fo understanding and solving this pb.
Please help me.

Config. :
Veeam 6.0 patch 3.0
Esx 5.0 on HP Proliant ML380 ad ML360
San source : EMC AX4-5i
San target : Datacore SanMelody 3.0.4.2 on Windows Srv 2008 R2

Rgds.
Fabrice.

[foggy]

Fabrice, please provide the bottleneck statistics numbers for both jobs (available by selecting Realtime Statistics on the job's shortcut menu). Please also check that the specified transport modes are effectively being used for reading VM data (in the same window, select the particular VM to the left). Thanks.

[f.daoulas]

Hello,

Here is the the both jobs statictics :

Transport mode=Virtual appliance

Code: Select all

16:34:37	Queued for processing at 21.05.2012 16:34:37
16:34:37	Required resources have been assigned
16:34:40	VM processing started at 21.05.2012 16:34:39
16:34:40	VM size: 279.0 GB
16:34:40	Using source proxy [b]RGDWH [hotadd;nbd][/b]
16:34:42	Using target proxy [b]RGDWH [hotadd;nbd][/b]
16:34:42	Discovering replica VM
16:34:42	Preparing replica VM
16:35:05	Preparing guest for hot backup
16:35:10	Creating snapshot
16:35:17	Releasing guest
16:35:23	Disk '[SanDataStore] RGGED/RGGED_2.vmdk' has been skipped
16:35:26	Processing configuration
16:35:38	Creating helper snapshot
16:35:42	Hard Disk 1 (39.0 GB) 1.1 Gb [b]read at 6 MB/s [CBT][/b]
16:39:01	Hard Disk 2 (240.0 GB) 214.0 MB  [b]read at 3 MB/s [CBT][/b]
16:40:17	Deleting helper snapshot
16:40:51	Truncating transaction logs
16:40:54	Removing snapshot
16:42:43	Swap file blocks skipped: 80.0 MB
16:42:43	Finalizing
16:42:43	1 restore point removed by retention policy
16:44:28	[b]Busy: Source 13% > Proxy 39% > Network 3% > Target 99%[/b]
16:44:28	Primary bottleneck: Target
16:44:28	Processing finished at 21.05.2012 16:44:28

Transport mode=Network

Code: Select all

17:34:50	Queued for processing at 21.05.2012 17:34:50
17:34:51	Required resources have been assigned
17:34:53	VM processing started at 21.05.2012 17:34:52
17:34:53	VM size: 279.0 GB
17:34:53	Using source proxy [b]RGDWH [nbd][/b]
17:34:55	Using target proxy [b]AVANTI [nbd][/b]
17:34:55	Discovering replica VM
17:34:55	Preparing replica VM
17:35:18	Preparing guest for hot backup
17:35:23	Creating snapshot
17:35:28	Releasing guest
17:35:34	Disk '[SanDataStore] RGGED/RGGED_2.vmdk' has been skipped 
17:35:38	Processing configuration
17:35:49	Creating helper snapshot
17:35:52	Hard Disk 1 (39.0 GB)  1.6 Gb [b]read at 26 MB/s [CBT] [/b]
17:37:00	Hard Disk 2 (240.0 GB) 214.0 MB  [b]read at 15 MB/s [CBT][/b]
17:37:16	Deleting helper snapshot
17:37:51	Truncating transaction logs
17:37:55	Removing snapshot
17:38:08	Swap file blocks skipped: 16.0 MB
17:38:09	Finalizing
17:38:09	1 restore point removed by retention policy
17:40:07	[b]Busy: Source 99% > Proxy 73% > Network 1% > Target 57%[/b]
17:40:07	Primary bottleneck: Source
17:40:07	Processing finished at 21.05.2012 17:40:07

Thank you for your help.

Rgds.
Fabrice

[tsightler]

These are replication jobs (see that there is both a source and target proxy in the logs). I noticed that in the first log using hot add the same proxy was used for both source and target, while for the second log it was two different proxy servers. I'm guessing that this is the difference, especially if a WAN is involved, but in any case you need to test like for like to have valid results.

[f.daoulas]

Hello,

A litle more information :

- For both logs it's the same replication job.
The only difference, is the setting of the backup proxy 'RGDWH' :
first log (at 16:34) Transport mode=Virtual appliance
second log (at 17:34) Transport mode=Network

- Source & target proxy = automatic selection

- No Wan involved (san source and target are 'local' icsii)

Rgds.
Fabrice

[tsightler]

Right, but because of this change for the first job the RGDWH proxy was used for both source and target, while for the second job it used AVANTI as the target proxy, thus the comparison is not the same. Changing the proxy mode is causing the automatic proxy selection to make different choices.

That being said, with the amount of data you are moving it's unlikely that there would be a significant difference between hotadd and network mode as hotadd has significantly more setup time. You entire replication is only taking about 1 1/2 minutes for network mode. Hotadd will likely use 1-2 minutes just for setup/teardown for the hotadd disks to the proxy, so there's no way it will make up for the performance difference.