replication much slower than backup job

[szafa]

Hi All

i installed Veeam B&R 7 trial version on VM to test it. I did some backup jobs and all was ok. Then I tried replication job and was very slow compering to backup job.

Replication processing rate : 12MB/s vs. 107MB/s full backup job and 222MB/s incremental. I tried other replication job and was about 54MB/s but then bigger replication job about 80GB again 12MB/s. I set hotadd mode on proxy because when I tried auto Veeam was using network mode.

My configuration

Veeam 7 installed on VM Windows Server 2008
VCenter cluster (6 hosts) Vmware 5.1
HP EVA 6000 4GB FC + second spare old 2GB FC HP SAN

Why is so big difference between replication and backup? Why 4GB FC target is bottle neck when (hotadd mode).And why when i leave default proxy settings is using LAN which was slow with backups (14MB/s)

Help Please

[Vitaliy S.]

Can you please tell me what is your bottleneck stats for replication job that is slow? Also can you please tell me what is your target host storage? Is is an old 2GB HP San?

[szafa]

Bottleneck is 99% target, both new 4GB FC and old 2GB are slow when replicating but ok when doing backup.

[tsightler]

When doing replication are both the source and target proxies showing that they are using hotadd?

[szafa]

I have just one proxy on same vm as veeam backup server

[foggy]

You should set up a proxy on the target host as well, so that target datastore could be populated via hotadd. Now you're writing through the host's network stack over management network directly from the source proxy, which is very inefficient.

[szafa]

this is test veeam VM appliance (server, proxy, repository on one vm as suggested in documentation). Why backups jobs are and just replication is so slow?

[foggy]

They have different targets and different data flow: backup jobs write to the backup repository (one of your SANs), while replication jobs write to VMFS datastore (and use repository to store replica metadata only). I suggest reviewing the corresponding user guide sections to get better understanding of different job types: Backup Architecture and Replication Architecture.

Thanks.

[Vitaliy S.]

Bottleneck is 99% target, both new 4GB FC and old 2GB are slow when replicating but ok when doing backup.

Do you use the same storage as a source and target for the replication jobs?

[szafa]

Hi I use two LUN's on old SAN. Still can't get what I did wrong. I don not have access to server at the moment. I'm not sure if I presented targed data store to esxi host. I'll check it on Monday. Anyway how is should presented target data store and do I need other proxy to replicate VM in hot add mode?

[foggy]

It is still not quite clear from where to where you're replicating. Could you please describe your setup in more detail?

[szafa]

Vsphere 5.1 cluster with 6x hosts, just one site, don't have DR site, SAN FC HP storage. Veeam 7 server is VM on one of the host all roles on same server (vmware tools installed), proxy is set to addhot mode, target backup storage presented to Veeam server. When I set proxy to auto it used network mode to do backups so I have set it to addhot and backups are flying now(why Veeam is choosing network mode insted of addhot?). Then I tried replicate VM from one host to other on my cluster and is extrymly slow. Please let me know if you need more details

[Vitaliy S.]

When I set proxy to auto it used network mode to do backups so I have set it to addhot and backups are flying now(why Veeam is choosing network mode insted of addhot?).

If hotadd backup mode is available for all VMs in the job, your proxy server should select this mode when automatic option selected. Can you confirm that there is no failover to network mode and you've used the same VMs to backup when hotadd mode was not selected by the job?

Then I tried replicate VM from one host to other on my cluster and is extrymly slow. Please let me know if you need more details

Try installing the second proxy and see if it makes any difference. Judging by the bottleneck stats your source storage slows down the whole process. Can you please tell us what operation takes most time if you look at the replication job session? Data transfer?

[szafa]

If hotadd backup mode is available for all VMs in the job, your proxy server should select this mode when automatic option selected. Can you confirm that there is no failover to network mode and you've used the same VMs to backup when hotadd mode was not selected by the job?

yes, i used same VM and when auto selected transfer was slow so now i set to hotadd mode.

i just check replication job scheduled to run Sunday and it took over 5h and again bottleneck 99% target.



I can try to add other proxy but looks like i have other issue and i don't know what could be wrong. Please let me know what I could check/do to improve replication.

[Vitaliy S.]

In order to troubleshoot it further, I would suggest doing two things:

1. deploying a secondary proxy and try it in hotadd and network modes
2. change target storage for replication job

Let us know the results of these two tests.

[szafa]

I replicated to other target and was little bit faster but still slow.

Then I added other proxy (mode auto but system again choose hotadd) but still same

Any suggestion why is still slow. Can I check if other proxy was used in replication job?

[foggy]

Right on this screenshot, you can see that the same proxy server (the default one - VMware Backup Proxy) was used as both source and target.

[szafa]

ok so what should I do to force this job to use both proxy's ?

[foggy]

You can explicitly select the required target proxy server at the Job Settings step of the replication wizard.

[szafa]

Thanks guys, finally I can see improvement. I set explicit proxy for source and target, set both to hotadd mode and replication is much faster now.

is anything else that I can do to improve replication time?

[szafa]

I just notice that nbd mode was used for target proxy even I selected virtual appliance mode. Why is that?

[szafa]

I tried again to set both proxy's to auto selection but again target proxy used nbd mode. Next thing is why is so big difference in read between VM disk (all are on same datastore)?

[Vitaliy S.]

I just notice that nbd mode was used for target proxy even I selected virtual appliance mode. Why is that?

Did you disable a "failover to network mode" option in the proxy settings?

Next thing is why is so big difference in read between VM disk (all are on same datastore)?

What data do you have on the Hard Disk 3 of this VM? What about other VMs in the job?

[veremin]

Also, are you positive target proxy is capable of Hot Add mode? Thanks.

[szafa]

I have installed VM Tools on other proxy and system is using hotadd mode but now replication is slower! it drooped from 70MB/s (mix mode, addhot on source and ndb on target or nbd on both) to about 17MB/s (hotadd on both proxy's). What is wrong now?

[Vitaliy S.]

And what is the new bottleneck in this configuration? Are you running incremental runs all the time?

[szafa]

yes, all jobs are incremental.

Bottleneck for NBD : source

18/09/2014 15:29:01 :: Busy: Source 99% > Proxy 41% > Network 7% > Target 44%
18/09/2014 15:29:01 :: Primary bottleneck: Source

Bottleneck for Hotadd : target

18/09/2014 15:56:40 :: Busy: Source 22% > Proxy 17% > Network 1% > Target 66%
18/09/2014 15:56:40 :: Primary bottleneck: Target

I have 2 very same VM's and I tried to run incremental replica on both on almost same time. It resulted in same data transfer and read but processing rate are different 5MB/s(Hotadd) to 46MB/s(NBD). What is causing hotadd mode so slow?

[Vitaliy S.]

I have 2 very same VM's and I tried to run incremental replica on both on almost same time. It resulted in same data transfer and read but processing rate are different 5MB/s(Hotadd) to 46MB/s(NBD). What is causing hotadd mode so slow?

The only explanation I have for this is either low performance of datastore (while doing writes using hotadd mode) or the hotadd proxy server itself that slows down the job. I would suggest contacting our technical team to verify what operation (by looking through the debug logs) takes most of the time on the hotadd run.

[szafa]

Will do. Thanks for all your help