Discussions specific to the VMware vSphere hypervisor
Post Reply
szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

replication much slower than backup job

Post by szafa » Sep 10, 2014 3:56 pm

Hi All

i installed Veeam B&R 7 trial version on VM to test it. I did some backup jobs and all was ok. Then I tried replication job and was very slow compering to backup job.

Replication processing rate : 12MB/s vs. 107MB/s full backup job and 222MB/s incremental. I tried other replication job and was about 54MB/s but then bigger replication job about 80GB again 12MB/s. I set hotadd mode on proxy because when I tried auto Veeam was using network mode.

My configuration

Veeam 7 installed on VM Windows Server 2008
VCenter cluster (6 hosts) Vmware 5.1
HP EVA 6000 4GB FC + second spare old 2GB FC HP SAN

Why is so big difference between replication and backup? Why 4GB FC target is bottle neck when (hotadd mode).And why when i leave default proxy settings is using LAN which was slow with backups (14MB/s)

Help Please

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 10, 2014 4:45 pm

Can you please tell me what is your bottleneck stats for replication job that is slow? Also can you please tell me what is your target host storage? Is is an old 2GB HP San?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 10, 2014 7:02 pm

Bottleneck is 99% target, both new 4GB FC and old 2GB are slow when replicating but ok when doing backup.

tsightler
VP, Product Management
Posts: 5418
Liked: 2240 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: replication much slower than backup job

Post by tsightler » Sep 10, 2014 7:37 pm

When doing replication are both the source and target proxies showing that they are using hotadd?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 10, 2014 9:11 pm

I have just one proxy on same vm as veeam backup server

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: replication much slower than backup job

Post by foggy » Sep 11, 2014 6:16 am

You should set up a proxy on the target host as well, so that target datastore could be populated via hotadd. Now you're writing through the host's network stack over management network directly from the source proxy, which is very inefficient.

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 11, 2014 8:07 am

this is test veeam VM appliance (server, proxy, repository on one vm as suggested in documentation). Why backups jobs are and just replication is so slow?

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: replication much slower than backup job

Post by foggy » Sep 11, 2014 8:20 am

They have different targets and different data flow: backup jobs write to the backup repository (one of your SANs), while replication jobs write to VMFS datastore (and use repository to store replica metadata only). I suggest reviewing the corresponding user guide sections to get better understanding of different job types: Backup Architecture and Replication Architecture.

Thanks.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 11, 2014 11:05 am

szafa wrote:Bottleneck is 99% target, both new 4GB FC and old 2GB are slow when replicating but ok when doing backup.
Do you use the same storage as a source and target for the replication jobs?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 11, 2014 2:31 pm

Hi I use two LUN's on old SAN. Still can't get what I did wrong. I don not have access to server at the moment. I'm not sure if I presented targed data store to esxi host. I'll check it on Monday. Anyway how is should presented target data store and do I need other proxy to replicate VM in hot add mode?

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: replication much slower than backup job

Post by foggy » Sep 11, 2014 3:51 pm

It is still not quite clear from where to where you're replicating. Could you please describe your setup in more detail?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 12, 2014 7:29 am

Vsphere 5.1 cluster with 6x hosts, just one site, don't have DR site, SAN FC HP storage. Veeam 7 server is VM on one of the host all roles on same server (vmware tools installed), proxy is set to addhot mode, target backup storage presented to Veeam server. When I set proxy to auto it used network mode to do backups so I have set it to addhot and backups are flying now(why Veeam is choosing network mode insted of addhot?). Then I tried replicate VM from one host to other on my cluster and is extrymly slow. Please let me know if you need more details

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 12, 2014 8:27 am

szafa wrote:When I set proxy to auto it used network mode to do backups so I have set it to addhot and backups are flying now(why Veeam is choosing network mode insted of addhot?).
If hotadd backup mode is available for all VMs in the job, your proxy server should select this mode when automatic option selected. Can you confirm that there is no failover to network mode and you've used the same VMs to backup when hotadd mode was not selected by the job?
szafa wrote:Then I tried replicate VM from one host to other on my cluster and is extrymly slow. Please let me know if you need more details
Try installing the second proxy and see if it makes any difference. Judging by the bottleneck stats your source storage slows down the whole process. Can you please tell us what operation takes most time if you look at the replication job session? Data transfer?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 15, 2014 9:57 am

If hotadd backup mode is available for all VMs in the job, your proxy server should select this mode when automatic option selected. Can you confirm that there is no failover to network mode and you've used the same VMs to backup when hotadd mode was not selected by the job?
yes, i used same VM and when auto selected transfer was slow so now i set to hotadd mode.

i just check replication job scheduled to run Sunday and it took over 5h and again bottleneck 99% target.
Image
Image

I can try to add other proxy but looks like i have other issue and i don't know what could be wrong. Please let me know what I could check/do to improve replication.

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 15, 2014 5:04 pm

In order to troubleshoot it further, I would suggest doing two things:

1. deploying a secondary proxy and try it in hotadd and network modes
2. change target storage for replication job

Let us know the results of these two tests.

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 16, 2014 3:53 pm

I replicated to other target and was little bit faster but still slow.
Image
Then I added other proxy (mode auto but system again choose hotadd) but still same
Image
Any suggestion why is still slow. Can I check if other proxy was used in replication job?

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: replication much slower than backup job

Post by foggy » Sep 16, 2014 3:55 pm

Right on this screenshot, you can see that the same proxy server (the default one - VMware Backup Proxy) was used as both source and target.

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 16, 2014 3:59 pm

ok so what should I do to force this job to use both proxy's ?

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: replication much slower than backup job

Post by foggy » Sep 16, 2014 4:01 pm

You can explicitly select the required target proxy server at the Job Settings step of the replication wizard.

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 17, 2014 8:13 am 1 person likes this post

Thanks guys, finally I can see improvement. I set explicit proxy for source and target, set both to hotadd mode and replication is much faster now.

is anything else that I can do to improve replication time?

Image

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 17, 2014 8:34 am

I just notice that nbd mode was used for target proxy even I selected virtual appliance mode. Why is that?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 17, 2014 9:00 am

I tried again to set both proxy's to auto selection but again target proxy used nbd mode. Next thing is why is so big difference in read between VM disk (all are on same datastore)?

Image

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 17, 2014 12:50 pm

szafa wrote:I just notice that nbd mode was used for target proxy even I selected virtual appliance mode. Why is that?
Did you disable a "failover to network mode" option in the proxy settings?
szafa wrote: Next thing is why is so big difference in read between VM disk (all are on same datastore)?
What data do you have on the Hard Disk 3 of this VM? What about other VMs in the job?

veremin
Product Manager
Posts: 16884
Liked: 1433 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: replication much slower than backup job

Post by veremin » Sep 17, 2014 1:00 pm

Also, are you positive target proxy is capable of Hot Add mode? Thanks.

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 18, 2014 2:51 pm

I have installed VM Tools on other proxy and system is using hotadd mode but now replication is slower! it drooped from 70MB/s (mix mode, addhot on source and ndb on target or nbd on both) to about 17MB/s (hotadd on both proxy's). What is wrong now?

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 18, 2014 4:05 pm

And what is the new bottleneck in this configuration? Are you running incremental runs all the time?

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 19, 2014 8:03 am

yes, all jobs are incremental.

Bottleneck for NBD : source

18/09/2014 15:29:01 :: Busy: Source 99% > Proxy 41% > Network 7% > Target 44%
18/09/2014 15:29:01 :: Primary bottleneck: Source

Bottleneck for Hotadd : target

18/09/2014 15:56:40 :: Busy: Source 22% > Proxy 17% > Network 1% > Target 66%
18/09/2014 15:56:40 :: Primary bottleneck: Target

I have 2 very same VM's and I tried to run incremental replica on both on almost same time. It resulted in same data transfer and read but processing rate are different 5MB/s(Hotadd) to 46MB/s(NBD). What is causing hotadd mode so slow?

Vitaliy S.
Product Manager
Posts: 22976
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: replication much slower than backup job

Post by Vitaliy S. » Sep 19, 2014 12:32 pm

szafa wrote:I have 2 very same VM's and I tried to run incremental replica on both on almost same time. It resulted in same data transfer and read but processing rate are different 5MB/s(Hotadd) to 46MB/s(NBD). What is causing hotadd mode so slow?
The only explanation I have for this is either low performance of datastore (while doing writes using hotadd mode) or the hotadd proxy server itself that slows down the job. I would suggest contacting our technical team to verify what operation (by looking through the debug logs) takes most of the time on the hotadd run.

szafa
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 10, 2014 3:41 pm
Contact:

Re: replication much slower than backup job

Post by szafa » Sep 19, 2014 12:52 pm

Will do. Thanks for all your help

Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests