Discussions specific to the VMware vSphere hypervisor
superdekster
Influencer
Posts: 12
Liked: 6 times
Joined: Oct 23, 2013 9:32 am
Full Name: Aleksandr Alpatov
Contact:

Re: Replication Destination Datastore performance

Post by superdekster » Nov 01, 2013 10:26 am

I have the same issue as lars@norstat.no - Case # 00446903. And Tech Support tell us to wait patch 2 on Veeam 7 as it will use different VDDK version.
We have 4000 MB/s performance in full replicas and ~30 MB/s in increment replicas with ~11 000 IOPS.
I hope R&D will take care about this big problem.

Vitaliy S.
Product Manager
Posts: 22284
Liked: 1411 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication Destination Datastore performance

Post by Vitaliy S. » Nov 01, 2013 11:08 am

Aleksandr, do you use hotadd or network mode for the target proxy server?

superdekster
Influencer
Posts: 12
Liked: 6 times
Joined: Oct 23, 2013 9:32 am
Full Name: Aleksandr Alpatov
Contact:

Re: Replication Destination Datastore performance

Post by superdekster » Nov 03, 2013 12:13 pm

Vitaliy S. wrote:Aleksandr, do you use hotadd or network mode for the target proxy server?
Vitaliy, I used hotadd for my target proxy because we have 8 Gbps FC SAN infrastructure and need good performance to do replicas. But when we detected hotadd problem with Veeam tech support I switched target proxy to network. And waiting patch 2 will solve the problem

Vitaliy S.
Product Manager
Posts: 22284
Liked: 1411 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication Destination Datastore performance

Post by Vitaliy S. » Nov 03, 2013 1:24 pm

Thanks for the update. Did switching to network mode did your performance any better? Just curious.

superdekster
Influencer
Posts: 12
Liked: 6 times
Joined: Oct 23, 2013 9:32 am
Full Name: Aleksandr Alpatov
Contact:

Re: Replication Destination Datastore performance

Post by superdekster » Nov 04, 2013 3:19 am

Vitaliy S. wrote:Thanks for the update. Did switching to network mode did your performance any better? Just curious.
A little bit, yes. To 40-50 MB/s

lars@norstat.no
Expert
Posts: 109
Liked: 14 times
Joined: Nov 01, 2011 1:44 pm
Full Name: Lars Skjønberg
Contact:

Re: Replication Destination Datastore performance

Post by lars@norstat.no » Nov 04, 2013 12:06 pm

Think i found a possible reason to why hotadd is not working as it should, at least in my case although i haven't been able to test it yet.

In the documentation for Vsphere 5 it states:

"HotAdd cannot be used if the VMFS block size of the datastore containing the virtual machine folder for the target virtual machine does not match the VMFS block size of the datastore containing the proxy virtual machine. For example, if you back up virtual disk on a datastore with 1MB blocks, the proxy must also be on a datastore with 1MB blocks."

Now i stored my proxies on the production SAN that where created using Vsphere 5.1 and the Replication target was created before that and then upgraded to the new VMFS version 5.xx.

But there is a difference, source SAN has version 5.58 and target SAN has version 5.54, and then i found this.

http://blogs.vmware.com/vsphere/2011/07 ... mfs-5.html

"VMFS-5 upgraded from VMFS-3 continues to use the previous file block size which may be larger than the unified 1MB file block size."
"VMFS-5 upgraded from VMFS-3 continues to use 64KB sub-blocks and not new 8K sub-blocks."

This can explain the unusally high iops and poor performance. Something to look into anyway, although i have switched back to SAN/NBD mode now and deleted all my proxies.

Gostev
SVP, Product Management
Posts: 24016
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Replication Destination Datastore performance

Post by Gostev » Nov 04, 2013 7:44 pm

Hi Lars, thanks for this, I will run this by the devs.

emachabert
Veeam Vanguard
Posts: 368
Liked: 164 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Replication Destination Datastore performance

Post by emachabert » Nov 05, 2013 3:34 pm 2 people like this post

Hi,

Sharing experience is priceless :-)

I am also working on a replication setup these days (HP 3Par storeserv to HP P2000) and I am using two target proxies, one using HotAdd and the other one using network mode. The source proxy is using direct SAN access.

Network mode is used on replication jobs running every hour as you don't loose time adding and removing disks.
HotAdd is used on jobs replicating once a day or 3 times a day (8 am, 12 am, 6 pm), as it has to replicate more data since replication occurs less often (If I remember correctly the documentation says hotadd is more efficient)

I was sceptical on the throughput I was having with the hotadd proxy. It was 3 times slower that network mode.
Reading this post made me look at the IOPS on the P2000 datastores receiving replicas, and guess what ...... it was topping at 12 000iops during replication, killing the array.

So I added a network card to the hotadd proxy, connected to the same network and the same vswitch as the management port of the ESXi it was running and replicating on. Doing this prevents NBD trafic going out through the physical network.

And....guess what....throughput became 3 times higher....

Thanks !
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

superdekster
Influencer
Posts: 12
Liked: 6 times
Joined: Oct 23, 2013 9:32 am
Full Name: Aleksandr Alpatov
Contact:

Re: Replication Destination Datastore performance

Post by superdekster » Nov 10, 2013 8:33 am

lars@norstat.no wrote:Think i found a possible reason to why hotadd is not working as it should, at least in my case although i haven't been able to test it yet.

In the documentation for Vsphere 5 it states:

"HotAdd cannot be used if the VMFS block size of the datastore containing the virtual machine folder for the target virtual machine does not match the VMFS block size of the datastore containing the proxy virtual machine. For example, if you back up virtual disk on a datastore with 1MB blocks, the proxy must also be on a datastore with 1MB blocks."

Now i stored my proxies on the production SAN that where created using Vsphere 5.1 and the Replication target was created before that and then upgraded to the new VMFS version 5.xx.

But there is a difference, source SAN has version 5.58 and target SAN has version 5.54, and then i found this.

http://blogs.vmware.com/vsphere/2011/07 ... mfs-5.html

"VMFS-5 upgraded from VMFS-3 continues to use the previous file block size which may be larger than the unified 1MB file block size."
"VMFS-5 upgraded from VMFS-3 continues to use 64KB sub-blocks and not new 8K sub-blocks."

This can explain the unusally high iops and poor performance. Something to look into anyway, although i have switched back to SAN/NBD mode now and deleted all my proxies.
Hi, Lars. I checked your interesting idea about different VMFS versions on source and destination datastores. In my case it did't help, performance still poor. Sadly.

andrewpetre
Influencer
Posts: 15
Liked: 4 times
Joined: Nov 11, 2013 10:53 pm
Full Name: a p
Contact:

Re: Replication Destination Datastore performance

Post by andrewpetre » Jan 29, 2014 12:30 am

I'm so glad this thread was here. I've been tearing my hair out trying to imagine why my brand new SAS array is so slow. We were getting 5-7MB/sec processing rates in replication jobs with maxed 10k/sec iops readings.

Changed the remote target proxy to Network Mode and now we're measuring 60+ MB/sec processing over the 1GB connection.

Current build 7.0.771, btw. So this is not fixed in the patch. I don't think that was in doubt, but just for reference.

Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests