Discussions specific to the VMware vSphere hypervisor
lobo519
Expert
Posts: 297
Liked: 34 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Replication Destination Datastore performance

Post by lobo519 » Nov 09, 2012 2:58 pm

I am trying to control the IOPS generated on our destination datastore for our replication jobs. Right now they are generating about 5000 IOPS and its killing the datastore. I would normally just use vCenter to limit the IOPS under resource allocation but because the drives are being added and removed constantly (Hotadd) this doesn't work.

I tried reducing the CPU shares which had no effect. (No much CPU usage anyway).

Any idea as to how I can control this?

We are currently on and aging MD3000i and the replication datastore is a separate RAID 10 so it doesn't affect other VMs. We are looking to replace the MD with an Equallogic and even though the IO size is very small - I think the replication jobs will still impact the other VMs on the array.

tsightler
VP, Product Management
Posts: 5297
Liked: 2147 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Replication Destination Datastore performance

Post by tsightler » Nov 09, 2012 3:54 pm 2 people like this post

Force the target proxies to network mode. This will likely have no major performance impact but will probably reduce the I/O load significantly. Let us know if that works.

lobo519
Expert
Posts: 297
Liked: 34 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Replication Destination Datastore performance

Post by lobo519 » Nov 09, 2012 7:28 pm

Wow - Seems to work great! Thanks!

Performance seems about the same and I/O down to like 200 - 400

Why the difference?

tsightler
VP, Product Management
Posts: 5297
Liked: 2147 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Replication Destination Datastore performance

Post by tsightler » Nov 09, 2012 8:39 pm

That is still being researched by R&D, but for some reason hotadd mode has the behavior on at least some configurations. There have been other threads on the forum that talk about this same thing. For now the easiest workaround is to use network mode. Actually, in some cases we see performance improvement due to the lower overhead of the target I/O and the decrease in setup time that's typically required on the proxy for hotadd.

DaxUK
Influencer
Posts: 13
Liked: 1 time
Joined: Aug 25, 2011 7:45 am
Full Name: David Wells
Contact:

[MERGED] IOPs requirement for replication target SAN

Post by DaxUK » Feb 04, 2013 9:48 am

Hi,

I have had a search on the forums and couldn't find anything on this, apologies if this is covered before.

We are seeing abnormally high IOP requirements on our SAN at our DR site when using replication of a number of VM's from our live environment. We are using Virtual appliance mode utilizing Proxies in each site

We are seeing spikes of up to 7k IO/sec when the replication is occurring on the DR SAN. Is there anyway to reduce this?

Also we wanted to confirm the actual process of disk activity in a replication job. Obviously in a backup job writing to a Veeam repository the dedup occurs and changes are written to the repository.

However in a replication job the data is written to our SAN at our DR site but this is obviously not a Veeam repository so we wondering how the data is written to the replica target, is the disk completely rewritten each time a replication job occurs? which may explain the huge IO requirements.

Many Thanks for any help.

Dave

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: IOPs requirement for replication target SAN

Post by chrisdearden » Feb 04, 2013 10:34 am

Can you try and switch your target proxies to network mode - do you see the same issue?

DaxUK
Influencer
Posts: 13
Liked: 1 time
Joined: Aug 25, 2011 7:45 am
Full Name: David Wells
Contact:

Re: Replication Destination Datastore performance

Post by DaxUK » Feb 04, 2013 11:35 am

Hi,

We have now tested this and it has greatly reduced the IOPs heading to the destination SAN to 200-400 from 7000.

Is there any update on when this is likely to be fixed as this is not optimal for us as our Backup proxy in the target site has to traverse a firewall to talk to Vcenter/ESXi and we would like to avoid this for backup data.

Regards
Dave

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Replication Destination Datastore performance

Post by chrisdearden » Feb 04, 2013 12:14 pm

Hi Dave,
Is the firewall traversal due to security policy? Assuming your Target proxy is a VM ( which it would need to be to use hotadd ) , connecting it to e network with vmkernel access is technically not a problem , but I appreciate there are often many other reasons for why this would not be implemented.

DaxUK
Influencer
Posts: 13
Liked: 1 time
Joined: Aug 25, 2011 7:45 am
Full Name: David Wells
Contact:

Re: Replication Destination Datastore performance

Post by DaxUK » Feb 04, 2013 12:46 pm

Hi,

Yup it is a security policy reason why we cant put the proxy in the same front end network as the vCenter server.

That is why we are quite keen to get hotadd working to elevate the firewall from the equation.

Regards
Dave

foggy
Veeam Software
Posts: 17717
Liked: 1482 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Destination Datastore performance

Post by foggy » Feb 05, 2013 1:53 pm

DaxUK wrote:Is there any update on when this is likely to be fixed as this is not optimal for us as our Backup proxy in the target site has to traverse a firewall to talk to Vcenter/ESXi and we would like to avoid this for backup data.
Still under investigation. The problem is that this issue is not 100% reproducible and more likely comes from VMware side under certain circumstances. Our R&D worked with VMware on it but did not come to a reasonable resolution so far.

It would be much appreciated if you could open a support case and provide debug logs and environment-specific information to our R&D.

There is also another thread regarding the same problem.

lars@norstat.no
Expert
Posts: 109
Liked: 14 times
Joined: Nov 01, 2011 1:44 pm
Full Name: Lars Skjønberg
Contact:

Re: Replication Destination Datastore performance

Post by lars@norstat.no » Oct 30, 2013 10:47 am

I have the same problem with high IO on target SAN. I have had slow performance since upgrading to v7 so yesterday i cleaned out the database, deleted all replicas and started over. The replication of about 16TB took a few hours the speed where screaming fast and i maxed out my target SAN at about 4Gb/s. From IBM Storage manager i can see that Read Percentage is about 1.5 % and IO is about 800-900 IOPS.

Then when i try the first incremental replication on a small VM with only 40 GB disk i get a Read percentage of about 34 % and the IO rise to about 11 000 IOPS, if i try 2-3 VM's i max out on IO at about 36 000 IOPS on my DS4700. The read percentage is still about 34 %. The speed of transfer is also pretty slow.

My source SAN is humming along at about 4000 IOPS, but this is the production SAN so there is probably more going on there as well .... The source SAN is a Storwize v7000.

I'm running Vsphere 5.1 and Veeam v7 with the latest patch.

This is 100% reproducible.

foggy
Veeam Software
Posts: 17717
Liked: 1482 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Destination Datastore performance

Post by foggy » Oct 30, 2013 11:23 am

Have you tried switching to network mode? Am I understanding right, that you did not observe this behavior prior to v7 upgrade? What are the bottleneck stats for this job?

lars@norstat.no
Expert
Posts: 109
Liked: 14 times
Joined: Nov 01, 2011 1:44 pm
Full Name: Lars Skjønberg
Contact:

Re: Replication Destination Datastore performance

Post by lars@norstat.no » Oct 30, 2013 11:53 am

Switching to network mode is not really a solution, i'm using hotadd because i have a 4Gb/s fiber link to the disaster site and only a 2Gb/s network link and even if i could get a faster network link i would not run all this traffic on my production LAN. I'm sure that IO would drop a lot if using the network, but i'm not using that option.

I had the same problem before v7 as well, the reason i started over after the upgrade was that it was even slower than before. Down from 50 MB/s proccessing rate to about 600KB/s.

When the system was working right, i think the last time was version 6.1 i saw a processing rate of about 3GB/s on my larger servers on incremental jobs using hotadd.

The bottleneck stats is not surprisingly showing
Source 4% > Proxy 25% > Network 7% > Target 95%

lars@norstat.no
Expert
Posts: 109
Liked: 14 times
Joined: Nov 01, 2011 1:44 pm
Full Name: Lars Skjønberg
Contact:

Re: Replication Destination Datastore performance

Post by lars@norstat.no » Oct 30, 2013 9:32 pm

As you can see from my newest post i went back to using physical Veeam server and direct SAN so my previous has no relevance for me anymore, but hotadd need to be fixed either by you or Vmware because it's obviously broken ....

Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Replication Destination Datastore performance

Post by Gostev » Oct 31, 2013 6:20 pm

We write data to replica disks through VMware API, so that's where the issue sits... I wonder if we can may be bypass it, unless the issue is in actual hypervisor.

Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests