Poor Vmwares performance issue when Replication is running

[cdechamps]

Hello,

I have recently renew our vmware infrastructure (2Hosts with 2CPUs AMD 6274 and 1EMC VNXe3300 30HDDs)
A third host with local storage is used for replication to a DRP site.
For the first replication I have kept the third host locally so I can run the job at full speed but I have encountered a big issue I didn't have before.
When running the replication job it has a huge impact on the performances of some Vmwares even if not among the vmwares replicated...
The job is configured as follow:
Source proxy: Vmware (so I can use the Hotadd feature)
Target proxy and repository server: physical server (traffic:nbd)
Job config: WAN, Best compression, slow bandwith option activated

The 2 hosts have a total of 64 Cores and 160GB RAM (currently only 25% used by the Vmwares so I am far to run out of ressources) so could it be due to the EMC or network (it is a Gbps network)???
I receive alerts for source proxy as it uses 100% of the CPUs when the job is running but I guess this is normal. As those Vmware are not important it is not an issue but apparently it is an impact on other Vmwares...

I am going to make some tests to try to determine the source of this performance issue but any help will be greatly appreciate

thank you

[Gostev]

Yes, it's normal for proxy to eat 100% CPU if your connection to DR site is fast. So as far as host CPU and memory, are you saying that the host load stays at 25% for both CPU and memory?

[cdechamps]

Hi,

The Vmware used as proxy takes all its available ressources but of course (and also fortunately) it doesn't use all the Host resources. What I was saying is that currently the hosts total charge is about 25% for the entire Vmwares.

After a few tests I have figured out the source of our problem. When using a Vmware as source proxy together with a high compression rate (Best) the amount of Iops increases enormously and so it has an impact on all Vmwares stored on the same disks. In my case when the job was running the Iops on the Raid on which the Vmware to be replicated was increased until 5000iops... and so it impacted all other Vmware stored on that same Raid.

I haven't checked deeply yet but is there a way to limit Iops in Veeam backup ? I know it is possible in Vmware but I have to do it per disk/per Vmware which makes no really sense as Veeam will replicate several Vmware. I would like to know if it is possible to limit that by Veeam itself.

thank you

[Gostev]

Sure, this is easy - just reduce the amount of CPU resources available to proxy VM in VMware (remove some vCPUs, or reduce VM CPU shares). This will directly affect the amount of data the proxy will be able to fetch and process per second, as proxy CPU resources will become a bottleneck. Thanks!

[lobo519]

Hi,

The Vmware used as proxy takes all its available ressources but of course (and also fortunately) it doesn't use all the Host resources. What I was saying is that currently the hosts total charge is about 25% for the entire Vmwares.

After a few tests I have figured out the source of our problem. When using a Vmware as source proxy together with a high compression rate (Best) the amount of Iops increases enormously and so it has an impact on all Vmwares stored on the same disks. In my case when the job was running the Iops on the Raid on which the Vmware to be replicated was increased until 5000iops... and so it impacted all other Vmware stored on that same Raid.

I haven't checked deeply yet but is there a way to limit Iops in Veeam backup ? I know it is possible in Vmware but I have to do it per disk/per Vmware which makes no really sense as Veeam will replicate several Vmware. I would like to know if it is possible to limit that by Veeam itself.

thank you

I am interested in this as well but on the target side. I'm not sure that limiting the vCPUs is going to work as my proxy really isn't using much CPU (750mhz - 10%) to begin with but it is chewing up 5000 iops. I was also trying to limit the iops via VMware but as you stated, I don't think that will work...

Any other thoughts? Were you successful in making any changes?