Bad Performance of Replication Job

[sceddie]

Hi,

i have a terribly bad performance for my Replication jobs compared to Backup Jobs.

Both Jobs have an identical configuration:
source (same remote physical raid, host and guest)
target (same physical raid and host)
network (1Gb)
proxy (one one the source host and one on the target)

My nightly incremental Backup Job takes usually a few minutes for a 900GB Guest.
The Replication Job als run daily (and not at the same time), takes sometimes several hours showing the target as a bootleneck (99%). The disk latency goes up to sometimes 700 ms. This also happens if a run the Replica Job continously where I would expect that there are only a few changes.

Does anybody have an idea what might going on there? Because of that perfromance I can't use Replica Jobs what I would like to do. Which additional Information might help to analyze this problem?

I'm running Veeam Backup & Replication 6.5.0.128 / 64 Bit, the Vmware Hosts are both 5.1.

Best regards
Arne

[foggy]

Arne, could you please check what transport mode is being used by the target proxy server to populate replica VM data on the target datastore?

[sceddie]

Hi Foggy,

the target proxy is the one that came with the Veeam Installation since the Target and the Veeam are running on the same host. I did not change the configuration for that proxy. The transport mode shows "Automatic Selection".

The source proxy has transport mode "Network".

thanks for looking at it
Arne

[foggy]

What you could do is try to switch the target proxy transport mode (i.e. if hotadd mode is used, set it to use network mode, and vice versa) and check whether replica metadata are stored at the source side. Also, check what operation takes most of the time in the job session log for that VM. Thanks.

[sceddie]

Ok, I looked into the statistics for my replica jobs "hotadd" was used. I will set it to "network". I will post my results.

Maybe one question back: how to "check whether replica metadata are stored at the source side"?

[veremin]

how to "check whether replica metadata are stored at the source side"?

Replication Job Settings -> Repository for replica metadata.

Thanks.

[sceddie]

Ah ok, I can only choose my "Backup Repositories" which are all connected (physically) local.
Looking into the logfile i can see, that reading the remote harddisks do take most of the time. I have rates far below 1MB/s. My Backup jobs run between 30-80MB/s.
All of the other steps seem to be reasonable compared my Backup Jobs e.g. creation and removal of snapshots (which are running fine).

Btw: it is a great software you created here, and I'm confident that this problem will also be solved

[sceddie]

Thanks!
I will do some additional tests, but after running two different Replica Jobs it seems, that changing the transport mode from Automatic (hotadd was used) to network did the job. The two Jobs are finished after a few minutes which is the expected behaviour.

Anyhow I'm courious why using hotadd had such a massive influence on the performance and why the automatic mode has used the wrong setting?
If you need some more infos for your analysis - in case you want to analyse - feel free to drop me a note.

Again, thanks and best regards
Arne

[veremin]

Why the automatic mode has used the wrong setting?

According to the way the backup proxy is selected, this is expected behavior, indeed.

Anyhow I'm courious why using hotadd had such a massive influence on the performance

As to the question regarding slow Hot-Add performance, only the job logs are likely to shed a light on it. So, should need be, please open a ticket with our support team and let them investigate it directly.

Thanks.

[foggy]

Anyhow I'm courious why using hotadd had such a massive influence on the performance and why the automatic mode has used the wrong setting?

Actually, this issue had already come up previously.

If you need some more infos for your analysis - in case you want to analyse - feel free to drop me a note.

Yes, it would be much appreciated if you could open the case with our technical support to help investigating it.