Slow replication with 12.1.2.172

[jfargo]

Is anyone else seeing slow VM replication after the upgrade to 12.1.2.172? I've opened ticket 07275542 but we haven't made any progress.

I'm not seeing any slowdown of backups or with the SOBR offloads.

For comparison, the Exchange replication job ran on:

Code: Select all

5/15 transferred 138 GB at 37 MB/s 2hrs 52 mins
5/16 transferred 130 GB at 36 MB/s 3 hrs 9 mins
5/17 transferred 202 GB at 36 MB/s 2 hrs 55 mins

After upgrade:

Code: Select all

5/22 transferred 206 GB at 12 MB/s 11 hrs 49 mins
5/23 transferred 133 GB at 11 MB/s  8 hrs 28 mins
5/24 transferred 134 GB at 11 MB/s 8 hrs 28 mins
5/25 transferred 143 GB at 9 MB/s 12 hrs 4 mins
5/26 transferred 190 GB at 14 MB/s 7 hrs 19 mins
5/27 transferred 68 GB at 9 MB/s 7 hrs 32 mins
5/28 transferred 73 GB at 10 MB/s 7 hrs 12 mins

The SQL replication job ran:

Code: Select all

5/16 transferred 974 GB at 93 MB/s 4 hrs 45 mins
5/17 transferred 845 GB at 91 MB/s 4 hrs 24 mins
5/18 transferred 585 GB at 127 MB/s 2 hrs 53 mins

After upgrade:

Code: Select all

5/23 transferred 489 GB at 19 MB/s (20 hours and 45 mins) 1 VM failed
The run on 5/24 was missed because the 5/23 job and retry is still running
5/24 retry of 5/23 job just finished, transferred 145 GB at 8 MB/s 14 hrs 42 mins (1 VM)
5/26 transferred 533 GB at 49 MB/s 26 hrs 49 mins
5/27 transferred 356 GB at 23 MB/s 11 hrs 33 mins

John

[david.domask]

Hi John,

Thank you for sharing the case number and the information -- I can see there's been a bit of back and forth with the Engineer, and that the last suggestion was to try Virtual Appliance mode (hotadd), but to confirm, it didn't show any improvement? Is it happening with all proxies that you have in the environment or just specific ones?

At the moment, no known issues that present with this behavior, so please continue with the Support Case and we will monitor, but in a quick check on the case I didn't see the above points confirmed explicitly, so if you would, please check and update your Engineer.

If there are concerns on elements of the case, please reach out to Support Management and explain the situation, Support Management will review the details and assist.

[s88urd85]

Hi,

A customer of ours is experiencing the same issue.
Using NBD transport mode the transfer speed of all replication jobs have plummeted from around 37MB/sec to around 8MB/sec.

[s88urd85]

Sorry, not much context on my previous reply.

Around 15 replication jobs all using a single physical target side proxy (so is using NBD mode) this is the only proxy available at the target side.
Source side is using SAN snapshots.

Jobs are spread out as much as possible so the proxy is not too overloaded, and were completing in good time prior to updating to 12.1.2.172.
Target side proxy has been rebooted and all remote components updated.
All replication jobs are impacted by this drop in performance and no other changes have been made (that I have been made aware of) to the infrastructure.
Currently I am awaiting to be assigned a case administrator so I can get this logged by the proper channels with Veeam.

[nicolas.pro]

Hello, I've the same issue from upgrade 12.1 to 12.1.2 only with replica.

Job was between 100 and 500 MB/s speed and now, they are max 6-8 MB/s... I opened a ticket from last Friday but I received a troubleshooting who wasn't linked with my problems. My configuration has not changed between these two upgrades...

I tried to clone a replica job, delete old job and old replica, run the new job and it's working ! But my joy was short-lived because the day after, my jobs revolve around 8 MB/s...

Any solutions ?

Thank you.

Nicolas.

[s88urd85]

Nicolas, are your replica jobs using NBD transport mode too?

[jfargo]

I'm glad we're not the only ones seeing the slowdown. So far the engineer is saying that nobody else is seeing the issue.

The replication jobs were all running with NBD transport before the update so nothing has changed there. We've rebooted the proxies and the metadata repository at the source but that has not changed anything.

I don't want to change my target proxies to hotadd. Those proxies also backup VMs locally and I use SAN mode to mitigate the stun on snapshot deletion. I have tested the proxies in hotadd mode and we're seeing the following errors:
5/29/2024 10:03:35 AM :: Unable to hot add target disk xxxxxxx_3-000021.vmdk, failing over to network mode...
5/29/2024 10:04:01 AM :: Hard disk 1 (150 GB) 6.2 GB read at 9 MB/s [CBT]

The manual test for appliance mode in https://www.veeam.com/kb1184 worked fine. We've gone through fio tests for throughput but nothing so far. The next step is they want me to replicate a VM using different storage at both the source and target side, thinking it's a problem with our Pure SANs.

Hopefully when the different storage test comes back with the same result, we can escalate the ticket.

John

[s88urd85]

Hi John thanks for working with them on this.
Exactly the same situation as myself. Jobs were always using NBD mode (and although not the quickest) they were completing within an acceptable time frame and we were seeing on average between 20-40MB/sec.
After the update to 12.1.2.172 this has dropped to around 8MB/sec on literally every replication job.

[SidWise]

Just wanted to say we have the same issue, currently working with support. Case # 07290171.

[nicolas.pro]

Nicolas, are your replica jobs using NBD transport mode too?

Yes, my case #07283597.

screen : https://postimg.cc/BPJsh3fj

[halllowien]

Hello,

we experience the same Problem since Upgrade.
Source is hot add, destination is NBD with physical Servers as Proxys.

Greets
halllowien

[Remyc]

Hello,

Following the update to 12.1.2.x of one of my clients, I also encountered the same problem.
Replication in place for several years. Constant flow. Source proxy in Direct SAN mode. Destination proxy in NBD mode. Change from 60 MB/s to 10 MB/s for all jobs.

I confirm that setting up a destination proxy in Virtual Appliance mode allowed to find correct flow rates (100 MB/s ^^).

It also seems that the new replication jobs (created after the update) do not have the problem. TO BE CONFIRMED.

At this moment, i did not open a case.

[HannesK]

Hello,
we changed default VDDK parameters for NBD in 12.1.2. The change increases backup speed on average by 2x (of course, only if there is no bottleneck) and should also increase replication performance (but not that much). I tested that with several customers, as one can apply the parameters also manually before 12.1.2.

Very likely, one can just turn back the VDDK parameters to pre-12.1.2, but we should also find out why you had such a dramatic impact. I will talk to support.

It also seems that the new replication jobs (created after the update) do not have the problem. TO BE CONFIRMED.

that's interesting, please let us know the outcome.

Best regards,
Hannes

[HBSR]

Just wanted to say we have the same issue, currently working with support. Case # 07290171.

[kubimike]

You writing to a ReFS repo ?

[s88urd85]

As it's a replication job the target is not a repository but a VMFS datastore (in my case as we use VMWare)
Source is either direct from VM or VM from SAN Snapshot, so no repository really involved as such (apart from for metadata)

[kubimike]

Ok darn, I was hoping these were read from backup from a ReFS repository. Thanks

[nicolas.pro]

Hello,

Do you have a solution about this case ?

Best regards.

Nicolas.

(#07283597)

[Manu80]

Hello,
same problem for us.

Do you have solution about this ?
(#07300729)

[Manu80]

Hello,

Following the update to 12.1.2.x of one of my clients, I also encountered the same problem.
Replication in place for several years. Constant flow. Source proxy in Direct SAN mode. Destination proxy in NBD mode. Change from 60 MB/s to 10 MB/s for all jobs.

I confirm that setting up a destination proxy in Virtual Appliance mode allowed to find correct flow rates (100 MB/s ^^).

It also seems that the new replication jobs (created after the update) do not have the problem. TO BE CONFIRMED.

At this moment, i did not open a case.

Hello,
in our case recreating jobs after update not solve the problem, same slow speed on replication.

[SnakeSK]

Same here on Hyper-V. ReFS to ReFS, replication jobs require 3-4 times the amount to finish. Case 07303960

[Manu80]

Hello,
for us we use physical proxy with NBD network mode.
I replace all proxy with VM and network mode hotadd, and now all works fine.
I think bug are with NBD after latest veeam upgrade.

[s88urd85]

Anyone have any further progress with this at all?

[HannesK]

Hello,
@SnakeSK: that sounds like a "backup copy job", which works different than replication jobs (hypervisor to hypervisor vs. repository to repository)

07275542 - the customer stopped answering support, so no outcome here
07290171 - after changing VDDK settings with help of support, he had better speed again, but not pre-12.1.2 performance
07283597 - no outcome yet
07300729 - as he wrote above, he switched to HotAdd. Workaround, but no real soluion

To be clear: the VDDK settings only apply to VMware with NBD replication mode. Definitely not all customers are affected, because otherwise we would have many more cases. For most customers, they get better speed. But there are some cases where things slowed down.

Best regards,
Hannes

[nicolas.pro]

Hi,

For "07283597 - no outcome yet", I refused to give a feedback because Veeam ask me to make a workaround with a config file on each proxies servers and co in my prod environment. Veeam isn't sure to will be functionnal after and I spend a lot of my time for this subject.

So, on the only 1 good replica job, we saw this following message : "Resetting CBT per job settings for active fulls" (on each VMs into the replica job). The second one is "Creating VM snapshot".
On my jobs who are slowly, we don't have the first message. We have directly the second.
Into all of my replica jobs settings ("advanced...", tab "vSphere"), all jobs are configured with CBT enabled (both options enabled too).

Best regards,

Nicolas.

[HannesK]

Hello,

ask me to make a workaround with a config file on each proxies servers and co in my prod environment

yes, that's how VDDK settings are set. here a customer shows the default settings of 12.1.2. Support should have tried to roll back to the pre-12.1.2 settings.

Best regards,
Hannes

[nicolas.pro]

Hi,

So, I don't have any problem with my backup jobs. Only replica.

EDIT :

Re: Solution for poor NBD performance

Post by Gostev » May 22, 2024 6:48 pm
Manual tuning is not something I would recommend because RAM consumption per NBD connection must be coordinated with the maximum NBD connections limit. Otherwise your jobs will start experiencing random failures due to OOM errors. We've seen this a few times at customers who beta tested this tuning.

Will we have an official fix from Veeam in the coming days?

Nicolas.

[HannesK]

Hello,
no fix is planned for now because the new settings make it faster for most customers. Now we need to find out why it it gets slower for some customers

Best regards,
Hannes

[nicolas.pro]

Gostev, Chief Product Officer, nevertheless said that making changes manually could cause problems for some. I don't want to create another problem for us.

I hope we will have an official patch in the next few days and not just unofficial solutions.

[Gostev]

Actually, rollback to pre-12.1.2 values is totally safe. My statement was about implementing NEW, high-performance NBD settings, which do put more pressure on host resources (but also double the performance in most cases). In fact, as a result of field testing we made some settings less aggressive in 12.1.2 than originally planned, because a few customer did experience some random job failures with initial settings due to host overload.