Upgrading v11 to v12 Slowing D2D performance?

[mpark]

Currently working with support but wanted to see if anyone else had similar issue and have some insights. Case # 05945109

We recently upgraded our backup infrastructure from VBR v11 to v12. Since then, I noticed slow backup performance.
We run nightly reverse increment backups that usually finished in 6 hours, now finishing in almost 12 hours. Basically doubling the time it takes to backup.

I checked the backup summary and it shows. (v12)
Bottleneck: Source 0% | Proxy 2% | Network 0% | Target 96%

These numbers looked odd so I went back to my previous backup summaries when Veeam was on v11 and this is what I see.
Bottleneck: Source 78% | Proxy 76% | Network 67% | Target 93%

We have made no changes to our backup infrastructure that would impede on backup performance. Has anyone had similar issue I'm experiencing?

Thanks,
mp

[Mildur]

Hi Min

Thanks for sharing your case number. I see that there is an active discussion on going.
I'll recommend to keep working with your assigned support engineer.

Best,
Fabian

[bytewiseits]

Check if you were using either of the UseUnbufferedAccess and DisableHtAsyncIo registry values previously as they have now been replaced with the DataMoverLegacyIOMode value. (refer V12 Whats New pg6)

Even if you had not used those keys in V11, is still worth a shot testing V12 with DataMoverLegacyIOMode set to 1 and see if your storage performs any better. If no better (or worse) just remove the key to revert.

From what we saw in many of our VBR setups, the disk access 'improvments' they they made with V11 and further again with V12 just killed our disk storage performance. Setting UseUnbufferedAccess in V11 and DataMoverLegacyIOMode in V12 more than doubles performance in some of our particular instances (particularly on our Ubuntu XFS repos).

[mpark]

Hello Fabian,
Yes I'm still working with the support to no avail. I keep repeating the fact that performance was not an issue with v11 yet but keeps receiving generic instructions for troubleshooting that really doesn't help.

Hello Bytewiseits,
Thanks for the suggestion. We did not use those registries in the past since we had no issues before.
I tried the registry DataMoverLegacyIOMode (DWORD 1). Unfortunately I did not see any improvements. We are using CBT fastclone and what used to process around 40MB/s average is now down to ~10MB/s after the upgrade.

mp

[naatalla]

Hello,
i have de same problem case 05948611
Any solutions
Regards

[mpark]

Hello naatalla,
I believe I got my backup back to where it is (still need more time to test). I have made no other changes since my last post except marking the job "high priority" and ran Active full backup. Even when there are no jobs running, seems like jobs marked with high priority gets processed faster.

I'm going to start reverting changes, high priority and removing the registry key DataMoverLegacyIOMode one at a time to see if the job processing speed stays the same.

mp

[jbrullemans]

Hi Mpark,

I'm also working with veeam on the same issue. Case 05947572.
Before V11 i had a reverse incr. backup speed of 500 MB/s after V12 only 60 MB/s.

I have now set the regkey but i need to wait till tomorrow for the resuls.
What else did you did to get the performance back?

John

[mpark]

Hello John,
I had done the following so far as a "fix" to the problem:
-Run Active full backup
-DataMoverLegacyIOMode (DWORD 1) registry key
-Mark the job as "high priority"

As a test, I removed DataMoverLegacyIOMode registry key since my last post. Backup speed was again impacted. I added the registry key back in and the speed came back. This reg key is definitely doing something to help backup go faster.

Next I'm going to remove the backup job from "high priority" to see if anything changes

mp

[Gostev]

It won't change anything, this setting only determines the placement of the job's machines in the overall processing queue when multiple jobs are running simultaneously.

[mpark]

Update:
Changing High Priority setting to the backup job indeed did nothing to the backup performance.
Only thing that would have helped with my case is adding DataMoverLegacyIOMode and/or running the active full backup of the job

mp

[naatalla]

Hello,

Still no solution on my side, and the veeam support doesn't seem to be working either
the Full backup reaches 2Gbits/s but once it switches to reverse incremental I get 30Mbits/S
I think it has a link with the CBT and FAST CLONE, which is reset with the full backup and then reactivated for reverse incremental .....

[jbrullemans]

Hi,

I have received an update from Veaam Escalation.
They have a hotfix for this it contains new Veeam Agent and my perfomance is back to normal again!
I would contact support if i was you.

[Daniel_J]

What did the hotfix address?

[stevehei]

@jbrullemans
What was the name of the hotfix? We're having exactly the same issue. We'd like to suggest it to our support rep.

[HannesK]

as far as I see, that was the reverse incremental block cloning issue tracked as bug #497629

[stevehei]

Thank you HannesK - do you have a link to this info?

[HannesK]

I looked into the ticket 05947572. The support engineer can access that ticket.

[stevehei]

Hello all. Thank you Hannes - this information helped our support engineer provide us with the hotfix!

It does indeed seem to be related to reverse incremental jobs. Actual info on what the bug is may be forthcoming.

Hotfix given was named: TF493986

[mpark]

I have also applied the private fix along with the latest V12 update 12.0.0.1420 P20230412 and speed came back to normal.

For the record, removing DataMoverLegacyIOMode value had no impact on performance
Thanks all!

mp

[Kwa-GJ]

We had exactly the same issue.
Openend a case and mentioned the case numbers and hotfix in this topic.
A few hours later we recieved the hotfix.

[ATDCO]

I don't think they take much care with community users, I opened a case and received no such hot fix.
I can only assume I will have to wait for an update to be release which addresses this particular performance issue.

[Gostev]

Community Edition does not support installation of hotfixes in principle, so there's no sense for you to obtain one.

Note: TF493986 hotfix is for reverse incremental backup mode not leveraging fast cloning on ReFS/XFS.
Normally you should not be using reverse incremental backup mode on ReFS/XFS (no benefits, only drawbacks).

[cptkommin]

Hi,

Any update on this? I see a few people saying private fixes. Perhaps I can reference your case numbers in my open case with Veeam Support?

Tried the regkey for v12, no change in performance.

Thank You

[HannesK]

Hello,
and welcome to the forums.

Above you can find bug numbers and support case numbers. If you already have a support case open, then support should find out quickly, because the one above is "well known".

Best regards,
Hannes

[ma-strs]

Hello all,
I experienced the same issue after upgrading to version 12. My backup performance plummeted. Especially the reverse incremental jobs. I did the normal steps of upgrading the backup chain format, adding/removing DataMoverLegacyIOMode, UseUnbufferedAccess, and DisableHtAsyncIo reg keys. Nothing has restored our pre v12 backup performance. I changed my reverse incremental jobs to forever forward for a very minor improvement. I've had my case (02319536 and 06086074) open for 4 months now and haven't seen any beneficial suggestions (very unimpressive response from Veeam support overall)
I do think doing an active backup or deleting the existing backups and starting fresh would probably help tremendously. Unfortunately that would require 60 TB of space I don't have.