VM Copy & FF Incremental Job Processing speed issue after V12 "upgrade"

[ATDCO]

With no other changes to jobs, settings or hardware, I've dropped in processing speeds from 49MB/s to 11MB/s

Does anyone have an answer as to why this may have occurred with no other changes?

Details: ESXI 6.7.0 Update 2 (Build 13981272), Windows 2012 server.

Case #06041635

[Mildur]

Hi Richard

Welcome to the forum.
Please give our support engineers time to take over your case (Support on best effort for the community edition).
Which proxy transport mode are you using for the VM Copy?

Best,
Fabian

[ATDCO]

This is using a VMware Backup Proxy if that is what you are asking.
Veeam running on a secondary win10 machine with directly connected portable drives for offsite storage for the VMCOPY.
FF Incremental is targeting network storage drives.

[Mildur]

Thank you. I was asking about the transport mode:
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Have you used one of the following keys in pre-V12? UseUnbufferedAccess, DisableHtAsyncIo?
They were replaced by a new key.
- Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
- Name: DataMoverLegacyIOMode
- Type: DWORD
- Value: 1

Please remove the key if the jobs didn't performed better with it.

Best,
Fabian

[ATDCO]

Transport mode was set to Automatic both in V11, and now in V12. What do you recommend considering the backup infrastructure used?

Agent backups which are directed to NAS are much better processing speeds by comparison.

I have made no registry modifications prior to today. I have made the suggested change adding the DataMoverLegacyIOMode and value.
I will report back tomorrow after some incremental backups run overnight.

[Mildur]

Hi Richard

Please let me know after your tests.

Best,
Fabian

[ATDCO]

It appears there is no change in the FF Incremental processing speeds after the registry change.
the reported speed is only 12MB/s which is a definite problem going forward.

Do you have any other suggestions to what might have occurred with only a transition to V12?
Do you have tools to revert back to V11?

[Unreliably-Recurring]

I, too, am having this problem after upgrading in-place from the v7 with the recent security patch directly to 12.0.0.1420 p20230412.

The problem first manifested as a weekly backup-to-tape job repeatedly failing (along with its attempted re-starts) because it was taking so long that subsequent scheduled source jobs began interrupting it even though they hadn't conflicted for years prior.
So I found and checked the Prevent this job from being interrupted by source backup jobs checkbox to see how things would work, to see how things were affected by the next time it ran.

Now so many jobs -backup to tape, backup to VTL, syn full, and linked jobs- are all taking so long to run that I can't get logs that compare their pre-upgrade & post-upgrade performance.
The poor performance also, of course, very negatively affects RPO.
Once they're eventually able to finish (hopefully before this tape job's next scheduled increment next week) I'll finally be able to collect logs and create a Case.

[ATDCO]

Something is definitely amiss with this version, and I am hoping it is figured out soon.
Its almost like there is a default throttling setting even without the box checked to throttle throughput.
There really is no reason an upgrade should take processing speeds to 20% of what they were....

[Mildur]

Hi Unreliably-Recurring

Did you had a chance to open a support case?
We cannot assume what the issue is over the forum. Logs must be analyzed by our customer support.

Best,
Fabian

[Unreliably-Recurring]

Hello. Case #06054048.

[ATDCO]

Still awaiting any kind of answer and Veeam closes the cases without ANY resolution, hmmm? Why would they do that?

[Mildur]

Hi Richard

You are using the community edition. We provide support only on best effort for our free user and community edition users. Support cases from customers with active maintenance contracts are prioritized.
If our support team has free resources available, some of the free cases will also be processed.

For your case, no free resource was available, therefore the case was closed after 7 days. You can try to open a new case. With each case, you get a new chance that the case will be processed.
Or you can wait that a customer with an active maintenance contract gets a solution which may be included in a future patch if it was a bug.

You can find more information in our support policy:
https://www.veeam.com/support-policy.ht ... itions-sla

Best,
Fabian

[ATDCO]

Still I would like some answers as to what would cause that level of degradation in processing speeds...

[ATDCO]

Again, looking for answers, or when will a patch be available to fix the speed issues?

[ATDCO]

Anyone have any ideas or solutions to this? Or have similar results with V12 processing speeds?

[MarkBoothmaa]

If you do a windows file copy to the NAS and portable drives do you see the same loss of performance?

[Kwa-GJ]

Maybe same issue:

microsoft-hyper-v-f25/upgrading-v11-to- ... 85835.html

[ATDCO]

I guess I can only now wait for a patch as they do not help community users.

[ATDCO]

Here we are almost 2 months in and no update from Veeam on a release for fixing performance issues?

[Mildur]

We will release a new patch soon. You may test your backup server with the newest patch.
You can find all of our patches here:
https://www.veeam.com/kb4420

No promises that it solves the performance in your environment. Without a support case we cannot say what's working differently in your environment. We would have hundreds of support cases complaining about backup speed if there is something different with our v12 release.
And it would be mentioned in our Top Issue tracker:
veeam-backup-replication-f2/top-issues- ... 85262.html

Best,
Fabian

[ATDCO]

I am already running that version.
I'd guess its not a top issue tracker when the cases are closed with no resolution (At least for community edition).
I showed your team the difference in speeds prior and after the update.

I need to open another ticket possibly as now I'm getting windows agent backup errors.

[Mildur]

Hi Richard

The new patch was released yesterday.
You can find the update file in the provided kb article.

Best,
Fabian

[ATDCO]

I'm trying it, I wont know the results from the VM backups for a couple weeks however.

[ATDCO]

So far the VM copy jobs are running faster with the latest patch, but nothing like before the V12 update.
Before V12: about 50MB/s typically.
V12 Upgrade: 11MB/s
V12 20230718: 28-29MB/s

So clearly there was a fix for the speed issues, but we are still quote a bit slower on the read speeds...

Thoughts from the tech department?