-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Upgrading v11 to v12 Slowing D2D performance?
Currently working with support but wanted to see if anyone else had similar issue and have some insights. Case # 05945109
We recently upgraded our backup infrastructure from VBR v11 to v12. Since then, I noticed slow backup performance.
We run nightly reverse increment backups that usually finished in 6 hours, now finishing in almost 12 hours. Basically doubling the time it takes to backup.
I checked the backup summary and it shows. (v12)
Bottleneck: Source 0% | Proxy 2% | Network 0% | Target 96%
These numbers looked odd so I went back to my previous backup summaries when Veeam was on v11 and this is what I see.
Bottleneck: Source 78% | Proxy 76% | Network 67% | Target 93%
We have made no changes to our backup infrastructure that would impede on backup performance. Has anyone had similar issue I'm experiencing?
Thanks,
mp
We recently upgraded our backup infrastructure from VBR v11 to v12. Since then, I noticed slow backup performance.
We run nightly reverse increment backups that usually finished in 6 hours, now finishing in almost 12 hours. Basically doubling the time it takes to backup.
I checked the backup summary and it shows. (v12)
Bottleneck: Source 0% | Proxy 2% | Network 0% | Target 96%
These numbers looked odd so I went back to my previous backup summaries when Veeam was on v11 and this is what I see.
Bottleneck: Source 78% | Proxy 76% | Network 67% | Target 93%
We have made no changes to our backup infrastructure that would impede on backup performance. Has anyone had similar issue I'm experiencing?
Thanks,
mp
-
- Product Manager
- Posts: 10110
- Liked: 2696 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hi Min
Thanks for sharing your case number. I see that there is an active discussion on going.
I'll recommend to keep working with your assigned support engineer.
Best,
Fabian
Thanks for sharing your case number. I see that there is an active discussion on going.
I'll recommend to keep working with your assigned support engineer.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Service Provider
- Posts: 54
- Liked: 32 times
- Joined: Nov 23, 2018 12:23 am
- Full Name: Dion Norman
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Check if you were using either of the UseUnbufferedAccess and DisableHtAsyncIo registry values previously as they have now been replaced with the DataMoverLegacyIOMode value. (refer V12 Whats New pg6)
Even if you had not used those keys in V11, is still worth a shot testing V12 with DataMoverLegacyIOMode set to 1 and see if your storage performs any better. If no better (or worse) just remove the key to revert.
From what we saw in many of our VBR setups, the disk access 'improvments' they they made with V11 and further again with V12 just killed our disk storage performance. Setting UseUnbufferedAccess in V11 and DataMoverLegacyIOMode in V12 more than doubles performance in some of our particular instances (particularly on our Ubuntu XFS repos).
Even if you had not used those keys in V11, is still worth a shot testing V12 with DataMoverLegacyIOMode set to 1 and see if your storage performs any better. If no better (or worse) just remove the key to revert.
From what we saw in many of our VBR setups, the disk access 'improvments' they they made with V11 and further again with V12 just killed our disk storage performance. Setting UseUnbufferedAccess in V11 and DataMoverLegacyIOMode in V12 more than doubles performance in some of our particular instances (particularly on our Ubuntu XFS repos).
-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello Fabian,
Yes I'm still working with the support to no avail. I keep repeating the fact that performance was not an issue with v11 yet but keeps receiving generic instructions for troubleshooting that really doesn't help.
Hello Bytewiseits,
Thanks for the suggestion. We did not use those registries in the past since we had no issues before.
I tried the registry DataMoverLegacyIOMode (DWORD 1). Unfortunately I did not see any improvements. We are using CBT fastclone and what used to process around 40MB/s average is now down to ~10MB/s after the upgrade.
mp
Yes I'm still working with the support to no avail. I keep repeating the fact that performance was not an issue with v11 yet but keeps receiving generic instructions for troubleshooting that really doesn't help.
Hello Bytewiseits,
Thanks for the suggestion. We did not use those registries in the past since we had no issues before.
I tried the registry DataMoverLegacyIOMode (DWORD 1). Unfortunately I did not see any improvements. We are using CBT fastclone and what used to process around 40MB/s average is now down to ~10MB/s after the upgrade.
mp
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Mar 20, 2023 8:25 am
- Full Name: Naoufal AATALLA
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello,
i have de same problem case 05948611
Any solutions
Regards
i have de same problem case 05948611
Any solutions
Regards
-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello naatalla,
I believe I got my backup back to where it is (still need more time to test). I have made no other changes since my last post except marking the job "high priority" and ran Active full backup. Even when there are no jobs running, seems like jobs marked with high priority gets processed faster.
I'm going to start reverting changes, high priority and removing the registry key DataMoverLegacyIOMode one at a time to see if the job processing speed stays the same.
mp
I believe I got my backup back to where it is (still need more time to test). I have made no other changes since my last post except marking the job "high priority" and ran Active full backup. Even when there are no jobs running, seems like jobs marked with high priority gets processed faster.
I'm going to start reverting changes, high priority and removing the registry key DataMoverLegacyIOMode one at a time to see if the job processing speed stays the same.
mp
-
- Enthusiast
- Posts: 45
- Liked: 3 times
- Joined: Mar 09, 2011 8:22 am
- Full Name: John Brullemans
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hi Mpark,
I'm also working with veeam on the same issue. Case 05947572.
Before V11 i had a reverse incr. backup speed of 500 MB/s after V12 only 60 MB/s.
I have now set the regkey but i need to wait till tomorrow for the resuls.
What else did you did to get the performance back?
John
I'm also working with veeam on the same issue. Case 05947572.
Before V11 i had a reverse incr. backup speed of 500 MB/s after V12 only 60 MB/s.
I have now set the regkey but i need to wait till tomorrow for the resuls.
What else did you did to get the performance back?
John
-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello John,
I had done the following so far as a "fix" to the problem:
-Run Active full backup
-DataMoverLegacyIOMode (DWORD 1) registry key
-Mark the job as "high priority"
As a test, I removed DataMoverLegacyIOMode registry key since my last post. Backup speed was again impacted. I added the registry key back in and the speed came back. This reg key is definitely doing something to help backup go faster.
Next I'm going to remove the backup job from "high priority" to see if anything changes
mp
I had done the following so far as a "fix" to the problem:
-Run Active full backup
-DataMoverLegacyIOMode (DWORD 1) registry key
-Mark the job as "high priority"
As a test, I removed DataMoverLegacyIOMode registry key since my last post. Backup speed was again impacted. I added the registry key back in and the speed came back. This reg key is definitely doing something to help backup go faster.
Next I'm going to remove the backup job from "high priority" to see if anything changes
mp
-
- Chief Product Officer
- Posts: 31979
- Liked: 7441 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
It won't change anything, this setting only determines the placement of the job's machines in the overall processing queue when multiple jobs are running simultaneously.
-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Update:
Changing High Priority setting to the backup job indeed did nothing to the backup performance.
Only thing that would have helped with my case is adding DataMoverLegacyIOMode and/or running the active full backup of the job
mp
Changing High Priority setting to the backup job indeed did nothing to the backup performance.
Only thing that would have helped with my case is adding DataMoverLegacyIOMode and/or running the active full backup of the job
mp
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Mar 20, 2023 8:25 am
- Full Name: Naoufal AATALLA
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello,
Still no solution on my side, and the veeam support doesn't seem to be working either
the Full backup reaches 2Gbits/s but once it switches to reverse incremental I get 30Mbits/S
I think it has a link with the CBT and FAST CLONE, which is reset with the full backup and then reactivated for reverse incremental .....
Still no solution on my side, and the veeam support doesn't seem to be working either
the Full backup reaches 2Gbits/s but once it switches to reverse incremental I get 30Mbits/S
I think it has a link with the CBT and FAST CLONE, which is reset with the full backup and then reactivated for reverse incremental .....
-
- Enthusiast
- Posts: 45
- Liked: 3 times
- Joined: Mar 09, 2011 8:22 am
- Full Name: John Brullemans
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hi,
I have received an update from Veaam Escalation.
They have a hotfix for this it contains new Veeam Agent and my perfomance is back to normal again!
I would contact support if i was you.
I have received an update from Veaam Escalation.
They have a hotfix for this it contains new Veeam Agent and my perfomance is back to normal again!
I would contact support if i was you.
-
- Service Provider
- Posts: 29
- Liked: 3 times
- Joined: Dec 05, 2019 1:51 am
- Full Name: Daniel Judge
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
What did the hotfix address?
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 13, 2023 6:19 pm
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
@jbrullemans
What was the name of the hotfix? We're having exactly the same issue. We'd like to suggest it to our support rep.
What was the name of the hotfix? We're having exactly the same issue. We'd like to suggest it to our support rep.
-
- Product Manager
- Posts: 14973
- Liked: 3159 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
as far as I see, that was the reverse incremental block cloning issue tracked as bug #497629
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 13, 2023 6:19 pm
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Thank you HannesK - do you have a link to this info?
-
- Product Manager
- Posts: 14973
- Liked: 3159 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
I looked into the ticket 05947572. The support engineer can access that ticket.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 13, 2023 6:19 pm
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello all. Thank you Hannes - this information helped our support engineer provide us with the hotfix!
It does indeed seem to be related to reverse incremental jobs. Actual info on what the bug is may be forthcoming.
Hotfix given was named: TF493986
It does indeed seem to be related to reverse incremental jobs. Actual info on what the bug is may be forthcoming.
Hotfix given was named: TF493986
-
- Enthusiast
- Posts: 32
- Liked: 4 times
- Joined: Jan 20, 2016 4:43 am
- Full Name: Min Park
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
I have also applied the private fix along with the latest V12 update 12.0.0.1420 P20230412 and speed came back to normal.
For the record, removing DataMoverLegacyIOMode value had no impact on performance
Thanks all!
mp
For the record, removing DataMoverLegacyIOMode value had no impact on performance
Thanks all!
mp
-
- Novice
- Posts: 9
- Liked: 5 times
- Joined: Oct 02, 2013 9:47 am
- Full Name: KWA-GJ
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
We had exactly the same issue.
Openend a case and mentioned the case numbers and hotfix in this topic.
A few hours later we recieved the hotfix.
Openend a case and mentioned the case numbers and hotfix in this topic.
A few hours later we recieved the hotfix.
-
- Influencer
- Posts: 17
- Liked: never
- Joined: May 05, 2023 1:59 pm
- Full Name: Richard Bjornson
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
I don't think they take much care with community users, I opened a case and received no such hot fix.
I can only assume I will have to wait for an update to be release which addresses this particular performance issue.
I can only assume I will have to wait for an update to be release which addresses this particular performance issue.
-
- Chief Product Officer
- Posts: 31979
- Liked: 7441 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Community Edition does not support installation of hotfixes in principle, so there's no sense for you to obtain one.
Note: TF493986 hotfix is for reverse incremental backup mode not leveraging fast cloning on ReFS/XFS.
Normally you should not be using reverse incremental backup mode on ReFS/XFS (no benefits, only drawbacks).
Note: TF493986 hotfix is for reverse incremental backup mode not leveraging fast cloning on ReFS/XFS.
Normally you should not be using reverse incremental backup mode on ReFS/XFS (no benefits, only drawbacks).
-
- Novice
- Posts: 9
- Liked: 3 times
- Joined: Apr 17, 2023 6:25 am
- Full Name: Fred Lessing
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hi,
Any update on this? I see a few people saying private fixes. Perhaps I can reference your case numbers in my open case with Veeam Support?
Tried the regkey for v12, no change in performance.
Thank You
Any update on this? I see a few people saying private fixes. Perhaps I can reference your case numbers in my open case with Veeam Support?
Tried the regkey for v12, no change in performance.
Thank You
-
- Product Manager
- Posts: 14973
- Liked: 3159 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello,
and welcome to the forums.
Above you can find bug numbers and support case numbers. If you already have a support case open, then support should find out quickly, because the one above is "well known".
Best regards,
Hannes
and welcome to the forums.
Above you can find bug numbers and support case numbers. If you already have a support case open, then support should find out quickly, because the one above is "well known".
Best regards,
Hannes
-
- Novice
- Posts: 4
- Liked: never
- Joined: Apr 22, 2021 12:06 pm
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Hello all,
I experienced the same issue after upgrading to version 12. My backup performance plummeted. Especially the reverse incremental jobs. I did the normal steps of upgrading the backup chain format, adding/removing DataMoverLegacyIOMode, UseUnbufferedAccess, and DisableHtAsyncIo reg keys. Nothing has restored our pre v12 backup performance. I changed my reverse incremental jobs to forever forward for a very minor improvement. I've had my case (02319536 and 06086074) open for 4 months now and haven't seen any beneficial suggestions (very unimpressive response from Veeam support overall)
I do think doing an active backup or deleting the existing backups and starting fresh would probably help tremendously. Unfortunately that would require 60 TB of space I don't have.
I experienced the same issue after upgrading to version 12. My backup performance plummeted. Especially the reverse incremental jobs. I did the normal steps of upgrading the backup chain format, adding/removing DataMoverLegacyIOMode, UseUnbufferedAccess, and DisableHtAsyncIo reg keys. Nothing has restored our pre v12 backup performance. I changed my reverse incremental jobs to forever forward for a very minor improvement. I've had my case (02319536 and 06086074) open for 4 months now and haven't seen any beneficial suggestions (very unimpressive response from Veeam support overall)
I do think doing an active backup or deleting the existing backups and starting fresh would probably help tremendously. Unfortunately that would require 60 TB of space I don't have.
-
- Veteran
- Posts: 391
- Liked: 56 times
- Joined: Feb 03, 2017 2:34 pm
- Full Name: MikeO
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Whats the fix here ?
-
- Novice
- Posts: 9
- Liked: 3 times
- Joined: Apr 17, 2023 6:25 am
- Full Name: Fred Lessing
- Contact:
Re: Upgrading v11 to v12 Slowing D2D performance?
Issue was on Veeam Side. They hot fixed it in one of their patches.
Who is online
Users browsing this forum: No registered users and 65 guests