-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Hello,
we finally updated to V12 and everything seemed fine until our BCJs with up to 1760 VMs per job started. Copy jobs seem to be very slow, but not as slow as before the optimization we did with Veeam Dev in V11 (case 04822737). Instead of sending 5-10 GB/s to our remote site we only get 1 GB/s, nearly doing the transfer sequentially!
Since cpu of Veeam server is < 10 % and disk usage is also low we wanted to check Veeam process usage and found out that also Veeam.Backup.Manager.exe -SHOWPROXYUSAGES is no longer working...
Is there anything known already about these two issues? Will open case now..
Markus
we finally updated to V12 and everything seemed fine until our BCJs with up to 1760 VMs per job started. Copy jobs seem to be very slow, but not as slow as before the optimization we did with Veeam Dev in V11 (case 04822737). Instead of sending 5-10 GB/s to our remote site we only get 1 GB/s, nearly doing the transfer sequentially!
Since cpu of Veeam server is < 10 % and disk usage is also low we wanted to check Veeam process usage and found out that also Veeam.Backup.Manager.exe -SHOWPROXYUSAGES is no longer working...
Is there anything known already about these two issues? Will open case now..
Markus
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Log collection hangs at "creating folders" - it seems like there is a huge inefficiency in V12 which causes BCJ to stall and log collection to hang while this is happening... Opening SEV1 now
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Now all our copy jobs start to fail with "Cannot connect to target backup repository." and "Error: Item [XXX\XXX] is locked by running session Retention job [Retention job]" - Case 06233518
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
It gets worse: Synthetic Fulls now also take much longer (2-3 times normal). Our whole rather tight scheduling is falling apart thanks to V12... I hope support can find something.
-
- Enthusiast
- Posts: 42
- Liked: 6 times
- Joined: May 17, 2010 7:41 pm
- Full Name: Christian Moeller
- Location: Denmark
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
I've seem that same on an upgraded installation (Slow Copy job) this was due to the Proxys (maybe something in the upgrade process). We deployed new proxys and the problem was solved. We had this issue on 4 sites - all four was fixed by the proxy re-deployment. root cause was not found.
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
I am confused - what do Proxies have to do with copy jobs when not using WAN accelerators?
-
- Veteran
- Posts: 613
- Liked: 92 times
- Joined: Dec 20, 2015 6:24 pm
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Seeing your v12 upgrade issues here make me worry, tomorrow is v12 upgrade-day... Redeploying several dozen proxies would not be fun (even if I think its not the root cause in your case)
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Veeam support has verified that copy per VM takes much longer (minutes...), the same Devs that created the patch for V11 are on the case. I hope it will not take long...
-
- Service Provider
- Posts: 466
- Liked: 89 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Thanks for updating here! Will watch this
Veeam Certified Engineer
-
- Service Provider
- Posts: 466
- Liked: 89 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
any news on this one?
Veeam Certified Engineer
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Sadly, no. I tried to reach the engineer but was not able to reach him...
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Ok, we were told we need to upgrade our backup chains this should fix the issue. We did not do this before because we wanted to have the chance to downgrade...
Sadly, the first atempt failed (but still in process)....
20.08.2023 14:30:10 Failed Failed to upgrade backup XXXXX Error: The wait operation timed out
Sadly, the first atempt failed (but still in process)....
20.08.2023 14:30:10 Failed Failed to upgrade backup XXXXX Error: The wait operation timed out
-
- Service Provider
- Posts: 466
- Liked: 89 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
there should be log file for every attempt to upgrade the chain! perhaps more info to be found there. It should be on the vbr server within the C:\ProgramData\Veeam folder somewhere.
Veeam Certified Engineer
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
It is a "simple" SQL timeout. Jobs with 1100 VMs work, Jobs with 1700 do not. I asked Veeam support to give me a registry setting to increase the timeout, but for now i heard nothing but "in the mean time forget the chain and do new active full" which is unnacceptable for us as we also would have to do the same with our BCJ (and i don't even know if we have the space currently).
-
- Product Manager
- Posts: 15127
- Liked: 3232 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Hello Markus,
upgrading to per-machine backup with separate metadata files is recommended, agree. I get your point with "no way back", but on the other hand, everything was tested with the new per-machine chains for V12.
Best regards,
Hannes
agree. Active-full is an ugly hack and I will talk to support to see which other options we have (there should be hopefully an alternative)"in the mean time forget the chain and do new active full" which is unnacceptable for us
can I maybe get some more information on this? Is there no output at all (I see that for example when using CMD without "run as administrator") or is there something else that is wrong?Veeam.Backup.Manager.exe -SHOWPROXYUSAGES is no longer working...
upgrading to per-machine backup with separate metadata files is recommended, agree. I get your point with "no way back", but on the other hand, everything was tested with the new per-machine chains for V12.
not great, agree. I will also check on this.Jobs with 1100 VMs work, Jobs with 1700 do not.
Best regards,
Hannes
-
- Product Manager
- Posts: 14816
- Liked: 1771 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Hi Markus,
I've asked DEV and QA folks to look at the case details too. Stay tuned!
I've asked DEV and QA folks to look at the case details too. Stay tuned!
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Ok, there is hope! I just upgraded the primary job with 1755 VMs in little over an hour with SqlStatementTimeout=3600 Registry setting. Currently doing the same on the backup copy target.
If all goes well we will see in an hour if copy jobs work better.
For our 1100 VM job it already went much, much faster.
If all goes well we will see in an hour if copy jobs work better.
For our 1100 VM job it already went much, much faster.
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
It works! Currently copying with 8 GBit/s to the target 

-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Oh no - we have a new Problem. I tried to restart the tape backup which should write the freshly converted chain to tape. But now instead of one "Source backup files detected. VBK: XXX" it does this for EVERY VM - and takes nearly two minutes per VM!
We have 4600 VMs to be written to tape = 4600 * 1,5minutes = 6900 minutes / 60 = 115 hours = 4,7 days just for finding the backup points - the whole backup normally takes only 4-5 days including writing to tape!
We have 4600 VMs to be written to tape = 4600 * 1,5minutes = 6900 minutes / 60 = 115 hours = 4,7 days just for finding the backup points - the whole backup normally takes only 4-5 days including writing to tape!
-
- Service Provider
- Posts: 466
- Liked: 89 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
can't be right. hope to see support finding the cause.
Veeam Certified Engineer
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
How long does it take for you to add a restore point to the tape backup?
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
We found a "workaround" - manually starting the tape backup with "active full" seems to "solve" this issue and tape job starts writing immediately. I hope there is a workaround for scheduled start...
-
- Product Manager
- Posts: 14816
- Liked: 1771 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Markus, I've shared the issue with the tape team. Mind me asking if you have updated the fresh portion of logs to the existing case or you have a new support case for that? Thanks!
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Currently working on the existing case. I was told "If this requires a separate case, I will open one on your behalf".
There are logs for the extremly slow tape restore point scan, but not for the working "workaround" (i do not know if this is necessary).
There are logs for the extremly slow tape restore point scan, but not for the working "workaround" (i do not know if this is necessary).
-
- Product Manager
- Posts: 14816
- Liked: 1771 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Correct, I've shared the problem related to tape task build you've mentioned with the RnD folks and they've requested more detailed logs because original case did not have anything for tape job analysis. I believe right now its enough, if something else is required for RnD research support folks will reach you out. Thank you for the logs!I was told "If this requires a separate case, I will open one on your behalf". There are logs for the extremly slow tape restore point scan, but not for the working "workaround" (i do not know if this is necessary).
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
They have all the logs they need. Just got the info "The problematic piece of logic has been isolated" - lets hope a fix is developed fast...
-
- Veteran
- Posts: 1253
- Liked: 443 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
"The solution has already been implemented in v12a, and now it is being backported to v12." 

-
- Product Manager
- Posts: 14816
- Liked: 1771 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
I would not say 'backported' - guys are creating a private fix based on improvements and fixes they've made in 12a, so for any newcomers with similar symptoms it will be required to raise a support case to obtain the fix. Thank you for the update Markus!
-
- Influencer
- Posts: 11
- Liked: 2 times
- Joined: May 01, 2019 2:10 pm
- Full Name: Duco van der Nol
- Contact:
Re: V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES
Same issue here, we migrated to v12 on August 1. Since then copy jobs are much slower than in v11, now often exceeding our 24-hour copy window.
Anyone experience with a workaround? Is the SqlStatementTimeout "fix" worth a try? And where do you set the registry key?
Support case created, ticket 06276480
Anyone experience with a workaround? Is the SqlStatementTimeout "fix" worth a try? And where do you set the registry key?
Support case created, ticket 06276480
Who is online
Users browsing this forum: Google [Bot], Semrush [Bot] and 156 guests