V12: Slow backup copy jobs and no Veeam.Backup.Manager.exe -SHOWPROXYUSAGES

[mkretzer]

Hello,

we finally updated to V12 and everything seemed fine until our BCJs with up to 1760 VMs per job started. Copy jobs seem to be very slow, but not as slow as before the optimization we did with Veeam Dev in V11 (case 04822737). Instead of sending 5-10 GB/s to our remote site we only get 1 GB/s, nearly doing the transfer sequentially!

Since cpu of Veeam server is < 10 % and disk usage is also low we wanted to check Veeam process usage and found out that also Veeam.Backup.Manager.exe -SHOWPROXYUSAGES is no longer working...

Is there anything known already about these two issues? Will open case now..

Markus

[mkretzer]

Log collection hangs at "creating folders" - it seems like there is a huge inefficiency in V12 which causes BCJ to stall and log collection to hang while this is happening... Opening SEV1 now

[mkretzer]

Case 06232794

[mkretzer]

Now all our copy jobs start to fail with "Cannot connect to target backup repository." and "Error: Item [XXX\XXX] is locked by running session Retention job [Retention job]" - Case 06233518

[mkretzer]

It gets worse: Synthetic Fulls now also take much longer (2-3 times normal). Our whole rather tight scheduling is falling apart thanks to V12... I hope support can find something.

[chrmol]

I've seem that same on an upgraded installation (Slow Copy job) this was due to the Proxys (maybe something in the upgrade process). We deployed new proxys and the problem was solved. We had this issue on 4 sites - all four was fixed by the proxy re-deployment. root cause was not found.

[mkretzer]

I am confused - what do Proxies have to do with copy jobs when not using WAN accelerators?

[pirx]

Seeing your v12 upgrade issues here make me worry, tomorrow is v12 upgrade-day... Redeploying several dozen proxies would not be fun (even if I think its not the root cause in your case)

[mkretzer]

Veeam support has verified that copy per VM takes much longer (minutes...), the same Devs that created the patch for V11 are on the case. I hope it will not take long...

[JaySt]

Thanks for updating here! Will watch this

[JaySt]

any news on this one?

[mkretzer]

Sadly, no. I tried to reach the engineer but was not able to reach him...

[mkretzer]

Ok, we were told we need to upgrade our backup chains this should fix the issue. We did not do this before because we wanted to have the chance to downgrade...

Sadly, the first atempt failed (but still in process)....
20.08.2023 14:30:10 Failed Failed to upgrade backup XXXXX Error: The wait operation timed out

[JaySt]

there should be log file for every attempt to upgrade the chain! perhaps more info to be found there. It should be on the vbr server within the C:\ProgramData\Veeam folder somewhere.

[mkretzer]

It is a "simple" SQL timeout. Jobs with 1100 VMs work, Jobs with 1700 do not. I asked Veeam support to give me a registry setting to increase the timeout, but for now i heard nothing but "in the mean time forget the chain and do new active full" which is unnacceptable for us as we also would have to do the same with our BCJ (and i don't even know if we have the space currently).

[HannesK]

Hello Markus,

"in the mean time forget the chain and do new active full" which is unnacceptable for us

agree. Active-full is an ugly hack and I will talk to support to see which other options we have (there should be hopefully an alternative)

Veeam.Backup.Manager.exe -SHOWPROXYUSAGES is no longer working...

can I maybe get some more information on this? Is there no output at all (I see that for example when using CMD without "run as administrator") or is there something else that is wrong?

upgrading to per-machine backup with separate metadata files is recommended, agree. I get your point with "no way back", but on the other hand, everything was tested with the new per-machine chains for V12.

Jobs with 1100 VMs work, Jobs with 1700 do not.

not great, agree. I will also check on this.

Best regards,
Hannes

[Dima P.]

Hi Markus,

I've asked DEV and QA folks to look at the case details too. Stay tuned!

[mkretzer]

Ok, there is hope! I just upgraded the primary job with 1755 VMs in little over an hour with SqlStatementTimeout=3600 Registry setting. Currently doing the same on the backup copy target.
If all goes well we will see in an hour if copy jobs work better.

For our 1100 VM job it already went much, much faster.

[mkretzer]

It works! Currently copying with 8 GBit/s to the target

[mkretzer]

Oh no - we have a new Problem. I tried to restart the tape backup which should write the freshly converted chain to tape. But now instead of one "Source backup files detected. VBK: XXX" it does this for EVERY VM - and takes nearly two minutes per VM!
We have 4600 VMs to be written to tape = 4600 * 1,5minutes = 6900 minutes / 60 = 115 hours = 4,7 days just for finding the backup points - the whole backup normally takes only 4-5 days including writing to tape!

[JaySt]

can't be right. hope to see support finding the cause.

[mkretzer]

How long does it take for you to add a restore point to the tape backup?

[mkretzer]

We found a "workaround" - manually starting the tape backup with "active full" seems to "solve" this issue and tape job starts writing immediately. I hope there is a workaround for scheduled start...

[Dima P.]

Markus, I've shared the issue with the tape team. Mind me asking if you have updated the fresh portion of logs to the existing case or you have a new support case for that? Thanks!

[mkretzer]

Currently working on the existing case. I was told "If this requires a separate case, I will open one on your behalf".
There are logs for the extremly slow tape restore point scan, but not for the working "workaround" (i do not know if this is necessary).

[Dima P.]

I was told "If this requires a separate case, I will open one on your behalf". There are logs for the extremly slow tape restore point scan, but not for the working "workaround" (i do not know if this is necessary).

Correct, I've shared the problem related to tape task build you've mentioned with the RnD folks and they've requested more detailed logs because original case did not have anything for tape job analysis. I believe right now its enough, if something else is required for RnD research support folks will reach you out. Thank you for the logs!

[mkretzer]

They have all the logs they need. Just got the info "The problematic piece of logic has been isolated" - lets hope a fix is developed fast...

[mkretzer]

"The solution has already been implemented in v12a, and now it is being backported to v12."

[Dima P.]

I would not say 'backported' - guys are creating a private fix based on improvements and fixes they've made in 12a, so for any newcomers with similar symptoms it will be required to raise a support case to obtain the fix. Thank you for the update Markus!

[dvdn61]

Same issue here, we migrated to v12 on August 1. Since then copy jobs are much slower than in v11, now often exceeding our 24-hour copy window.
Anyone experience with a workaround? Is the SqlStatementTimeout "fix" worth a try? And where do you set the registry key?
Support case created, ticket 06276480