Issue with one slow backup job 'finalizing' long time on every VM causing knock-on issues following upgrade 11 -> 12.2

[IanBolton]

Hi

Let me try and explain the scenario.

We have several jobs which go to a primary SOBR. Each is copied by it's own BCJ to DR SOBR
Separately, each backup job ends up at one of two tape job which is retained for 1 or 3 years.

I have reported a case to support ( 07451965 ) earlier in the week as I noticed that one of the BCJs was very slow, causing other BCJs to slow down. Initial suggestions from support were to increase the task limit for each repo in the target SOBR. This is despite the bottleneck being reported in the GUI and log bundle being 'Source'

My own further investigations then found that the source backup for the BCJ was running significantly slower after the upgrade to 12.2 Support next suggestion is to increase the core count on proxies.

Further investigations led to me to discover that the source backup job shows the 'Finalizing' stage being significantly slower post-upgrade. For three VMs in the job, the 'Finalizing' stage went from 8s, 11s, 21s pre-upgrade to 38minutes(!) post-upgrade.

Support have said they're methodically working through troubleshooting the reported issue - which I can sort of understand, but to my mind the fact that fails to take account of new info, or look at the entire scenario when new information has come to light.

Given the approaching weekend and extended backup window that affords, I am tempted to run an active full of the source to see if that helps. I'm not sure though if that will have the desired impact on the BCJ. It feels like something more drastic is required, like abandon that job, clone the source job to a new job, along with a new BCJ, to start afresh with new DB entries, backup chain, metadata.

To confirm and clarify - NO OTHER CHANGES were made to vmware environments, VBR hardware, repos, proxies etc except the upgrade to 12.2.

[david.domask]

Hi Ian,

Thank you for sharing the case number and the details of the issue. The behavior is not expected, and off the top of my head no known issues match the behavior.

I've asked Support Management to review the case; the behavior is not expected and sounds like it should be reviewed by the Advanced Support team at Veeam Support.

I'm not confident an Active Full will fully correct the issue, though it may help temporarily, but it's best to let Support get to the root cause of the behavior.

Please stand by on the case while Support Management reviews the case and aligns further resources.

Thanks!

[IanBolton]

Thanks for your reply.
I heard nothing back from my support chap all working day Friday, and in the absence of further guidance, I proceeded with the active full, which I don't believe has made any improvement.

I heard back from the chap this morning pretty much echoing the same as you, and insisting that the previously suggested changes (additional/improved spec proxies) are tested. That feels a] futile and b] kinda missing the point.

[david.domask]

You're welcome Ian, and looks like the case is being prepared for escalation; your continued patience is appreciated, I think let's wait and see the results of the investigation by the more senior Support Engineers.

[IanBolton]

Just to update on this - it seems that the issue was that the backup chain format needed to be upgraded after the upgrade to 12.2
In the 'legacy' format all of the metadata for all VMs in the job are in one file. Once the chain is upgraded, the metadata is stored per-VM. Maybe the fact that the problem job had 160+ VMs was unhelpful...!

Anyway, backup chains upgraded on the backup job and the BCJ and all is back to how it was. Only slight hiccup seems to be an extra set of fulls being written to the daily media set of the gfs tape pool, but I'd always rather have an extra backup than one missing

Top marks to the engineer who dealt with the case on escalation, and thanks @david.domask for your help.

[david.domask]

Hi Ian,

Thank you for the update and for sharing the solution! It's great to hear Support was able to figure it out; had not considered the chain was not upgraded to the new format, but this would make a lot of sense.

Yes, regrettably the upgrade of the chains will result in an extra backup going to Tape due to the changes on the chain, but glad it's not a major issue.

Thank you for the kind words for the Support Engineer, I'll make sure that it gets passed along, and glad I could assist a bit.