"Upgrading backup chain index" running long

[mfc]

After upgrading to v12, the next SOBR offload jobs have been running extraordinarily long. They all seem to be at the "Upgrading backup chain index" step, but have been there for 83+ hours at this point. Our SOBR repositories have a local NAS performance tier and S3 (Wasabi) capacity tier.

I have not yet chosen "Upgrade backup chain format" on any of the backup sets as of yet.

Anyone else running into this? I assume it's normal and expected but the long running time has me concerned.

[Mildur]

Hi Mike

Which build number of V12 do you have installed? How many machines do you have in your SOBR?
We had an issue with the metadata upgrade which we solved with the latest update.

#3: Backup chain metadata upgrade takes extremely long time.
Scope: Local backup chain with immutable copies in Capacity Tier.
Symptoms: Upgrading metadata takes about 20 mins per machine.
Cause: Bug + general lack of optimizations for this upgrade scenario.
Status: Fixed in P20230412 or later.

Best,
Fabian

[mfc]

Hmm that does sound like exactly my issue, and I am using immutability on the capacity tier. I am on build P20230412 however.

[Xqualo33]

I upgraded yesterday, today all SOBR tiering its taking forever because of upgrading backup chain index... and removing checkpoint from capacity tier...
I'm at P20230412

[Mildur]

Hello

You are both on the latest version, then I suggest to open a support case and let our customer support inspect the logs. Please don't forget to provide me with the case number here in this topic.

Best,
Fabian

[kaffeine]

I upgraded yesterday, today all SOBR tiering its taking forever because of upgrading backup chain index... and removing checkpoint from capacity tier...
I'm at P20230412

Same here. All our SOBR offloading jobs were working 100% on V11, and right after the update to V12 the first SOBR tiering job (S3 offload) got stuck while "Upgrading backup chain index". It's stuck for a few hours now and all the subsequent SOBR offloading jobs are pilling up and getting jammed on the same issue, it doesn't look good.



We're running P20230412 and offloading all SOBR to Wasabi with immutability.

Considering the multiple similar reports here, I suppose it's not a error on our side.

[mfc]

Case opened and B&R server logs uploaded. Case #06096330

[Mildur]

Hi Mike

Thanks for the case number.

@kaffeine
Do you have opened a case with our customer support? If so, please provide me with the case number.

Best,
Fabian

[biohazrd]

Similar issues. On the latest patch but still seeing backup format upgrades taking roughly 20 minutes per VM.

Case 06101080

[kaffeine]

@kaffeine
Do you have opened a case with our customer support? If so, please provide me with the case number.

Unfortunately not. Due to time constrains (we're in the middle of a big roll-out - not Veeam related) we just orphaned the old wasabi buckets (which were created with V11) and created new ones for V12. As our buckets only had a few TBs, it was manageable, but those with astronomical buckets won't probably get away it it so easily =\

Btw as a sidenote (and in case it's relevant for further troubleshooting), we were offloading a mix of vanilla windows servers and some sql servers (with T-LOGs beeing backup up from B&R and offloaded as well).

Regards

[femi]

I have just completed my upgrade to V12 - P20230412 today and upgrading backup chain index... in progress for the past hour. Did resolving this require support or a patch?

[femi]

upgrading backup chain index... completed in 2 hours and my Offloads are completing successfully. Did nothing just gave it time.