Comprehensive data protection for all workloads
Post Reply
pirx
Veteran
Posts: 598
Liked: 87 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Backup chain format upgrade... extremely slow

Post by pirx »

Case 06217042

This is more a heads up for others...

Finally we have upgraded to v12, now I started the first backup chain upgrade. First small backup job yesterday with 5 VMs (850 GB) took 40min. Today I started another upgrade of a small job with 19 VMs (8 TB) and it is now running for 1:30h and has just processed 2 VMs. It seems to take a long time for the capacity tier. Currently no offloads are running. Usually I see good performance during offloads, peak throughput of ~500 MByte/s.

Code: Select all

18.08.2023 09:22:00 In progress [xxxx] Upgrading backup chain parts located on Capacity Tier and Archive Tier extents... 1:12:48
We have other jobs with 80-120 VMs and 70 TB. And copy chains of each job has to be upgraded too. Currently this looks like a cleanup issue in capacity tier and we need to tune some parameters.
Mildur
Product Manager
Posts: 9580
Liked: 2539 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Backup chain format upgrade... extremely slow

Post by Mildur »

Hello Pirx

Thanks for the case number.
Which build did you have installed? In the case I can see you have provided build: 11.0.1.1261 P20230227

We had a known issue about long chain upgrades where a capacity tier is involved.
But it is fixed in P20230412 or later ([V12] Top issues tracker - Issue 3 )

Best,
Fabian
Product Management Analyst @ Veeam Software
pirx
Veteran
Posts: 598
Liked: 87 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Backup chain format upgrade... extremely slow

Post by pirx » 1 person likes this post

Since Wednesday we are on latest v12 release. It seems that removing some offload limits from registry and Linux agent settings solved this issue. We had to slow down offloads because we got AWS S3 "slow down" messages and I was unable to delete some orphaned backups in capacity tier. Looks like I have to create a little script to change regkeys and agent settings depending on the use case :)
mkretzer
Veeam Legend
Posts: 1192
Liked: 412 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Backup chain format upgrade... extremely slow

Post by mkretzer »

For us, V12 chain upgrade takes little over one minute. Thats very bad as we have ~1500 objects per backup copy job. So if we do this we have no copies for over a day.

Is there any way to speed this up?
pirx
Veteran
Posts: 598
Liked: 87 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Backup chain format upgrade... extremely slow

Post by pirx »

I didn't have problems with the upgrade time anymore, but scheduling is broken afterwards.
As per known issue: If chain upgrade was terminated incorrectly, it may leave a partially upgraded backup set in the database table [backup.model.backups].
So double check if your jobs really start as expected if you not have any other checks in place. Not all schedules were broken, but some jobs just didn't start. Nice, 3 known issues one week after update ;) And this is already 3. patch for v12.
mkretzer
Veeam Legend
Posts: 1192
Liked: 412 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Backup chain format upgrade... extremely slow

Post by mkretzer »

Interestingly, it works now much faster, but we had to increase SQL timeout significantly to get it to work correctly at all for some backups...
V12 is really not much fun :-(
Dima P.
Product Manager
Posts: 14619
Liked: 1664 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Backup chain format upgrade... extremely slow

Post by Dima P. »

Hi folks,
I didn't have problems with the upgrade time anymore, but scheduling is broken afterwards.
We've checked with support folks and investigation is in progress. Please stay tuned for updates from support team!
Interestingly, it works now much faster, but we had to increase SQL timeout significantly to get it to work correctly at all for some backups... V12 is really not much fun
Sorry to hear that you've faced problems after the upgrade Markus! The issues you've faced have different root cause are now investigated by support team and RnD folks. Lets await for the investigation results!
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], MarvinMichalski, renatorichina and 102 guests