Repository upgrade times after upgrading to V8

[dotdk]

Hi forum

Did anyone try updating larger environments with object storage repositories that are 50 TB+? How long did that take, and how large were the repositories?

I know from the documentation it can take up to 15 mins pr TB, but in our smaller environments we never saw it take anything close to that, but now we started on our larger environments, and the upgrade seems to be stuck at 99%, at least it has been for the last 6 hours.

Raised a ticket #07517078 to see if there are any logs where we can track if anything is still happening in the background. Proxy logs show no copy actions the last 6 hours:

Code: Select all

[25.11.2024 12:15:08.773]     58 (1)  Rows copied: 8385 (from: Jet-3328401eba1e42b6aabf1b56b32af25c_WebsDataIndex, to: backup.blob_versions)
[25.11.2024 12:15:08.904]     96 (1)  Copying rows: 119917 (from table: 3328401eba1e42b6aabf1b56b32af25c_FilesDataIndex, to table: backup.blob_versions)...
[25.11.2024 12:15:18.860]     92 (1)  Rows copied: 119917 (from: Jet-3328401eba1e42b6aabf1b56b32af25c_FilesDataIndex, to: backup.blob_versions)
[25.11.2024 12:15:18.887]     80 (1)  Copying rows: 2708 (from table: FilesIndexRoots, to table: backup.blob_versions)...
[25.11.2024 12:15:19.103]     80 (1)  Rows copied: 2708 (from: Jet-FilesIndexRoots, to: backup.blob_versions)
[25.11.2024 12:15:19.194]     80 (1)  Copying rows: 5392 (from table: 3328401eba1e42b6aabf1b56b32af25c_WebPartsDataIndex, to table: backup.blob_versions)...
[25.11.2024 12:15:19.611]     80 (1)  Rows copied: 5392 (from: Jet-3328401eba1e42b6aabf1b56b32af25c_WebPartsDataIndex, to: backup.blob_versions)
[25.11.2024 12:15:19.632]     80 (1) Finished action: BlobVersionsIndexesJetPostgresUpgradeAction after: 00:07:44.6885051...
[25.11.2024 12:15:19.633]     74 (1)   Updating database version to: 815...
[25.11.2024 12:15:19.639]     74 (1) Repository database upgraded (repository ID: 4aa2b1f7-5734-42ef-8618-8864ee6e528e)
[25.11.2024 12:15:27.628]    114 (1) Workload: Processing: 0 (resolve: 0, CPU: 30,5, memory: 43,6, available memory MB: 9237,7, connections: 351)...
[25.11.2024 12:15:32.745]    116 (1) 

[dotdk]

Apparently I just need a bit more patience, since everything completed after 8 hours at 99%. Maybe this post can be a guideline for others.

A object storage repository of 53 TB took roughly 8 hours to update and index.

[Polina]

Thanks for sharing it, Thomas!

This post is useful for Veeam's RnD as well - we are always willing to know real-life cases from the field.

[DaStivi]

hey there.... there where some other posts about postgreSQL Tuning, might this help in lowering such numbers?

i do have 2x200TB S3, 1x64TB and some smaller ones... 30,15,2x10... i'm still on v7 and very hesitating to upgrade to v8 especially as my Colleges from Germany had massive issue while upgrading, i just contacted him if they even have finished upgrading the repos yet ...

[Polina]

Hi Stephan,

The latest patch 8.0.5.20 includes fixes for all the most common repository upgrade issues we saw after the v8 GA. Repository indexing takes time (and that's expected) but it doesn't impact your daily backups - the repository remains fully functional.

And a few things to consider that can help you to avoid potential issues with PostgreSQL:
- use a dedicated server
- use SSD with 5-10K IOPS (applicable to installation in Azure/AWS as well)
- if you run it on Windows, follow the AV recommendations