Errors on upgrading Repo's V8.0.2.159

[kristofpoppe]

After a succesfull upgrade of the latest v7 to V8, we started upgrading repo's. (Object Storage)

Multiple repo's fail with the following error:

23505: duplicate key value violates unique constraint "pk_exchange_posts_batch_index"
Detail: Detail redacted as it may contain sensistive data. Specify "include error detail' in the connection string to include this information.

Another repo upgrade fails with:

23502: null value in column 'organization_id" of relation "syncrhonization_infos' violates not-null constraint


https://imgur.com/eZoOnCZ

Anybody else facing these errors ?

Opened as Case #07403229

#MOD: Fixed with version 8.0.2.200 or later --> https://www.veeam.com/kb4656

[kristofpoppe]

First feedback on my ticket:

This is a known error on our side, the error appears at some point while attempting to collect the data from the persistent cache database to a centralized Postgres db.
Line 11011: [04.09.2024 09:25:12.435] 199 (1) Repository database upgrade failed (repository ID: 45450150-adc0-426c-90e9-7e3b7422c0fb)
Line 11012: [04.09.2024 09:25:12.449] 199 (1) Error: 23505: duplicate key value violates unique constraint "pk_exchange_posts_batch_index"

For the root cause of this matter, RnD team was already involved to investigate further. I'll keep you posted with the progress of this investigation.

[Mildur]

Thank you for the update about your Case, Kristof.
I can see the issue is already with our RnD team.

Best,
Fabian

[kristofpoppe]

The initial workaround was removing and re-adding the repo.

We've tested this on a small test-repo, after synchronisation when re-added we didn't had the possibility to select earlier restore points.
So it seems that this isn't really saving the history present in the repo.

[Mildur]

Have you synchronized the cache?

https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80

Best,
Fabian

[Seve CH]

The initial workaround was removing and re-adding the repo.

We've tested this on a small test-repo, after synchronisation when re-added we didn't had the possibility to select earlier restore points.
So it seems that this isn't really saving the history present in the repo.

Same experience here. I had 23505 error on unique constraint pk_teams. The error message is weird because it is mixed between German and English, but that's life in Switzerland .
23505: doppelter Schlüsselwert verletzt Unique-Constraint pk_teams
DETAIL: Detail redacted as it may contain sensitive data. Specify 'Include Error Detail' in the connection string to include this information.

It was a test installation (PoC) with 800 users and 10 TB or so of data on different jobs. During my tests I moved my Teams jobs' data from DB repos (dedicated to that jobs) to Object storage (I've used the PowerShell script provided by Veeam), I supposed something went wrong and the object storage had consistency problems, but seen this thread, I would say that there is a problem with the repo upgrade process.

This is what I did:

• Migrate DB repo to S3 storage.

• Test: Restore points were available.

• Update from v.7 to v.8

• Upgrade object storage (S3) repos -> pk_teams error

• Delete all copy jobs and repos used by copy jobs. Still pk_teams error

• Reimport the backup repo: Delete the teams backup job, delete the repo, re-add the repo, resync, reindex

The restore points were unavailable. I did a teams backup targeting the repo for a specific team. That team was available, any other team was still missing. I had to run a full backup again and only that restore point is now available to choose.

I didn't check if I have double object storage consumed (it is on-prem in testing, I don't have statistics yet) or if the deleted team data was missing (data that was not available on the new full backup, but it was backed up before).

As it is the PoC installation with a lot of experimenting on it, I didn't bother to open a support ticket. Next week we install the real thing. We hate being early adopters, but I prefer not being impacted with this upgrade problem so we are going directly to v.8.

Best regards

[kristofpoppe]

We had more or less the same experience with removing and re-adding the repo.

Point your backup jobs to a temporary storage and disable the jobs.
Remove the Repo and re-add the repo.
IN our case it came as 'upgradable' and we could manual upgrade the repo. (Upgrading, Indexing, Syncing)

Altough, when we point the repo to the job, no previous restore points were available.

We again removed the repo and added it again. Then the sync took much longer and when added to the jobs, the restore points from the past were visisble.

[Seve CH]

Hi Kristof
I am trying to do what you did:
Job disabled and pointed elsewhere.
Repo removed.
Repo added.
Now it's syncing the repo (the process is stuck on 61% since 20-30 minutes). The bucket is 1.5TiB big and has 232.249 objects. Other unrelated jobs are also running, so I see a lot of activity but I'm not sure if it is related to this specifically.
I will post if it worked or not.
Best regards
Seve

[Seve CH]

Well, it didn't work.
I can see:
The full backup I did yesterday (job with all teams + SPO sites, the one I moved from DBs to Object storage).
Older restore points for a different test job (Object-native) targeting 2 teams only. I cannot remember how that one was set because it doesn't exist anymore (as I said, this is a test/sandbox installation). I think that was the state before I reimported the repo today.

To the Veeam team: I suspect you might be migrating the metadata of one job only and the primary key violation might be an object being processed by several jobs.

I must remove all test data, so I won't be able to do tests with this anymore. Good luck hunting the bug

[kristofpoppe]

Hi Seve,

The 61% was also generating high stress levels at my side. For some repo's it took more than 4h to finalize.
Today I was able to validate again the workout. Remove, Add, remove again and Add again.
When the repo is added for the second time the sync takes his time. In the first add, on our side it was less than 5 mins.

During all these operations, I've disabled my jobs and pointed them to another repo.

[Polina]

Hi All,

RnD is already researching the issue. The case opened by Kristof was escalated to RnD.
I'll update the thread once there's some news.

[TecIT-Philipp]

Good Morning together,

are there some news? We've got the same on one REPO :/

Regards

[Polina]

Hi Philipp,

Since midnight, RnD has identified the problem, and they continue researching the root cause and ways to resolve it.

Thanks!

[TecIT-Philipp]

thx for response!

[edh]

Hi,

YEsterday night we perform and upgrade from Office 365 v7.1 to V8.

We got a problem with the repository sync to PosteSQl with and know error (Case #07406941 )

" Error: 23505: Duplicate key violates uniqueness constraint "pk_account_personal_sites""

The work arround suggested was:

Workaround:
You can delete the repository and re-add again -> we will not migrate the persistent cache from Jet to Postgres
Drawbacks of the workaround - full sync on the backup jobs that use the affected repository and sequentially increased Teams API costs

• Disable all the jobs and pointed them to a temp repository
• Remove the affected repository and add it again
• Edit the job to use the same repo again and enable the job
• It should start syncing the repository, once this is done please run the job again and let me know if this helps you fix the issue

Please tell us that chain are not lost due upgrade from Jet-DB to PostgreSQL

Regards

[Mildur]

Hi Manuel

I moved your topic to Kristofs topic. Because you are facing the same issue as discussed in this topic.
Please see Polinas answer. Our RnD team is working on a fix for the update issues.

For your question about the missing restore points, please check directly with your support engineer. I assume the cache re-sync process is still running. Kristof had a similar situation but in the end all previous restore points were available in the end.

Best,
Fabian

[ptwisterling]

I am facing the same issues with pk_exchange_posts_batch_index.

After removing and adding back a repository i now get another error:
21000: ON CONFLICT DO UPDATE command cannot affect row a second time

I will file a support request

[Polina]

Hi Patrick,

Yes, please open a support case.

Some updates from the RnD side. The pk_exchange_posts_batch_index is caused by conflicts in Teams channel messages backup and impacts repositories that contain Teams restore points created in v6. The team is working on the fix as we speak and I hope to get it early next week.

Thanks!

[edh]

Whats the recomendation from Veeam perspective, we continue doing jobs or disable all jobs?

Looks like jobs are incremental...

In our case the repo got 100TB data and 24.8 Millions Objects...

Regards

[edh]

Hi,

We recovered all our customers backups old restore points before upgrade.

What we done:

- Tunning PostgresSQL main configuration is not good for setup, we upgraded a lot of setting , sometimes we hitted 30K -100kTPS.
- Created a pool of 2 servers with total 36 cores and 128 gb ram. This server are just VBO Proxy , not other role.
- Optimize proxy.xml in VBO Proxys for allow more current task because psql get exausted so fast.
- Create a Temp Object storage and move all task to this temp object storage.
- Delete the Production Object Storage and re-join production Object selecting as Proxy the new Pool of Servers.
- Move all task back to the new production object that got the new pool of servers as proxys.
- Do a re-index of the object storage. In our case 100TB took a whole day.

As note, Index status task are show in Veeam.Archiver.Proxy_*** with lines like:

Mailbox backup dates were updated (mailbox ID: 56d98941-2f7d-420f-874f-59316413e193, last backup attempt: 5/24/2024 10:20:04 PM, last archive: 5/24/2024 10:20:04 PM)

One of the most important thing we found in this issue is the Postgress generate by default a lot of timeouts and pool exausted.
Right now we're able to do restores from Jobs and from Organization without lost any restore points.

[Mike Resseler]

Just as an update, our teams are still working on this. So everyone who is facing this, please continue to open support cases. As of the testing and scenario's we have done, the reindexing and getting the data to PostgreSQL will take some time, but you should be able to continue to do backups and restores in the meantime (albeit a bit slower if I am not mistaken).

[ND40oz]

One of the most important thing we found in this issue is the Postgress generate by default a lot of timeouts and pool exausted.

I have a ticket open for this as well. Originally I was having an issue with on of my proxy servers staying connected but once it finally made it through a backup of all the jobs it was used for it seems to have cleared itself up. But the PostgreSQL timeout errors are still there and take multiple retries of the job to complete.

[Mike Resseler]

Update: I do expect for most of these issues a fix by tomorrow (if all regression testing on them succeeds). So I hope that this is the case and support can contact you with them

[edh]

I have a ticket open for this as well. Originally I was having an issue with on of my proxy servers staying connected but once it finally made it through a backup of all the jobs it was used for it seems to have cleared itself up. But the PostgreSQL timeout errors are still there and take multiple retries of the job to complete.

We changed in proxy.xml this values and added last 2, last one helped a lot.
maximum pool size=100;connection idle lifetime=5;connection pruning interval=5;CommandTimeout=120

At prostgres conf we used an external pgoptimizer https://pgtune.leopard.in.ua/ that helped us to improve the PLSQL.

In the config.xml at ControllerServer we changed the <ControllerPostgres> to MaxPoolSize=500;ConnectionIdleLifetime=5;ConnectionPruningInterval=5"

With this setup at least now we dont see errors of nplsql library at our logs just some EWS Throtthle again...

Please take these recommendations with a grain of salt, each setup is different and we normally got huge servers in term of RAM and CPU.

[pat_ren]

just updating this post, have this same error of Error: 23505: Duplicate key violates uniqueness constraint "pk_account_personal_sites on nine different repos after upgrading.
so far am about 4 hours into the upgrade process and still have around 500 repos unindexed, not going as quick as I expected after reading the pre-update KB.
case has been logged for various errors although most have been resolved through a reboot and retry post upgrade (except the one mentioned above)
case # 07414697

a suggestion for anyone reading this to hold off a bit longer on updating if you can avoid it.

[dotdk]

Same issue for me with:

"23502: null value in column 'organization_id" of relation "synchronization_infos' violates not-null constraint"

Case - 07415267

[dotdk]

Our problem was fixed with a hotfix provided by support, and we are now on version 8.0.2.200

[pat_ren]

thanks, no such luck with my case yet, no hotfix provided.

[Mildur]

Hi Pat / Hi all

We have published a new version (P20240910 - 8.0.2.200) that addresses the situation discussed in this topic.
Kindly refer to the resolved issues in our "Release Information" for more information and download links.
--> https://www.veeam.com/kb4656

Best,
Fabian

[pat_ren]

thanks Fabian, I started patching another server tonight and I noticed after installing (from the ISO) that it's already on v8.0.2.200 so I came back to check and saw your post with the fix, thank you.