Stuck merges

[jround]

Does anyone have any idea how to diagnose stuck backup merges with Veeam?

The issue is we have a Veeam 9.5 which is fully updated to patch 4a but there are 4 jobs (we create a job for each individual VM) which seem to be stuck on the merging of oldest incremental backups into full backup files procedure

Server 1 : Merge (60% done) - VM size 250GB
Server 2 : Merge (17% done) - VM size 1TB
Server 3 : Merge (38% done) - VM size 675GB
Server 4 : Merge (33% done) - VM size 450GB

They have been stuck overnight as I thought I would leave them to see if they cleared, but nope. Then I did a full server restart which of course failed the jobs so I could try starting them again, sadly they then just kick in to 'complete previously interrupted transform' and get stuck at exactly the same point.

Repository storage is all local to the server and the server itself is more than powerful enough, Xeon Silver 4114, 64GB RAM etc, and all the other jobs are running and merging absolutely no problem at all, it just seems to be 4 jobs that are totally and utterly stuck for some reason!

I am sadly still awaiting details off the former engineer who left last week as to our details so I can login and create a support ticket! so just wondered if anyone on here had any quick ideas before I manage to get hold of those, as a last resort I guess it would be deleting the problematic jobs and starting again from scratch? (obviously we would then lose the backup chain for the servers though so a slight risk which I would rather avoid if at all possible!)

[Vek17]

Is the target repository here a simple Repo or SoBR?
If the repo is an SoBR is it set to performance or data locality?

This sounds similar to something I experienced on the transition of 4a. If you are using an SoBR in performance mode there is a bug in the repository agent that causes massive degradation in synthetic operations (like merges). If so you will need to contact Veeam support to verify but I believe it is Fix_168898_25b4556cf0 (or at least that is what the fix was called when it was loaded to me).

If that is not the case can you provide more details regarding your repository setup?

I will say if your jobs are getting stuck without errors, it is entirely possible that getting rid of the broken chain and reseeding would not fix the problem so I would avoid that course if at all possible.

[jround]

Thank you, I have just checked the configuration and it is indeed a Scale Out Backup Repository set to performance mode, 4A was installed just the Friday gone 31/05 so looks as if it may be linked and I will need to obtain the fix

[jround]

Just an update, I finally got the support credentials figured out and support very kindly sent through the patch which resolved the issue for me

[mdiver]

I assume this is fixed in 4b?

[Vitaliy S.]

OP didn't' share his support case ID, so I cannot confirm that. But given that there is a hotfix available, then it should have been included in the cumulative 4b rollup.

[mdiver]

It is. Customer had the same issue. Case #03668026

Handled very professionally and to the point by Veeam support! Thanks!

I'll follow up with a feature request in another post related to merge issues…

Mike