Comprehensive data protection for all workloads
Post Reply
jround
Novice
Posts: 7
Liked: 3 times
Joined: Jun 04, 2019 3:01 pm
Contact:

Stuck merges

Post by jround » Jun 04, 2019 5:06 pm

Does anyone have any idea how to diagnose stuck backup merges with Veeam?

The issue is we have a Veeam 9.5 which is fully updated to patch 4a but there are 4 jobs (we create a job for each individual VM) which seem to be stuck on the merging of oldest incremental backups into full backup files procedure

Server 1 : Merge (60% done) - VM size 250GB
Server 2 : Merge (17% done) - VM size 1TB
Server 3 : Merge (38% done) - VM size 675GB
Server 4 : Merge (33% done) - VM size 450GB

They have been stuck overnight as I thought I would leave them to see if they cleared, but nope. Then I did a full server restart which of course failed the jobs so I could try starting them again, sadly they then just kick in to 'complete previously interrupted transform' and get stuck at exactly the same point.

Repository storage is all local to the server and the server itself is more than powerful enough, Xeon Silver 4114, 64GB RAM etc, and all the other jobs are running and merging absolutely no problem at all, it just seems to be 4 jobs that are totally and utterly stuck for some reason!

I am sadly still awaiting details off the former engineer who left last week as to our details so I can login and create a support ticket! :roll: so just wondered if anyone on here had any quick ideas before I manage to get hold of those, as a last resort I guess it would be deleting the problematic jobs and starting again from scratch? (obviously we would then lose the backup chain for the servers though so a slight risk which I would rather avoid if at all possible!)

Vek17
Service Provider
Posts: 39
Liked: 10 times
Joined: May 29, 2018 8:42 pm
Contact:

Re: Stuck merges

Post by Vek17 » Jun 04, 2019 8:11 pm

Is the target repository here a simple Repo or SoBR?
If the repo is an SoBR is it set to performance or data locality?

This sounds similar to something I experienced on the transition of 4a. If you are using an SoBR in performance mode there is a bug in the repository agent that causes massive degradation in synthetic operations (like merges). If so you will need to contact Veeam support to verify but I believe it is Fix_168898_25b4556cf0 (or at least that is what the fix was called when it was loaded to me).

If that is not the case can you provide more details regarding your repository setup?

I will say if your jobs are getting stuck without errors, it is entirely possible that getting rid of the broken chain and reseeding would not fix the problem so I would avoid that course if at all possible.

jround
Novice
Posts: 7
Liked: 3 times
Joined: Jun 04, 2019 3:01 pm
Contact:

Re: Stuck merges

Post by jround » Jun 05, 2019 12:01 am 1 person likes this post

Thank you, I have just checked the configuration and it is indeed a Scale Out Backup Repository set to performance mode, 4A was installed just the Friday gone 31/05 so looks as if it may be linked and I will need to obtain the fix

jround
Novice
Posts: 7
Liked: 3 times
Joined: Jun 04, 2019 3:01 pm
Contact:

Re: Stuck merges

Post by jround » Jun 07, 2019 10:33 am 2 people like this post

Just an update, I finally got the support credentials figured out and support very kindly sent through the patch which resolved the issue for me :)

mdiver
Service Provider
Posts: 56
Liked: 11 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Stuck merges

Post by mdiver » Jul 17, 2019 7:31 am

I assume this is fixed in 4b?

Vitaliy S.
Product Manager
Posts: 22860
Liked: 1538 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Stuck merges

Post by Vitaliy S. » Jul 17, 2019 10:10 am

OP didn't' share his support case ID, so I cannot confirm that. But given that there is a hotfix available, then it should have been included in the cumulative 4b rollup.

mdiver
Service Provider
Posts: 56
Liked: 11 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Stuck merges

Post by mdiver » Jul 17, 2019 1:57 pm 1 person likes this post

It is. Customer had the same issue. Case #03668026

Handled very professionally and to the point by Veeam support! Thanks! :)

I'll follow up with a feature request in another post related to merge issues…

Mike

Post Reply

Who is online

Users browsing this forum: emoe, Google [Bot] and 30 guests