Comprehensive data protection for all workloads
Post Reply
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Forward incremental takes 10+ mins on Finalizing stage

Post by namiko78 »

Hello

I have a forward incremental job with 130 vm's that seems to take it's longest amount of time on the finalizing stage. There can be 10+ vm's in the list that are at 99% and sitting at this stage. I have 2 virtual proxies, and the repository is physical with iscsi storage. The bottleneck reports to be 'source' but what is it doing at this stage that takes so long? My proxies have 8 concurrent tasks each max, and the repository has 10 concourrent allowed.

Where is my bottleneck?

I do notice the database server (remote SQL 2012) can sometimes spike to 100% for a few seconds. Could that be my issue?

Thanks
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by Shestakov »

Hello Steven,

Could you provide us your job statistics?
What transport mode is used to retrieve VMs data from the datastore? An easy way to figure it out is to click on a VM in the job session log(left down corner) and check a transport mode tag ([san], [hotadd], or [nbd]) next to the name of the proxy server assigned for the processing.
Thanks.
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by namiko78 »

Sure.

The 2 backup proxies are using hotadd.

end of jobs stats are : busy: source 97% proxy 61% network 2% target 0%
Typical transferred amount is 50gb ish, 241gb read. 12.9TB processed.

Are those what you're looking for?

Thanks
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by Shestakov »

Steven,

Thanks for the info. Is that statistics related to an incremental or full backup run?
To have better job performance it makes sense to try to configure Direct SAN mode.

By the way, what versions of vSphere and Veeam B&R are you at?

Thank you.
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by namiko78 »

We're ordering a new backup server with 10gb network, so once that arrives i'll switch to direct SAN mode. We're at the latest patch and version level.
So what is it doing in that finalizing stage that's taking so long? Are you saying it's related to my hot-add proxies?

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by foggy »

Steven, if you select the particular VM that stucks at 99% in the job stats window, what is the last action it is stuck on?
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by namiko78 »

It's doing the finalizing task, for 5-10 mins, then completes the rest quickly after it turns green on that task. Is this normal behavior?
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by namiko78 »

I guess i'm confused by this. If i have a VBK, and 7 VBR's that contain the previous block changes, if the oldest restore point has to be removed, can't you just simply remove the oldest VBR and the changes that go along with it?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by Shestakov »

Sorry for the confusion Steven,

I`ve mixed up Finalizing stage of replication and backup and finally deleted the post.
It will be helpful for you to contact Veeam support team for logs investigation. It might help to find the problem.
Please provide us your support case so we can follow the situation. Also could you tell us how your job statistics changes once you switch to Direct SAN mode?
Thanks.
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by namiko78 »

Ok. I may be converting these jobs to reverse increment mode soon, so i'll wait until that is decided. Hoping my repo has enough IO to run in this mode.
Murigar
Enthusiast
Posts: 37
Liked: 4 times
Joined: Nov 05, 2014 4:19 pm
Contact:

[MERGED] : "Finalizing" Requires 20 min

Post by Murigar »

I am having an issue where all VMs in one job require between 19-25 min each to complete "Finalizing".

All other jobs and VMs complete this in less than 5 seconds.
All jobs are using traditional incremental backup method with weekly synthetic and monthly active full.

Can someone explain what "Finalizing" is doing? And some reasons why all with this job are taking 20 min vs 5 seconds?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by veremin »

Can someone explain what "Finalizing" is doing?
Meta files update, summary saving, vCenter connections' disposal, data movers' stop, etc. The problem with slow finalizing step might be related to size of meta, slow target, stuck datamovers, etc.

But it's hard to tell without seeing logs. Therefore, please, open a ticket with our support team.

Thanks.
Murigar
Enthusiast
Posts: 37
Liked: 4 times
Joined: Nov 05, 2014 4:19 pm
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by Murigar » 2 people like this post

As a follow up. I "Cloned" the job and this resolved the issue entirely.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Forward incremental takes 10+ mins on Finalizing stage

Post by veremin »

Thank you for sharing the resolution. Should the issue occur again, feel free to contact support team directly.
Post Reply

Who is online

Users browsing this forum: Tesla2k and 52 guests