Forward incremental takes 10+ mins on Finalizing stage

[namiko78]

Hello

I have a forward incremental job with 130 vm's that seems to take it's longest amount of time on the finalizing stage. There can be 10+ vm's in the list that are at 99% and sitting at this stage. I have 2 virtual proxies, and the repository is physical with iscsi storage. The bottleneck reports to be 'source' but what is it doing at this stage that takes so long? My proxies have 8 concurrent tasks each max, and the repository has 10 concourrent allowed.

Where is my bottleneck?

I do notice the database server (remote SQL 2012) can sometimes spike to 100% for a few seconds. Could that be my issue?

Thanks

[Shestakov]

Hello Steven,

Could you provide us your job statistics?
What transport mode is used to retrieve VMs data from the datastore? An easy way to figure it out is to click on a VM in the job session log(left down corner) and check a transport mode tag ([san], [hotadd], or [nbd]) next to the name of the proxy server assigned for the processing.
Thanks.

[namiko78]

Sure.

The 2 backup proxies are using hotadd.

end of jobs stats are : busy: source 97% proxy 61% network 2% target 0%
Typical transferred amount is 50gb ish, 241gb read. 12.9TB processed.

Are those what you're looking for?

Thanks

[Shestakov]

Steven,

Thanks for the info. Is that statistics related to an incremental or full backup run?
To have better job performance it makes sense to try to configure Direct SAN mode.

By the way, what versions of vSphere and Veeam B&R are you at?

Thank you.

[namiko78]

We're ordering a new backup server with 10gb network, so once that arrives i'll switch to direct SAN mode. We're at the latest patch and version level.
So what is it doing in that finalizing stage that's taking so long? Are you saying it's related to my hot-add proxies?

Thanks

[foggy]

Steven, if you select the particular VM that stucks at 99% in the job stats window, what is the last action it is stuck on?

[namiko78]

It's doing the finalizing task, for 5-10 mins, then completes the rest quickly after it turns green on that task. Is this normal behavior?

[namiko78]

I guess i'm confused by this. If i have a VBK, and 7 VBR's that contain the previous block changes, if the oldest restore point has to be removed, can't you just simply remove the oldest VBR and the changes that go along with it?

[Shestakov]

Sorry for the confusion Steven,

I`ve mixed up Finalizing stage of replication and backup and finally deleted the post.
It will be helpful for you to contact Veeam support team for logs investigation. It might help to find the problem.
Please provide us your support case so we can follow the situation. Also could you tell us how your job statistics changes once you switch to Direct SAN mode?
Thanks.

[namiko78]

Ok. I may be converting these jobs to reverse increment mode soon, so i'll wait until that is decided. Hoping my repo has enough IO to run in this mode.

[Murigar]

I am having an issue where all VMs in one job require between 19-25 min each to complete "Finalizing".

All other jobs and VMs complete this in less than 5 seconds.
All jobs are using traditional incremental backup method with weekly synthetic and monthly active full.

Can someone explain what "Finalizing" is doing? And some reasons why all with this job are taking 20 min vs 5 seconds?

[veremin]

Can someone explain what "Finalizing" is doing?

Meta files update, summary saving, vCenter connections' disposal, data movers' stop, etc. The problem with slow finalizing step might be related to size of meta, slow target, stuck datamovers, etc.

But it's hard to tell without seeing logs. Therefore, please, open a ticket with our support team.

Thanks.

[Murigar]

As a follow up. I "Cloned" the job and this resolved the issue entirely.

[veremin]

Thank you for sharing the resolution. Should the issue occur again, feel free to contact support team directly.