Veeam 365 Backups Staklling - Multiple Clients Taking 18+ hrs

[KClark.Sigma]

Hey there - I was hoping to see if anyone had heard of this before. Lots of backstory for an aging issue, summary on bottom of post.

I've been working this most recently in Case #05438177, though the history of the issue itself extends back some cases.

--------------------------------------------

A couple weeks ago, our Veeam 365 backups started going far beyond their expected running time. For our 30ish clients, times rose from a couple minutes and quickly grew into hours.

This had the effect of compounding the issue, as when backups were missed, the data needing to be backed up would grow. This spiral eventually caused every backup to run overlong (Without ever finishing, in many cases)

Much has changed since then, our larger clients are on their own repos as a safeguard, while the bulk of our smaller clients are contained in the old Repo. Aside from bandwidth issues and the time problem of writing entirely new backups to the repos, this appears to have resolved the issue for the handful of our larger clients who needed them.

However.

As it stands, it is still frequent for our smaller clients to run 24+ hrs over the course of their backups. These are clients with less than a GB needing to be backed up, and frequently stalling on the final couple objects (Common hangups include something like 17/18 objects backed up)

Support has found a work-around, revolving around targeting the mailboxes that find themselves in this stalled state. The affected objects need to be isolated and set into a separate job and manually run, all to correct any bad data that has been attempting to write.

As has been explained to me, these stalled mailboxes are attempting to backup superfluous metadata that is rewritten every backup - Assigning them to a new job forces the repo to write the uncorrupted data and once complete, resolve the issue of these stalled jobs. Once this is done, the new isolated job can be safely removed.

I am fine with this workaround, however the time required to fix a backup in this manner is sizable, but my question is this:

- Is there any way to discover why this began happening in the first place? If it's a known issue with a known workaround, is there something in a future release to stop the issue altogether?

[jorgedlcruz]

Hello Kenneth,
Thank you for the ticket number and the backstory, I have been reviewing the whole ticket you attached and a few others a bit older. I can see sometimes some timeouts on your error messages, etc.

Would you mind to double-check with the S3 Object Storage provider you have that they are not publishing any DNS entry type A for the region you are using, like us-east-1 when you query that on DNS gives you multiple IPs, as because DNS loads balance, some IP might be causing some issues. I guess they would have already called on this from others, but you never know.

I know it sounds random, but so do backups that were working fine for months, suddenly this started to happen.

I am talking purely for those random timeouts I saw from your latest responses, for the other problem where you said support has come with a workaround, I would always follow support guidance, moreover, I would recommend keeping working with them, providing logs, applying hotfixes if required, latest CU, etc. but everything with the Support hand, they are the ones that can troubleshoot these errors much quicker and efficient than us on the forums.

I will double-check internally to see if this is a known issue, or if there is something around and come back to you.

[Mike Resseler]

@KClark.Sigma

It seems you are a VCSP?

We have a dedicated forum for them

To apply for the VCSP forum, go into the User Control Panel of this forum, select User Groups in the tabs, select "Veeam Cloud Service Provider” and finally the “Join selected” option. As soon as your email address is verified to be part of the Service Provider domain, your application will be approved.

[KClark.Sigma]

@mike

Done!