Extremly slow navigation in the "Backups" section

[FBraendle]

Hi Guys

We have several physical veeam backup servers.
All of them work actually fine but "the biggest" (backs up ~140 VMs).

Everything works fine and is fast but as soon as the "Backups" section on the left is selected, the veeam console literally dies away... (meaning it gets ultra slow. expanding a backup job to see the vms takes minutes(if it works at all)
I cant get rid of old backup points of deleted vms so they get dragged along leading to increased space consumption of the backups..

Is there maybe some maintenance task I can run against the DB or something to speed it up again?

Kind Regards
Felix

[Vitaliy S.]

Hello Felix,

Could yo please tell us what version/build of Veeam B&R are you using? Thanks.

[FBraendle]

Hi Vitaly

Sure, sorry
its: 5.0.1.198 (32 bit)

I'm actually just downloading version 5.0.2.230.

ill have it upgraded this afternoon, lets see, ill inform you as soon as i know more...

Greets
Felix

[Vitaliy S.]

Yes, please, though I'm pretty sure that your issue will be fixed after the upgrade, see the Release Notes.

[tsightler]

I'm pretty sure you won't have the problem with 5.0.2.230 as I had the same issue (taking as much as 60 seconds for every click in this window). Unfortunately, there's appears to be a new behavior in the same screen that causes deadlocks when attempting to remove backups using the "Remove from backups" option. I've tried to work with Veeam support on the issue since it's 100% reproducible for me, but so far that's been a typical, modern-day support experience (i.e. "not good"). It appears that this screen is always "auto refreshing" and if you happen to be deleting a job that takes longer than the "refresh" then you get a deadlock. Perhaps something weird in my environment but the support tech said that they've had other reports of this issue and the "proper" method is to just keep choosing the option over and over until all the backups are finally removed.

[FBraendle]

Hi Guys

As stated, it seems to be fixed now(its "wicked" fast again )
Lets hope i don't run into the issue that tsightler has...
ill start to remove the old machines and will give another feedback how the console is reacting...

Greets & thanx
Felix

[FBraendle]

update:
i just wanted to write: "no deadlocks, works fine..."

but exactly at this point i got a EM:
======================
failed to execute stored procedure, Name: [[Backup.Model.DeleteOIB]], Params: [@id = 48422f62-0946-4909-aee0-365e1ec50f28].
Transaction (Process ID 55) was deadlocked on lock | generic waitable object resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
======================
my experience:
I OK the message.
i select the machine again, selection disappears(seems to be some kind of refresh, as stated before)
i select the machine once more and remove it without a problem...

so there is still a bug/issue...


Greets
Felix

[Vitaliy S.]

Felix, could you please reach our support team as well? On doing it, please send me a PM with your ticket number, so I could update this topic once this issue is resolved.

[FBraendle]

Hi Vitaly

I'm gonna do that tomorrow and will update you with the needed information...

Greets
Felix

[tsightler]

BTW, my ticket number is 5132925, opened May 23rd. Here is the initial response from support:

The deadlock errors are coming from your SQL server not from Veeam. we have seen this a few times and so far what you are doing is the only solution.

Which seems pretty bogus. Of course the SQL server is the source of the message, but the deadlock itself is caused by some SQL Veeam is running. I've attempted to explain this to the support people but didn't get very far. This issue is 100% reproducible on my Veeam servers, although it does require a delete that takes longer than a screen refresh, so basically a job that includes a lot of rollbacks or a lot of VM's (or both).

[Vitaliy S.]

Tom, thanks. I've already passed this information to our R&D team.

[a.fokkema]

I'm pretty sure you won't have the problem with 5.0.2.230 as I had the same issue (taking as much as 60 seconds for every click in this window). Unfortunately, there's appears to be a new behavior in the same screen that causes deadlocks when attempting to remove backups using the "Remove from backups" option. I've tried to work with Veeam support on the issue since it's 100% reproducible for me, but so far that's been a typical, modern-day support experience (i.e. "not good"). It appears that this screen is always "auto refreshing" and if you happen to be deleting a job that takes longer than the "refresh" then you get a deadlock. Perhaps something weird in my environment but the support tech said that they've had other reports of this issue and the "proper" method is to just keep choosing the option over and over until all the backups are finally removed.

I have the same issue when I try to remove an old Backup job which from my Veeam Backup & Recovery servers. We're running 5.0.2.230 x64 with a local SQL server. After ten times or so, I am finally able to remove the old job. Hope this issue can be fixed in the next release.

[Vitaliy S.]

Arne, could you please contact our support team to have a look at your environment? We haven't heard about this issue since 5.0.2 release.

[tsightler]

Vitally, he's having the issue with remove jobs and support has verified that they've seen this issue multiple times. I've reproduced the issue on a remote session with support and they gathered a SQL trace and I'm currently awaiting a fix. I have come to believe that this may only impact jobs with forward incrementals, at least that what I'm seeing in my testing.

[daniflexx]

same problem over here. If you try to delete the job over and over again eventually you will succeed.

[Gostev]

Hi, Dani. Thanks for reusing existing topic. You can obtain a hotfix for this issue from our support. Thanks!

[brupnick]

I just tried to import a backup and received the deadlock error. Does the hotfix that you mentioned address this as well or only when deleting backups (which I also have)?

[Vitaliy S.]

Yes, it should solve this as well. Anyway, with all hotfix related questions please contact our support team directly. Thanks!

[jmcmurry]

+1 "me too" on SQL stored procedure deadlocks while removing backups. I just deleted a set of imported backups and it took 10-15 tries to completely remove the set. My backups are all forward incrementals as well.

[brupnick]

I got the hotfix from support, but when I applied it, VeeamShell would crash every time I tried to open the program. I have a ticket open with Support and am trying to schedule a WebEx with them for Monday.