Veeam B&R Console Freezing

[wa15]

Just updated to V10 from 9.5U4b and we have noticed that the B&R console keeps on freezing and we have to end the Backup Shell process in task manager to restart it. This is happening on multiple systems (Windows 10 1909 and Server 2016 Standard 1607). I have been able to see a pattern on when B&R freezes, but usually it's after some time of inactivity. Sometimes after 15 minutes, sometimes after 2 hours.

Not sure if related, but when opening the console, it sits at the splash screen for 3-5 minutes before starting.

Has anyone else experienced similar symptoms? Unfortunately, this is making our backup environment almost useless as we are still trying to deploy AHV proxies etc.

Case #04095157

[Dima P.]

Hello wa15,

Thank you for your post, I've merged it to existing discussion. Please keep working with our support team in order to get the mentioned fix when it's out. Cheers!

[wa15]

Thanks Dima. Any ETA on when it'll be out? This is making our environment almost unusable. We are backing up about 150 VMs with this environment. Veeam sits on a physical 14gen Dell server on flash storage (SQL Express install) so having Veeam freeze every 20 minutes, it's very difficult to use.

Also, unlike the title of this post, the console completely freezes for 10-15 minutes (sometimes longer) so it's not just slow, it just freezes.

[wa15]

Spending more time with this, the issue I am running into is more severe (and different) than what this thread is reporting. In my case, the console simply just freezes for up to an hour, sometimes 2. Eventually starts responding again for a few minutes, then dies again.

Resources are good (24 physical cores, 64GB of memory, ~150VM environment). Rebooted multiple times. Quite frustrating.

[Gostev]

Sorry for a bad merge. It's a different issue (looks like bug 204167), also known by now and support may have a hotfix already. Thanks!

[wa15]

@Gostev, is there anything we can do to speed up access to the hotfix? We are dead in the water right now as Veeam is next to unusable. Been following up for support but still waiting to hear back as of 3/31 at 5PM PST. Considering rolling back to v9.5

Also, not sure how common this bug is or what the cause is, but if it's somewhat common, perhaps it should be added to the top issues tracker thread? If I would have known the pain, I would have held off on updating

[Gostev]

No, it's not common, which is why it is not listed in the top issue tracker. If this is the bug I'm thinking about, then it only triggers due to overloaded or misbehaving SQL Server where the configuration database is hosted. So, it's more of a corner case. But at this point it's just guess based on your description, your issue could be different than the bug I'm thinking about.

[Gostev]

Checked our support system and indeed there's not much activity on your case it seems, so I asked support management to escalate it into R&D.

[Gostev]

@wa15 so I was checking on this support case, and looks like you asked to close it because the issue resolved itself? Do you have any guesses as to what may have caused it, was it some temporary problem with the environment?

[wa15]

@Gostev - Yes, the issue resolved itself around 72 hours later. Interestingly enough, CPU/Memory/Disk utilization during the console freezes were low so I can't imagine Veeam doing any data intensive tasks. Also, during the 72 hours, nothing changed that I can think of, besides usual jobs kicking off here and there as jobs were still completing successfully from what I could see, but the console itself was NOT responsive.

This environment does have AHV proxies (7 of them). I performed the V10 update in another environment with very similar physical hardware (Dell R740xd), and that completed successfully but that one did not have AHV proxies so maybe there is a bug in environments where AHV is involved? Just a rough guess.

[mrholm]

Hi
We also see this behaviou ron a newly upgraded site, 10.0.0.4461 SP2 on Windows 2016 servers,
The GUI freezes and in Resource Manager we see it as "not responding". During this time Veeam uses the temdb files a lot for both read and write although the only jobs running are SQL t-log jobs nothing else.
CLosing tdown the GUI from Task Manager and opening up again seems to work as a workaround...

[wishr]

Hi Mats,

Have you already raised a support case? If so, please share your case ID with us.

Thanks

[mrholm]

04234067 is the support ID, now support want me to follow https://www.veeam.com/kb1995 but I still say this is after upgrade to v10 and we will have a webex session on Tuesday.

[mrholm]

Hi
had a webex session with support and still refering to the above KB and also decrease the number of sessions to keep in the database, today we show 500 sessions and keep sessions for 53 weeks (defaultvalues). The KB says to decrease the number of sessions and also to drop old sessions directly in the database. To me this is just a workaround, we have had the same configuration during the 2½ years we have run veeam in the whole enviroment and have never seen this on previous versions on this server. If there's a problem keeping sessions in the database for 53 weeks in an environmnet of 75 backup jobs and similair amount of copy jobs then there should be a info about that written in some best practise or so.

[wishr]

Hi Mats,

We are keeping an eye on the case and will assist where needed.

With each B&R release, the dataset is obviously growing due to lots of new objects and parameters being added with each release and that forces us to decrease the data retention in order to guarantee the appropriate level of performance. Speaking about the direct solution, we are planning to enhance the approach to storing sessions data in the next product versions, but it's obviously not a "quick fix".

Thanks

[pdigiacinto]

Im having the same problem. Im running Version 10a. The memory on my Windows 2019 server gets up to 70 percent, the backup shell states its "not responding" and the server literally locks up.
I put in a support ticket and tried the decrease that value for SQL and that did not resolve it. Support said to create a new server, install SQL and import the Config file. I then created a new Windows 2019 server from scratch, installed V10a and imported the config file. It worked or about two weeks. Its now back the freezing up again. This is so frustrating.

[Gostev]

Please continue working with support, as this is not a know issue with 10a. This topic is from v10 times.