Veeam 11 Remote Console Changes

[FECV]

Has anyone else had performance issues with the Veeam 11 VBR console running remotely? I have been using the VBR console connecting from my laptop to my VBR server over my companies VPN for years. For the most part the performance has been good as far as being able to click on jobs to see status, edit a job, scan a repository, etc. Immediately after updating to v11 the console is unusable. I notice no issues on my server connecting to localhost, but on my laptop the data is often out of date as far a completion percentages, clicking on a job or anything caused the console to become unresponsive for 5 min at a time not allowing me to even click the x to close the application. I have uninstalled, reinstalled and most of the basics. Support is currently looking at my logs, but has informed me nothing has changed in the way the console connects to the VBR server. Based on the how immediately the issue was noticed after I updated my console I really think something has changed. Another example that leads me believe something has indeed changed is that v10 used to pop up a window when my VPN timed out saying something to the effect of Veeam had reached the maximum amount of retires to connect, please click retry when the server is reachable. That pop up is no longer present in v11 when my VBR server is not accessible via the network.

[Egor Yakovlev]

Hi Frederick.

It is a first report of such remote console behavior that you are experiencing.
Can you please share your case ID so I will have a look at it.

/Thanks!

[FECV]

Case # 04670999

[darcon]

We do not have a case open, but we are also seeing performance issues with the v11 console that are greatly amplified when working remote. I do have a good internet connection (200/10, 20ms latency to the office), we also have a good synchronous line at the office. It takes about ~2 minutes to load once I successfully login, then an additional ~2 minutes after loading the console is unusable. I try not to keep the console open when not needed, so this poor performance is having an impact on my workflow.

Performance on-site is much better, comparable to v10.

[veremin]

Kindly, open a support ticket with our support team, so that we can gather more information about the problem you're experiencing. Thanks!

[andybishop]

I am seeing similar issues with the console since upgrading too, I am working remotely over a VPN but have a really good connection.

[aich365]

We have problems with the Home - Tape screen whether run remotely or on the VCC Console itself.
None of the jobs are listed but if you click on where you expect the job to be, the bottom window appears with job details.
We have also observed long wait times for the screen to populate before an error message appears, clicking OK to the message then brings up the screen.

We thought this may be addressed in the 0 day update so are waiting to deploy this before logging a call.
we have also noticed our SQL server is maxed out on CPU so will be assigning more resources at the next maintenance window to see if this helps.

Regards

[alexander sh]

@darcon have you had a change to open the case? If so, could you post the case number? What you describe sounds like something we definitely should look into.

[alexander sh]

@aich365 I don't think we have this problem addressed in cumulative patch 1, so definitely log a support case.

[FECV]

@aich365 Not sure your SQL environment but our Veeam instance never seems to use any CPU on the SQL server and only about 1GB of RAM. Sounds like you have your SQL running with other databases. So if its the same issue we are having the extra CPU is not going to help.

Our VBR server has 768GB RAM and 2 18 Core 72 Thread Processors at 3.37GHz. We are using the SQL Express bundled with the Veeam installer with 66 jobs. There are always plenty of resources not in use, so I doubt the issue we are seeing is a resource one either.

[pirx]

We have problems with the Home - Tape screen whether run remotely or on the VCC Console itself.
None of the jobs are listed but if you click on where you expect the job to be, the bottom window appears with job details.
We have also observed long wait times for the screen to populate before an error message appears, clicking OK to the message then brings up the screen.

We thought this may be addressed in the 0 day update so are waiting to deploy this before logging a call.
we have also noticed our SQL server is maxed out on CPU so will be assigning more resources at the next maintenance window to see if this helps.

Regards

Maybe check - or let support check - your indexes on SQL tables. We had problems with SQL cpu load, even SQL deadlocks, added more and more CPU. In the end one additional index solved all our problems.

[mbainbridge]

Just upgraded from v10 to 11. I'm also experiencing the console being slow when used over a VPN (Zscaler). v10 was just as fast as using it on the server, v11 locks up for several seconds to several minutes depending on what I click on.

I've also logged a case. #04707268

[frankho]

We are seeing the same issues with the console since upgrading to v11 if we are remotely working via VPN. If we are working onsite there´s no problem.
Our veeam case ID is:04690201

[Gostev]

Those of you affected, do you have a fast and reliable VPN connection?

We think we have found something that may make V11 console behave worse on unreliable connections. But we're wondering if we also need to dig if there were any changes in the bandwidth requirements. So knowing the connection parameters for those experiencing the issue can save us some time.

[frankho]

we are using checkpoint endpoint VPN client and don't have an VPN unreliable connection. But the packet latency time is of course slower via VPN compared to be directly on site.

[mbainbridge]

The Zscaler VPN I'm using isn't a classic point-to-point tunnel VPN, but it worked OK with whatever TCP/UDP ports v10 used. As for quality, I don't have any other problems with it. It's pretty much rock-solid for things like Windows fileshares, Remote Desktop and the usual office stuff.

I've just downloaded a file from the office at 3.5MB/s, which is about the limit of my broadband connection. Upload is worse, at 350-400KB/s, but again, that's about as much as the broadband can deliver. Speedtest.net through Zscaler is showing 32.5Mbps down and 3.58Mbps up.

Are there any new ports that v11 is using? It could be I need to get these added to some list.

[FECV]

My VPN is all Cisco AnyConnect and ASAs. I ran a speed test transferring a file directly to the VBR server from my laptop and was getting 20Mbps writing data to the server and 90Mbps reading data from the server. No dropped pings over the course of a few hours even while the console was hanging.

[FECV]

Support had me uninstall the AWS and Azure plugins on my console machine that were installed when my console was upgraded from v10 to v11. It did not seem to have any impact on my issue. Just an FYI.

[misteritgeek]

I logged case #04730113 for this issue as well.

I'm managing a fresh v11 CloudConnect environment remotely using PowerShell over a 300Mbps connection on the console server with a consistent 56ms ping to the CloudConnect server. I noticed that it takes upwards of 1.5 min for the "Connect-VBRServer -Server my.server.com..." command to complete. "Disconnect-VBRServer" then takes 1 min to complete. Performing other PowerShell admin tasks after a connection has been established works at a normal speed.

The same test performed using a PowerShell session using a stable connection with 160ms ping results in a new connection taking 4.7 min to complete, so the issue appears to scale linearly with connection latency.

The same Connect and Disconnect Cmdlets complete in less than 3 sec and 1 sec respectively when performed from the Service Provider LAN with <1ms ping times. Removing the AWS and Azure plugins from the freshly built B&R console VM made no difference.

[Egor Yakovlev]

Thanks for sharing ID and your findings.
Case is under investigation.
Stay tuned!

[oleg.feoktistov]

@misteritgeek, I noted your case from powershell perspective. I'll keep tracking it and ask internally meanwhile. Thanks!

[anarinian]

case ID: #02296086

We have the same issue. Upgraded from v10 to v11 yesterday (the full suite with veeam one, EM, and VBR) ;; Console is unresponsive and jobs stopped working. Last night I had multiple jobs complete in normal time but then seemingly out of no where, everything stopped functioning. We are currently unable to open VBR console from our laptops. The only working console is open on the VBR server. Currently exporting logs.. (its been 2 hours since the log export started and it's only at 65%) ;; seems SUPER slow.

[FECV]

I was told that R&D may have an idea of what the issue is, and that there is a hotfix being worked on. No ETA or anything yet.

[Dima V.]

Hello we prepared the hotfix. Please ask support for it

[jeffers]

I am experiencing extreme slowness over a VPN connection too, only since updating to v11

Right now the console won't open at all, giving an error after sitting on the splash screen for ages: User request [xxxxx] for license refresh failed due to timeout.

I have a ticket logged - Case #04740766

[aich365]

We have added more resources to the SQL server and the VCC console is more responsive but we still have issues with the Home Tape Screen
On first login the display is okay and loads within 10 seconds, but as you move between items it becomes unresponsive and even returning to the first menu selected it does not redraw the screen correctly, so we click on Home Jobs Tape and none of the jobs are displayed, but if you click where you expect a job to be it does display details in the lower window.

I will be documenting details and raising a call.
Regards

[nathano]

I'm also noticing a slow down in the v11 console, running on the VBR server. some tape jobs don't appear in the list of currently running jobs. and when clicking a selection on the left (e.g. "Running Jobs" on the home screen) I then get scrolling dots at the top of the right screen while it populates. I've seen in on 2 different VBR servers that I manage, both upgraded from 10.

Case 04785889 logged.

[aich365]

Case #04790415

History of tape jobs does not include the last 2 weeks

Thanks

[nathano]

I can see in vCentre that the average CPU usage of my VBR has increased significantly since the v11 upgrade. and it's the sql process that's chewing the CPU

[aich365]

Had a remote session with Veeam Engineer today.
There is an issue with SQL Locks numbering over 1million instead of the more usual 500,000.
Changing the History from 26 weeks to 4 weeks in Veeam has not resulted in less sessions as the script to reduce this times out.
The sessions are dropping but very slowly.

Thanks