Veeam8 console frequent 'Not Reposnding' in Task Manager

[prabhash.jena]

Hi,

I am struggling with this issue from past 1 month now.

The issue started on Server 2008R2, we had frequent Veeam console hang and Enterprise manager timeouts. Admin user can't logout or login on some occasions. The only option was to reset the server.
I deployed new VM Server 2012R2 from template and installed Veeam 8 and used the same database but issue remained. Thinking that the issue might be because of template or database I wiped off everything and did everything from scratch again.
I installed OS from ISO used new database while installing Veeam and to my surprise the issue came again. My colleague at Service desk sometime can't use Enterprise manager link as it times out. If they login to the server the Veeam console hangs every now and than. This is a sporadic issue and we often experience it during our routine work.

Any suggestions where o look at to fix the issue.

Regards
Prabhash Jena

[Vitaliy S.]

Hi Prabhash,

How many backup jobs do you have configured for the backup console? How much RAM and CPU resources do you have assigned for this VM? Do you see this issue when doing some particular action in the console or it just random?

Thanks!

[prabhash.jena]

Hi Vitaliy,

I have 27 jobs running as daily backup jobs and 4 copy jobs.
4 CPU 16 GB RAM. Issue is very random sometime it happens while editing a job sometime while viewing a job history. I don't see any pattern.

[foggy]

Hi Prabhash, is this server also used as a proxy and/or repository or just a dedicated Veeam B&R/Enterprise Manager console server? Have you monitored resources consumption on it?

[prabhash.jena]

Hi,
This is a dedicated server for Veeam B&R/Enterprise Manager.
There is absolutely no issues with resources, the CPU and memory are always normal, no spikes what so ever.
Its a process deadlock issue.

[foggy]

I'd recommend involving support then, haven't you contacted them for logs review?

[prabhash.jena]

Hi,

I have a support case with Veeam from past 15 days and so far we have not made any significant progress. I posted to check if any one has some suggestion or fix. If I disable all Veeam services server works great so I am certain its Veeam modules (esp. Shell modules) which are corrupt.

[Vitaliy S.]

Can you tell us your case ID?

[prabhash.jena]

My case ID is 00967046, after a long delay I received communication that someone is going to do a log analysis.
I hope to get a resolution.

[Andreas Neufert]

Maybe use Veeam ONE to analyse the Virtual Environment where the B&R Server is running on.
CPU waits?
RAM Ballooning?
High Disk read/write latency?

[prabhash.jena]

We have more than 250 servers running on the same vSphere platform and I don't see any performance issue on any server, neither our customers have reported any performance issues so far (during the time we had issues with Veeam server). We have 4 offsite jobs to a NAS box for one customer, I had disabled those jobs on Monday and since than the issue has not reoccurred. The port to which the NAS box is connected at customer end has been tapped with 10 Mbit (very slow) and hence the copy jobs actually never finishes sometime it hangs at merge and sometime other failures. I highlighted this to the support engineer as well however I was advice that it won't make any issues. In my case it has as after 4 days of monitoring the server has not hanged once. Could it be TCP Window size\scaling causing hang?

[Andreas Neufert]

Did I get it right ? Between the Veeam Server and the NAS Box is no additional Server with a Veeam Repository?

If so, you try to write directly to a NAS box over an 10Mbps link?
The standard setup for this would be
B&R write to a repository. You create a new Repository with the NAS box and install a Proxy Server on the same site as the NAS (CIFS) box is. Then you create a new Repository to the NAS box and select the Proxy (from same site as NAS box) as Gateway Proxy (see Repository setup).

So the data flow goes between 2 Repository or Proxy Server from Veeam at the 10Mbps link. This is supported. But to write directly to a NAS box with 10Mbps is not an good option for ANY windows system.

If you have Enterprise Plus licenses, you can add WAN Accelerator role at each site.

[prabhash.jena]

We host the servers for this customer so there is no option available to host a server at customer site and make it a proxy\repository (this has worked in Veeam version 7). I enabled the jobs and the issue re-surfaced so this is a bug in the new version. I have provided the logs to support team and am waiting for further response.

[Andreas Neufert]

So you write directly to a CIFS share through a 10Mbps WAN link. This is a ugly solution and is totally against our best practices and documentation.

[foggy]

Prabhash, setup is indeed not ideal. Could you please tell what is the source for those backup copy jobs?

[prabhash.jena]

Prabhash, setup is indeed not ideal. Could you please tell what is the source for those backup copy jobs?

I have selected the source to be from backup files which reside on a CSV cluster shared volume.

[prabhash.jena]

So you write directly to a CIFS share through a 10Mbps WAN link. This is a ugly solution and is totally against our best practices and documentation.

I agree this may not be the best solution but it worked on Veeam 7 & the issue started after we upgraded to Veeam 8. The jobs should fail but it shouldn't hang the server atleast and make it unusable.

[prabhash.jena]

So you write directly to a CIFS share through a 10Mbps WAN link. This is a ugly solution and is totally against our best practices and documentation.

I completely agree, it’s not an optimal solution however the fact remains that this hangs the whole backup management server. It is acceptable that the job fails or takes forever to finish but in no condition it should hang the whole server and put down the all jobs. Atleast not expected from a finished product like Veeam.
Moreover Veeam support was not able to highlight this that the suboptimal setup is causing the whole server to fail tough I indicated that I have copy jobs directly to CIFS offsite share and the case lasted for more than an month. Finally when I disabled the jobs completely the server remains stable. So it’s defiantly a bug which need to be addressed because I never experienced the issue with v7.
Now not sure how things will respond when I upgrade to v9.


Thanks
Prabhash