Real-time performance monitoring and troubleshooting
Post Reply
ngerasimatos
Novice
Posts: 4
Liked: never
Joined: Oct 02, 2010 2:39 am
Full Name: Nicholas Gerasimatos
Contact:

vCenter keeps disconencting

Post by ngerasimatos »

Installed Veeam Monitor as a trial, we are currently testing it to see if it can provide the information, logging, and reporting we need for our two vCenter servers (environments) however, one of the two vCenter servers keeps disconnecting constantly and as such cannot pull enough data to use the reporting feature. The alert it says is regarding connectivity being lost.

Troubleshooting steps are below:

Verified the vCenter server and server housing Veeam Monitor are not lossing network connectivity (ping -t show no packets dropped, ever)

Checked database integrity of vCenter and Veeam.

Checked credentials for vCenter server.

Rebooted both Veeam monitor server and vCenter servers.

Any suggestions where to go from here?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Hello Nicholas,

Do you have anything special with this very vCenter Server? What is the vCenter Server's version in question? Have you tried opening a MOB web page when you have a disconnected error message within a Veeam Monitor?

I would also suggest reviewing VeeamDCS.log to shed some light on this behaviour. Please note that if you need any assistance with this, don't hesitate to contact our technical team.

Thanks!
ngerasimatos
Novice
Posts: 4
Liked: never
Joined: Oct 02, 2010 2:39 am
Full Name: Nicholas Gerasimatos
Contact:

Re: vCenter keeps disconencting

Post by ngerasimatos »

Where is the VeeamDCS.log located?

Version is 4.1
Build 258902

Not sure what a "MOB web page" is.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

To find VeeamDCS.log, please navigate to Options->Server Settings->Logs.
ngerasimatos
Novice
Posts: 4
Liked: never
Joined: Oct 02, 2010 2:39 am
Full Name: Nicholas Gerasimatos
Contact:

Re: vCenter keeps disconencting

Post by ngerasimatos »

Can I email you the log?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vCenter keeps disconencting

Post by Gostev »

Hi Nicholas, for any technical issues you should open a support case and work with our support team directly. You are eligible for support during your official trial license period. Thanks!
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

hi.

I've got the same problem. vCenter is disconnecting. Replication job ends with:

Checking "server name" host license level
Failed to login to "vCenter server name" by SOAP, port 443, user "vcenter\user name", proxy srv: port:0
Unable to connect to the remote server
No connection could be made because the target machine actively refused it 1xx.xx.xxx.xx:443

there should be a ticket regarding this: ID 557286

My configuration:
2 sites
5 ESX
2 veeam B&R server (on each site)
1 monitoring server
1 vCenter

I tought, that the problem was in vCenter database settings (by default, max. number of db connection on vcenter server is set to 10)
(in vSphere client go to Administration -> vCenter server settings -> Database)

I've changed it to 100, but it seems that it didn't help.

problems started after implementation of second B&R server.

Jozef
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Jozef, I doubt that the max number of db connections is the real cause for this issue. Do you have those connection drops while Veeam Monitor is running? or this issue can be reporduced just with a replication job? By the way, do you have any errors in the Windows Event log where vCenter server is installed?
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

I've turned veeam Monitor server off, and vCenter is running fine for 5 days...

next step: reinstalling veeam Monitor server.
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

bad news. after 5 days, error occurred again...

Checking "server name" host license level
Failed to login to "vcenter" by SOAP, port 443, user "username", proxy srv: port:0
Unable to connect to the remote server
No connection could be made because the target machine actively refused it 192.xxx.xxx.xxx:443

My next step will be reinstalling vCenter server.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Jozef, before doing that I would suggest contacting VMware support guys to assist in vCenter Server log files investigation. Also reading back through my response above, have you had a chance to check Winows Event logs for vCenter Server?
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

Vitaliy S -> there is nothing special in Windows event logs on vCenter server. Ticket to veeam support is already created.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Jozef, I would appreciate if you could update this topic with your findings later on.
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

Vitaliy S >
I really would like to understand, how is the communication made between Veeam B&R, Veeam Monitor <--> vCenter, esx hosts.

I can't reinstall vCenter server, so at this moment, I am just monitoring symptoms of the issue: (anyway, if I reinstall vCenter, I will have to do all my replicas again...and that is more than 3 TB datas via WAN, so I wont get approval from customer soon....)

1. when the Veeam monitor server is running + 2x Veeam B&R server, vCenter will refuse connection on https port 443 approx after 1 day (sometimes 2x per day, than restart of vCenter helps)

2. when the Veeam monitor is TURNED OFF, only 2x Veeam B&R servers are running, vCenter keeps working for about 5 days, but after, issue is the same (and again restart of vCenter)

first I tought, that allowing more connection to vCenter SQL database will helps, but I was wrong.

do you have any idea, how can I check, why https port on vcenter side is refused after some time?....and of course, maybe I am going wrong way with my investigation, so please advise

thanks

btw...in Veeamagent_srv.log I can see following information

[19.01.2011 06:11:06] < 2888> srv| Single-session server has been started.
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted

sometime 2 lines, sometimes 10, 15 lines of the same info....
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Jozef wrote:1. when the Veeam monitor server is running + 2x Veeam B&R server, vCenter will refuse connection on https port 443 approx after 1 day (sometimes 2x per day, than restart of vCenter helps)
2. when the Veeam monitor is TURNED OFF, only 2x Veeam B&R servers are running, vCenter keeps working for about 5 days, but after, issue is the same (and again restart of vCenter)
Sounds like the vCenter Server might be starving on the some resources (might be memory). Can you tell me the hardware configuration of your vCenter server?
Jozef wrote:do you have any idea, how can I check, why https port on vcenter side is refused after some time?....and of course, maybe I am going wrong way with my investigation, so please advise
The first thing that I would look at is vCenter Server log files. I think it might be even worth taking someone from VMware Support for assistance with this.

On top of that, don't be afraid to play with vCenter Server... what I would recommend to do though, is to backup its configuration database. So if you happen to re-install it, you will just need to point vCenter server to the backed up database, meaning that you won't lose your Inventory settings, VM IDs etc.

Thank you!
gchapman
Novice
Posts: 8
Liked: never
Joined: Feb 03, 2011 6:32 pm
Full Name: Gabriel Chapman
Contact:

Re: vCenter keeps disconencting

Post by gchapman »

Somewhat same issue here that has plauged us since we installed Veeam Monitor.

Veeam Monitor simply stops collecting data from the vCenter server daily leaving large gaps in historical analysis.

03/02/11 10:55:27 Error 0x0388 vcenter server "Unable to load/parse clients.xml: Unable to connect to the remote server"
03/02/11 10:55:27 Error 0x0388 vcenter server "XML Content:
(null)"
03/02/11 10:55:27 Error 0x0388 vcenter server "Collecting thread has failed to initialize (Unable to connect to the remote server) and will be stopped"

Troubleshooting this issue with support for almost 3 weeks now, and today recieved a message that it is an issue with ESXi and that they are working with VMWare to resolve the issue.

Suffice to say, I don't find that response all that encouraging, and right now the product simply does not perform as advertised. I would be curious to know if others with pure ESXi 4.1 environments are having similar issues.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Gabriel, can you tell me your ticket number, so I could have a look?
gchapman
Novice
Posts: 8
Liked: never
Joined: Feb 03, 2011 6:32 pm
Full Name: Gabriel Chapman
Contact:

Re: vCenter keeps disconencting

Post by gchapman »

PM sent with ticket number.

Thanks
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

Vitaliy S. wrote: The first thing that I would look at is vCenter Server log files. I think it might be even worth taking someone from VMware Support for assistance with this.
I already have logs from vCenter but that'ss more than 1GB of logs....can you please navigate me to the most important log files? thanks.

btw, I don't think, that vmware support will helps here because vCenter without veeam software is working fine.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Jozef,

I believe there is an issue with the interaction between vCenter Server and 3rd party tools (not only Veeam). It might somehow be connected to scalability or configuration settings, so I do suggest reaching out VMware team.

As for the log files, I would have a look at vpxd*.log at first, then look through the most recent logs in the folder below:

Code: Select all

C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs
Thanks!
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

I think that finaly, I've find out the issue.
MS SQL 2005 express offers only 4GB of database files (db+log),...and the DB on vCenter was about 4 GB....when DB is full, every new transaction is going to cash.
when cash is full, databse will just stop = vCenter can't run without access to DB.
After reboot, cash is free again, so vCenter works again until the cash is full. (in my case, 12 - 18 hours)

Now my question. Will vCenter works on MS SQL 2008 express? what is the correct upgrade path? I don't want to reinstall vCenter, just export/import database.
Anybody has experience?

thanks
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter keeps disconencting

Post by Vitaliy S. »

Thank you for this update, Jozef.

Based on the Release Notes for vCenter Server 4.1 u1, MS SQL 2008 R2 is fully supported.

I guess all you need to keep the current infrastructure is to migrate the database to the new SQL instance and point vCenter Server to that database during installation process. Please make sure you've backed up your vCenter Server prior performing a migration.
Jozef
Novice
Posts: 8
Liked: never
Joined: Feb 23, 2011 1:17 pm
Full Name: Jozef Beres
Contact:

Re: vCenter keeps disconencting

Post by Jozef »

final solution -> cleanup vCenter database

more information:
http://kb.vmware.com/selfservice/micros ... Id=1025914
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests