vCenter keeps disconencting

[ngerasimatos]

Installed Veeam Monitor as a trial, we are currently testing it to see if it can provide the information, logging, and reporting we need for our two vCenter servers (environments) however, one of the two vCenter servers keeps disconnecting constantly and as such cannot pull enough data to use the reporting feature. The alert it says is regarding connectivity being lost.

Troubleshooting steps are below:

Verified the vCenter server and server housing Veeam Monitor are not lossing network connectivity (ping -t show no packets dropped, ever)

Checked database integrity of vCenter and Veeam.

Checked credentials for vCenter server.

Rebooted both Veeam monitor server and vCenter servers.

Any suggestions where to go from here?

[Vitaliy S.]

Hello Nicholas,

Do you have anything special with this very vCenter Server? What is the vCenter Server's version in question? Have you tried opening a MOB web page when you have a disconnected error message within a Veeam Monitor?

I would also suggest reviewing VeeamDCS.log to shed some light on this behaviour. Please note that if you need any assistance with this, don't hesitate to contact our technical team.

Thanks!

[ngerasimatos]

Where is the VeeamDCS.log located?

Version is 4.1
Build 258902

Not sure what a "MOB web page" is.

[Vitaliy S.]

To find VeeamDCS.log, please navigate to Options->Server Settings->Logs.

[ngerasimatos]

Can I email you the log?

[Gostev]

Hi Nicholas, for any technical issues you should open a support case and work with our support team directly. You are eligible for support during your official trial license period. Thanks!

[Jozef]

hi.

I've got the same problem. vCenter is disconnecting. Replication job ends with:

Checking "server name" host license level
Failed to login to "vCenter server name" by SOAP, port 443, user "vcenter\user name", proxy srv: port:0
Unable to connect to the remote server
No connection could be made because the target machine actively refused it 1xx.xx.xxx.xx:443

there should be a ticket regarding this: ID 557286

My configuration:
2 sites
5 ESX
2 veeam B&R server (on each site)
1 monitoring server
1 vCenter

I tought, that the problem was in vCenter database settings (by default, max. number of db connection on vcenter server is set to 10)
(in vSphere client go to Administration -> vCenter server settings -> Database)

I've changed it to 100, but it seems that it didn't help.

problems started after implementation of second B&R server.

Jozef

[Vitaliy S.]

Jozef, I doubt that the max number of db connections is the real cause for this issue. Do you have those connection drops while Veeam Monitor is running? or this issue can be reporduced just with a replication job? By the way, do you have any errors in the Windows Event log where vCenter server is installed?

[Jozef]

I've turned veeam Monitor server off, and vCenter is running fine for 5 days...

next step: reinstalling veeam Monitor server.

[Jozef]

bad news. after 5 days, error occurred again...

Checking "server name" host license level
Failed to login to "vcenter" by SOAP, port 443, user "username", proxy srv: port:0
Unable to connect to the remote server
No connection could be made because the target machine actively refused it 192.xxx.xxx.xxx:443

My next step will be reinstalling vCenter server.

[Vitaliy S.]

Jozef, before doing that I would suggest contacting VMware support guys to assist in vCenter Server log files investigation. Also reading back through my response above, have you had a chance to check Winows Event logs for vCenter Server?

[Jozef]

Vitaliy S -> there is nothing special in Windows event logs on vCenter server. Ticket to veeam support is already created.

[Vitaliy S.]

Jozef, I would appreciate if you could update this topic with your findings later on.

[Jozef]

Vitaliy S >
I really would like to understand, how is the communication made between Veeam B&R, Veeam Monitor <--> vCenter, esx hosts.

I can't reinstall vCenter server, so at this moment, I am just monitoring symptoms of the issue: (anyway, if I reinstall vCenter, I will have to do all my replicas again...and that is more than 3 TB datas via WAN, so I wont get approval from customer soon....)

1. when the Veeam monitor server is running + 2x Veeam B&R server, vCenter will refuse connection on https port 443 approx after 1 day (sometimes 2x per day, than restart of vCenter helps)

2. when the Veeam monitor is TURNED OFF, only 2x Veeam B&R servers are running, vCenter keeps working for about 5 days, but after, issue is the same (and again restart of vCenter)

first I tought, that allowing more connection to vCenter SQL database will helps, but I was wrong.

do you have any idea, how can I check, why https port on vcenter side is refused after some time?....and of course, maybe I am going wrong way with my investigation, so please advise

thanks

btw...in Veeamagent_srv.log I can see following information

[19.01.2011 06:11:06] < 2888> srv| Single-session server has been started.
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted
[19.01.2011 06:11:06] < 2888> srv| system:10048: Only one usage of each socket address (protocol/network address/port) is normally permitted

sometime 2 lines, sometimes 10, 15 lines of the same info....

[Vitaliy S.]

1. when the Veeam monitor server is running + 2x Veeam B&R server, vCenter will refuse connection on https port 443 approx after 1 day (sometimes 2x per day, than restart of vCenter helps)
2. when the Veeam monitor is TURNED OFF, only 2x Veeam B&R servers are running, vCenter keeps working for about 5 days, but after, issue is the same (and again restart of vCenter)

Sounds like the vCenter Server might be starving on the some resources (might be memory). Can you tell me the hardware configuration of your vCenter server?

do you have any idea, how can I check, why https port on vcenter side is refused after some time?....and of course, maybe I am going wrong way with my investigation, so please advise

The first thing that I would look at is vCenter Server log files. I think it might be even worth taking someone from VMware Support for assistance with this.

On top of that, don't be afraid to play with vCenter Server... what I would recommend to do though, is to backup its configuration database. So if you happen to re-install it, you will just need to point vCenter server to the backed up database, meaning that you won't lose your Inventory settings, VM IDs etc.

Thank you!

[gchapman]

Somewhat same issue here that has plauged us since we installed Veeam Monitor.

Veeam Monitor simply stops collecting data from the vCenter server daily leaving large gaps in historical analysis.

03/02/11 10:55:27 Error 0x0388 vcenter server "Unable to load/parse clients.xml: Unable to connect to the remote server"
03/02/11 10:55:27 Error 0x0388 vcenter server "XML Content:
(null)"
03/02/11 10:55:27 Error 0x0388 vcenter server "Collecting thread has failed to initialize (Unable to connect to the remote server) and will be stopped"

Troubleshooting this issue with support for almost 3 weeks now, and today recieved a message that it is an issue with ESXi and that they are working with VMWare to resolve the issue.

Suffice to say, I don't find that response all that encouraging, and right now the product simply does not perform as advertised. I would be curious to know if others with pure ESXi 4.1 environments are having similar issues.

[Vitaliy S.]

Gabriel, can you tell me your ticket number, so I could have a look?

[gchapman]

PM sent with ticket number.

Thanks

[Jozef]

The first thing that I would look at is vCenter Server log files. I think it might be even worth taking someone from VMware Support for assistance with this.

I already have logs from vCenter but that'ss more than 1GB of logs....can you please navigate me to the most important log files? thanks.

btw, I don't think, that vmware support will helps here because vCenter without veeam software is working fine.

[Vitaliy S.]

Jozef,

I believe there is an issue with the interaction between vCenter Server and 3rd party tools (not only Veeam). It might somehow be connected to scalability or configuration settings, so I do suggest reaching out VMware team.

As for the log files, I would have a look at vpxd*.log at first, then look through the most recent logs in the folder below:

Code: Select all

C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs

Thanks!

[Jozef]

I think that finaly, I've find out the issue.
MS SQL 2005 express offers only 4GB of database files (db+log),...and the DB on vCenter was about 4 GB....when DB is full, every new transaction is going to cash.
when cash is full, databse will just stop = vCenter can't run without access to DB.
After reboot, cash is free again, so vCenter works again until the cash is full. (in my case, 12 - 18 hours)

Now my question. Will vCenter works on MS SQL 2008 express? what is the correct upgrade path? I don't want to reinstall vCenter, just export/import database.
Anybody has experience?

thanks

[Vitaliy S.]

Thank you for this update, Jozef.

Based on the Release Notes for vCenter Server 4.1 u1, MS SQL 2008 R2 is fully supported.

I guess all you need to keep the current infrastructure is to migrate the database to the new SQL instance and point vCenter Server to that database during installation process. Please make sure you've backed up your vCenter Server prior performing a migration.

[Jozef]

final solution -> cleanup vCenter database

more information:
http://kb.vmware.com/selfservice/micros ... Id=1025914