vSphere vCenter Appliance - problems with service stopping

[GrZeCh]

Hello,

this might or might not be partially related to Veeam B&R but I would like to ask if someone is using vSphere vCenter Appliance with Veeam B&R? I've recently started testing B&R and noticed that one of my vCenter Appliances (I'm managing two VMWare Essentials vCenters) is having some problems. Few times during the day (especially backup hours) vCenter service is being stopped (https://IP:5480/#virtualcenter.status). Does someone encountered similar error and knows how to debug this situation? vCenter Web interface (IP:5480) is not providing any logs access and I don't know where to look when connected to vCenter Appliance using SSH (probably /var/log but where exactly there?).

Thank you

[dellock6]

Hi,
the service stops "especially" during backup hours or "only" during backup hours? Did you followed deployment requirements of the vCenter appliance for example? Usually it comes with 8Gb ram requirements but since it's used in smal environments I've seen many customers shrinking it to 4 GB, and sometimes this caused issues to the appliance itself running out of memory.

Luca.

[GrZeCh]

Hi,
the service stops "especially" during backup hours or "only" during backup hours? Did you followed deployment requirements of the vCenter appliance for example? Usually it comes with 8Gb ram requirements but since it's used in smal environments I've seen many customers shrinking it to 4 GB, and sometimes this caused issues to the appliance itself running out of memory.

Luca.

I would say especially because vSphere Appliance web interface is not informing when service has been stopped and I'm not refreshing it during whole backup process (service stopping is very random). After deploying vSphere I didn't changed anything except setting static IP address. I've had this problem one week ago so I've removed whole vCenter Appliance and deployed clean one from OVF file and this problem is occurring again. Second vCenter is not having this kind of problems (has less VMs).

[dellock6]

A quick search retunrned me this link:
http://www.r71.nl/kb/technical/325-moni ... e-via-snmp
maybe you can give it a try to check when services goes offline.

Luca.

[GrZeCh]

I think I'm having similar problem like someone here:

http://communities.vmware.com/thread/403862

my

/dev/sdb1 20G 173M 19G 1% /storage/core

is being filled on one of my vCenters. I've cleaned it but previously I had 75% used. It is not 100% like someone from vmware community forum but probably something similar. If I will still start to experience problems with vCenter Appliance connectivity I will probably try to create some NFS storage just like someone described here:

http://vmwire.com/tag/vcenter-server-virtual-appliance/

Now my qustion is can I run NFS storage on the same windows system where I'm having Veeam B&R (with proxy and repositories)?

[Vitaliy S.]

If you're referring to Windows NFS Server, then these servers cannot co-exist on the same machine, as both servers (Windows and vPower NFS) are using the same port. You way want to use vPower NFS Server as a backend storage for vCSA log files. Thanks!

[GrZeCh]

Good idea but what NFS export I should provide? IP:/REPOSITORYNAME ?

[GrZeCh]

On the other hand I can confirm that my vCenter service has been stopped in moment when I've started backup job which uses two vCenter as a containers.

Entry from /var/log/messages:

Code: Select all

Jul  2 23:10:01 localhost /usr/sbin/cron[25149]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
Jul  2 23:10:01 localhost /usr/sbin/cron[25150]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

in /var/log/warn I have only

Code: Select all

Jul  2 23:12:57 localhost slapd[5958]: warning: /etc/hosts.allow, line 65: host name mismatch: localhost.localdom != (null) (::ffff:127.0.0.1)

/var/log/vmware/vpx/vpxd.log

Code: Select all

2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [Init:838] Calling: VpxCryptInit()
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [Init:839] Calling: VpxLRO::Init()
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [Init:840] Calling: VpxLroList::Init(ltud)
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 640 max LROs
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved internal LROs
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved blocker LROs
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved short LROs
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 8 reserved long LROs
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second completed task lifetime
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 60-second minimum completed task lifetime
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 200 maximum completed tasks
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second unregistered task lifetime
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [Init:841] Calling: VpxdCharacterizeThreadpool(ltud)
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'ThreadPool'] [SetConfig] Thread info: IoMin: 2, IoMax: 1990, TaskMin: 59, TaskMax: 1990, Max Thread: 3980, Keepalive: 61, thread kill: 600, max fds: -1
2012-07-02T22:36:04.108+02:00 [7FFFF3B09700 info 'Default'] [Init:842] Calling: VpxdCertificate_Load(gDB, CERTIFICATE_VMDBPATH )
2012-07-02T22:36:04.109+02:00 [7FFFF3B09700 info 'Default'] [Init:843] Calling: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
2012-07-02T22:36:04.110+02:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''
2012-07-02T22:36:05.400+02:00 [7FFFF3B09700 error 'Default'] [Vdb::LockRepositoryHelper] SQLExecDirect failed: 57011:-438:ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0438N  Application raised error or warning with diagnostic text: "Error deleting from VPX_SESSIONL".  SQLSTATE=57011
-->
2012-07-02T22:36:05.401+02:00 [7FFFF3B09700 error 'Default'] Unable to get exclusive access to vCenter repository.                           Please check if another vCenter instance is                           running against the same database schema.
2012-07-02T22:36:05.409+02:00 [7FFFF3B09700 error 'Default'] [Init] Init failed: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
2012-07-02T22:36:05.409+02:00 [7FFFF3B09700 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
2012-07-02T22:36:05.409+02:00 [7FFFF3B09700 info 'Default'] Wrote uptime information
2012-07-02T22:36:05.409+02:00 [7FFFF3B09700 error 'Default'] Alert:false@ /build/mts/release/bora-455964/bora/vpx/vpxd/util/vpxdVdb.cpp:408
--> Backtrace:
--> [00] rip 00007fffeed9ed14
--> [01] rip 00007fffeecf59e2
--> [02] rip 00007fffeec9b38b
--> [03] rip 00007ffff58c613d
--> [04] rip 00007ffff4ea5a40
--> [05] rip 00007ffff4eb4e21
--> [06] rip 00007ffff4ea2953
--> [07] rip 00007fffed03abc6
--> [08] rip 00007ffff4ea1c19
-->
2012-07-02T22:36:05.410+02:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''
2012-07-02T22:36:06.600+02:00 [7FFFF3B09700 error 'Default'] [Vdb::LockRepositoryHelper] SQLExecDirect failed: 57011:-438:ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0438N  Application raised error or warning with diagnostic text: "Error deleting from VPX_SESSIONL".  SQLSTATE=57011
-->
2012-07-02T22:36:06.600+02:00 [7FFFF3B09700 error 'Default'] Unable to get exclusive access to vCenter repository.                           Please check if another vCenter instance is                           running against the same database schema.
2012-07-02T22:36:06.607+02:00 [7FFFF3B09700 error 'Default'] Unhandled exception <Error[VdbError] (-24) Vdb instance is not initialized or terminated>
^C

Code: Select all

localhost:/var/log # service vmware-vpxd status
vmware-vpxd is stopped
tomcat is running
localhost:/var/log # service vmware-vpxd start
Waiting for embedded DB2 database to startup: .success
Cleaning session lock table: success
Verifying EULA acceptance: success
Shutting down ldap-server..done
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: ..success
Starting tomcat: (tomcat is already running) success
Executing startup scripts...

[Vitaliy S.]

Good idea but what NFS export I should provide? IP:/REPOSITORYNAME ?

I believe it should be just a hostname of the Veeam repository server.

On top of that, looks like there is a VMware KB on the issue you describe, at least it mentions the same error message as you've posted above.

[GrZeCh]

I believe it should be just a hostname of the Veeam repository server.

On top of that, looks like there is a VMware KB on the issue you describe, at least it mentions the same error message as you've posted above.

I think this is not a problem with free space because my vCenter again has been stopped and disk space:

/dev/sdb1 20G 1.4G 18G 8% /storage/core

usage is only at 8%. Regarding VMware KB article it is for vCenter installed on Windows system using Oracle DB. I'm using vCenter Appliance where I'm using embedded DB (since one vCenter is only managing 3 ESXi servers).

[GrZeCh]

I think I've found my problem.

/var/log/vmware/vpx/vpxd.log

Code: Select all

2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] [VdbStatement] Diagnostic data from driver is 57011:1:-964:[IBM][CLI Driver][DB2/LINUXX8664] SQL0964C  The transaction log for the database is full.  SQLSTATE=57011
-->
2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] [VdbStatement] Diagnostic data from driver is 01504:2:513:[IBM][CLI Driver][DB2/LINUXX8664] SQL0513W  The SQL statement will modify an entire table or view.  SQLSTATE=01504
-->
2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] [VdbStatement] Bind parameters:
2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] VpxdStatsCollector::ClearTempTable failed: "ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0964C  The transaction log for the database is full.  SQLSTATE=57011
--> " is returned when executing SQL statement "delete from VPX_TEMPTABLE0"
2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] VpxdStatsCollector::ProcessTempTable Failed to process temp table. The table can't be returned to insert further perf data ("ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0964C  The transaction log for the database is full.  SQLSTATE=57011
--> " is returned when executing SQL statement "call process_temptable0_proc")
2012-07-03T00:33:57.909+02:00 [7FFFDF37C710 error 'Default' opID=SWI-64f33f78] Alert:false@ /build/mts/release/bora-455964/bora/vpx/vpxd/stats/vpxdStatsCollector.cpp:193

Maybe fix from this blog entry will help:

http://www.virtualserverguy.com/blog/20 ... abase.html

[GrZeCh]

I just want to let you know that changing DB settings just like presented on http://www.virtualserverguy.com/blog/20 ... abase.html blog probably fixed my problem. From moment when I've changed DB settings I didn't lost connection to vCenter.

[Vitaliy S.]

Thanks for coming back and updating the topic.

[GrZeCh]

No problem. Now when vCenter service is not stopping also /storage/core partition on vCenter Appliance is not being filled too.