Veeam B&R causes vCenter 6.5.0b SDK webservices to crash

VMware specific discussions

Re: Veeam B&R causes vCenter 6.5.0b SDK webservices to crash

Veeam Logoby stevenrodenburg1 » Thu Apr 06, 2017 8:28 pm

Update: my case has been escalated.
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Veeam B&R causes vCenter 6.5.0b SDK webservices to crash

Veeam Logoby Mike Resseler » Fri Apr 07, 2017 5:27 am

@Jordan: I think the others are right and this is another issue. But that doesn't matter, that needs to be solved also so keep working with support!

@Steven: Thanks for the update. I assume it will go to R&D now for further investigation
Mike Resseler
Veeam Software
 
Posts: 3161
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Veeam B&R causes vCenter 6.5.0b SDK webservices to crash

Veeam Logoby SysAdmAcc » Mon Apr 10, 2017 7:30 am 1 person likes this post

Just to inform that we also updated to the latest version of vCentre Applicane (6.5.0.5300 Build Number 5178943) and running Veeam 9.5.0.823 and have NO issues running backups.
SysAdmAcc
Lurker
 
Posts: 1
Liked: 1 time
Joined: Fri Nov 30, 2012 12:26 pm
Full Name: System Administrator

Re: Veeam B&R causes vCenter 6.5.0b SDK webservices to crash

Veeam Logoby stevenrodenburg1 » Sun Apr 16, 2017 7:26 pm

Update: Veeam found the cause of vCenter dying in vCenter's own logs. It's a SQL statement trying to insert a duplicate key in an object.

I replied this (snippet out of a larger email) to Support just now:
---------------------------------------------------
Support wrote:
“The error was caused by a primary key violation. Most probably removing of the row in question from table VPX_VM_VIRTUAL_DEVICE will resolve the issue.”

Me:
Those “cannot insert duplicate key row in object” errors are actually quite common.
If you simply google “vcenter cannot insert duplicate key row in object” you end up with a bunch of VMware KB articles. They talk about various versions of vCenter and about a wild variety of things causing such errors. None of which apply to this particular case by the way.

These errors only ever appear when Veeam talks to vCenter. I looked at the logs from the past week and these SQL errors do not appear. But as soon as Veeam starts talking because of backup-jobs (like today), I start seeing these SQL errors and vCenter dies.

My (Steven) conclusion so far: vCenter does not have these SQL errors when Veeam does not talk to it (is not running jobs). The vpxd.logs from the past week and today clearly prove this. As soon as I fired off those jobs today, the SQL errors started appearing.
I therefore have the feeling that Veeam is causing these SQL errors somehow. It could very well be a bug in vCenter 6.5.0b (not 6.5.0a) where, for example, Veeam fires off API calls to vCenter and vCenter messes it up doing incorrect SQL statements causing it’s own death shorty after."
-------------------------------------------------

I'm now waiting for the reply.
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: Veeam B&R causes vCenter 6.5.0b SDK webservices to crash

Veeam Logoby dellock6 » Mon Apr 17, 2017 8:17 am 1 person likes this post

Hi Steven,
I've seen the same errors on a newly created VCSA 6.5b and even without Veeam activity I've seen those errors happening, and ultimately crashing vcsa every hour. I ended up doing a backup and restore of the appliance, and since that I've seen no errors anymore. Also, regardless what "strange" call Veeam would do, the write operation is done by vCenter against its own database, so if there's any chance that's the root cause, actually it feels to me that VMware should fix that problem. Veeam only engages official vSphere API.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5052
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Previous

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 7 guests