Veeam-one bug with vsphere 5.1

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Veeam-one bug with vsphere 5.1

Veeam Logoby tntteam » Tue Feb 23, 2016 1:35 pm

Hi there,

I have no time atm to open a SR and send logs etc my priority was getting production VMs back online. I post this here just for information and maybe save some time to people like me.

I'm using HP blades with latest SPP and vmware 5.1 with latest patches and veeam-one v9. I don't know if this can happen on other versions.

I think it's a very RARE bug, I have no idea what could have caused this. Please refer to my post (sharantyr3) here : https://communities.vmware.com/message/2577347#2577346

The vmotion of veeam-one VM to another host allowed ESXi to remove the faulty locks on the VM files (validated).

So just to tell you, if by any chance you get struck VMs and see entries like this in vmkwarning.log :
cpu16:8252)WARNING: Swap: vm 9930238: 4780: Failed to unlink /vmfs/volumes/55533920-9ee23892-fbf5-2c768aae2b42/maria-db-01/vmx-maria-db-01-794194787-1.vswp: Maximum kernel-level retries exceeded

and entries like this in vmkernel.log :
2016-02-23T12:53:51.161Z cpu4:9937646)DLX: 4230: vol 'BJ-ISN-06', lock at 43160576: [Req mode: 1] Not free:
2016-02-23T12:53:51.161Z cpu4:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
2016-02-23T12:53:51.161Z cpu4:9937646)Res3: 5732: Rank violation threshold reached: cid 0xc1d00002, resType 4, cnum 1
2016-02-23T12:53:53.760Z cpu8:9937646)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:
2016-02-23T12:53:53.760Z cpu8:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]


Simply try to vmotion your veeam-one server to another ESXi. After the vmotion, I could successfully remove the old VM file and got this in the ESXi logs :

2016-02-23T13:09:43.259Z cpu26:5686842)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:
2016-02-23T13:09:43.259Z cpu26:5686842)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
2016-02-23T13:09:43.260Z cpu26:5686842)DLX: 3321: Clearing wrong owner for lock at 43160576 with [HB state abcdef01 offset 3624960 gen 1088 stampUS 1480954430581 uuid 00000000-00000000-0000-000000000000 jrnl <FB 0> drv 14.58]
tntteam
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Fri Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby PTide » Tue Feb 23, 2016 1:51 pm

Hi,

Thank you for sharing your experience!

I followed the link you've provided and also checked the support system and discovered that another user had a ticket opened with our support team on the same issue, however our techs did not have a chance to conduct a deeper investigation because the customer had not uploaded all required logs. I will pass this info directly to QA so they can try to reproduce the issue. Also I'd really appreciate if you opened a case with our support team when you have a spare minute as we really need your logs.

Thank you in advance.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby tntteam » Tue Feb 23, 2016 2:38 pm

I'll try to do it this week or next.

Also I think I am faulty here too, as veeam-one is only on the testing phase and I did use a domain account with admin rights on vcenter. Right now I switched my veeam-one service account to a read only account on vcenter, I think this will help.

Can you tell me what reference to put in my SR so the support team will link it as you did ?

Thanks
tntteam
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Fri Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby PTide » Tue Feb 23, 2016 3:02 pm

Can you tell me what reference to put in my SR so the support team will link it as you did ?
Another user has opened case # 00503976

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby Vitaliy S. » Wed Feb 24, 2016 11:17 am

Hi tntteam,

Thank you for posting your experience. Did you try to power off Veeam ONE VM before migrating it? Did you see any related events in the event log of the vCenter Server?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19554
Liked: 1100 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby tntteam » Mon Feb 29, 2016 11:05 am

Hi,

No I didn't try to power off before migrate. I'm not a fan of vcenter event log as it is way too much verbose. But I can upload them.

I'm currently exchanging with the support team via emails :)
tntteam
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Fri Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby Vitaliy S. » Mon Feb 29, 2016 11:24 am

Ok, can you please let us know your case ID?
Vitaliy S.
Veeam Software
 
Posts: 19554
Liked: 1100 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Veeam-one bug with vsphere 5.1

Veeam Logoby tntteam » Wed Mar 02, 2016 10:57 am 1 person likes this post

Hi,

It's 01707242
tntteam
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Fri Aug 28, 2015 12:40 pm
Full Name: tntteam


Return to Veeam ONE



Who is online

Users browsing this forum: No registered users and 4 guests