Veeam-one bug with vsphere 5.1

[tntteam]

Hi there,

I have no time atm to open a SR and send logs etc my priority was getting production VMs back online. I post this here just for information and maybe save some time to people like me.

I'm using HP blades with latest SPP and vmware 5.1 with latest patches and veeam-one v9. I don't know if this can happen on other versions.

I think it's a very RARE bug, I have no idea what could have caused this. Please refer to my post (sharantyr3) here : https://communities.vmware.com/message/2577347#2577346

The vmotion of veeam-one VM to another host allowed ESXi to remove the faulty locks on the VM files (validated).

So just to tell you, if by any chance you get struck VMs and see entries like this in vmkwarning.log :

cpu16:8252)WARNING: Swap: vm 9930238: 4780: Failed to unlink /vmfs/volumes/55533920-9ee23892-fbf5-2c768aae2b42/maria-db-01/vmx-maria-db-01-794194787-1.vswp: Maximum kernel-level retries exceeded

and entries like this in vmkernel.log :

2016-02-23T12:53:51.161Z cpu4:9937646)DLX: 4230: vol 'BJ-ISN-06', lock at 43160576: [Req mode: 1] Not free:
2016-02-23T12:53:51.161Z cpu4:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
2016-02-23T12:53:51.161Z cpu4:9937646)Res3: 5732: Rank violation threshold reached: cid 0xc1d00002, resType 4, cnum 1
2016-02-23T12:53:53.760Z cpu8:9937646)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:
2016-02-23T12:53:53.760Z cpu8:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]

Simply try to vmotion your veeam-one server to another ESXi. After the vmotion, I could successfully remove the old VM file and got this in the ESXi logs :

2016-02-23T13:09:43.259Z cpu26:5686842)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:
2016-02-23T13:09:43.259Z cpu26:5686842)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
2016-02-23T13:09:43.260Z cpu26:5686842)DLX: 3321: Clearing wrong owner for lock at 43160576 with [HB state abcdef01 offset 3624960 gen 1088 stampUS 1480954430581 uuid 00000000-00000000-0000-000000000000 jrnl <FB 0> drv 14.58]

[PTide]

Hi,

Thank you for sharing your experience!

I followed the link you've provided and also checked the support system and discovered that another user had a ticket opened with our support team on the same issue, however our techs did not have a chance to conduct a deeper investigation because the customer had not uploaded all required logs. I will pass this info directly to QA so they can try to reproduce the issue. Also I'd really appreciate if you opened a case with our support team when you have a spare minute as we really need your logs.

Thank you in advance.

[tntteam]

I'll try to do it this week or next.

Also I think I am faulty here too, as veeam-one is only on the testing phase and I did use a domain account with admin rights on vcenter. Right now I switched my veeam-one service account to a read only account on vcenter, I think this will help.

Can you tell me what reference to put in my SR so the support team will link it as you did ?

Thanks

[PTide]

Can you tell me what reference to put in my SR so the support team will link it as you did ?

Another user has opened case # 00503976

Thank you.

[Vitaliy S.]

Hi tntteam,

Thank you for posting your experience. Did you try to power off Veeam ONE VM before migrating it? Did you see any related events in the event log of the vCenter Server?

Thanks!

[tntteam]

Hi,

No I didn't try to power off before migrate. I'm not a fan of vcenter event log as it is way too much verbose. But I can upload them.

I'm currently exchanging with the support team via emails

[Vitaliy S.]

Ok, can you please let us know your case ID?

[tntteam]

Hi,

It's 01707242