Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
Post Reply
tntteam
Enthusiast
Posts: 63
Liked: 5 times
Joined: Aug 28, 2015 12:40 pm
Full Name: tntteam

Veeam-one bug with vsphere 5.1

Post by tntteam » Feb 23, 2016 1:35 pm

Hi there,

I have no time atm to open a SR and send logs etc my priority was getting production VMs back online. I post this here just for information and maybe save some time to people like me.

I'm using HP blades with latest SPP and vmware 5.1 with latest patches and veeam-one v9. I don't know if this can happen on other versions.

I think it's a very RARE bug, I have no idea what could have caused this. Please refer to my post (sharantyr3) here : https://communities.vmware.com/message/2577347#2577346

The vmotion of veeam-one VM to another host allowed ESXi to remove the faulty locks on the VM files (validated).

So just to tell you, if by any chance you get struck VMs and see entries like this in vmkwarning.log :
cpu16:8252)WARNING: Swap: vm 9930238: 4780: Failed to unlink /vmfs/volumes/55533920-9ee23892-fbf5-2c768aae2b42/maria-db-01/vmx-maria-db-01-794194787-1.vswp: Maximum kernel-level retries exceeded
and entries like this in vmkernel.log :
2016-02-23T12:53:51.161Z cpu4:9937646)DLX: 4230: vol 'BJ-ISN-06', lock at 43160576: [Req mode: 1] Not free:
2016-02-23T12:53:51.161Z cpu4:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
2016-02-23T12:53:51.161Z cpu4:9937646)Res3: 5732: Rank violation threshold reached: cid 0xc1d00002, resType 4, cnum 1
2016-02-23T12:53:53.760Z cpu8:9937646)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:
2016-02-23T12:53:53.760Z cpu8:9937646)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
Simply try to vmotion your veeam-one server to another ESXi. After the vmotion, I could successfully remove the old VM file and got this in the ESXi logs :
2016-02-23T13:09:43.259Z cpu26:5686842)DLX: 3706: vol 'BJ-ISN-06', lock at 43160576: [Req mode 1] Checking liveness:
2016-02-23T13:09:43.259Z cpu26:5686842)[type 10c00005 offset 43160576 v 78794, hb offset 3624960
gen 1087, mode 1, owner 56b5cf5d-f208f93a-4755-3c4a926c279c mtime 171709
num 0 gblnum 0 gblgen 0 gblbrk 0]
2016-02-23T13:09:43.260Z cpu26:5686842)DLX: 3321: Clearing wrong owner for lock at 43160576 with [HB state abcdef01 offset 3624960 gen 1088 stampUS 1480954430581 uuid 00000000-00000000-0000-000000000000 jrnl <FB 0> drv 14.58]

P.Tide
Product Manager
Posts: 5228
Liked: 450 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam-one bug with vsphere 5.1

Post by P.Tide » Feb 23, 2016 1:51 pm

Hi,

Thank you for sharing your experience!

I followed the link you've provided and also checked the support system and discovered that another user had a ticket opened with our support team on the same issue, however our techs did not have a chance to conduct a deeper investigation because the customer had not uploaded all required logs. I will pass this info directly to QA so they can try to reproduce the issue. Also I'd really appreciate if you opened a case with our support team when you have a spare minute as we really need your logs.

Thank you in advance.

tntteam
Enthusiast
Posts: 63
Liked: 5 times
Joined: Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Veeam-one bug with vsphere 5.1

Post by tntteam » Feb 23, 2016 2:38 pm

I'll try to do it this week or next.

Also I think I am faulty here too, as veeam-one is only on the testing phase and I did use a domain account with admin rights on vcenter. Right now I switched my veeam-one service account to a read only account on vcenter, I think this will help.

Can you tell me what reference to put in my SR so the support team will link it as you did ?

Thanks

P.Tide
Product Manager
Posts: 5228
Liked: 450 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam-one bug with vsphere 5.1

Post by P.Tide » Feb 23, 2016 3:02 pm

Can you tell me what reference to put in my SR so the support team will link it as you did ?
Another user has opened case # 00503976

Thank you.

Vitaliy S.
Product Manager
Posts: 22868
Liked: 1540 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam-one bug with vsphere 5.1

Post by Vitaliy S. » Feb 24, 2016 11:17 am

Hi tntteam,

Thank you for posting your experience. Did you try to power off Veeam ONE VM before migrating it? Did you see any related events in the event log of the vCenter Server?

Thanks!

tntteam
Enthusiast
Posts: 63
Liked: 5 times
Joined: Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Veeam-one bug with vsphere 5.1

Post by tntteam » Feb 29, 2016 11:05 am

Hi,

No I didn't try to power off before migrate. I'm not a fan of vcenter event log as it is way too much verbose. But I can upload them.

I'm currently exchanging with the support team via emails :)

Vitaliy S.
Product Manager
Posts: 22868
Liked: 1540 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam-one bug with vsphere 5.1

Post by Vitaliy S. » Feb 29, 2016 11:24 am

Ok, can you please let us know your case ID?

tntteam
Enthusiast
Posts: 63
Liked: 5 times
Joined: Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Veeam-one bug with vsphere 5.1

Post by tntteam » Mar 02, 2016 10:57 am 1 person likes this post

Hi,

It's 01707242

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest