Discussions specific to the VMware vSphere hypervisor
Post Reply
euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

HotAdd Test fails per KB

Post by euscada » Jan 14, 2019 4:29 pm

******-------****** Case #03359726 was created. ******-------******


Hi all,

We have been attempting to setup HotAdd mode in our environment and it appears we can not pass the test per this KB article from Veeam --

https://www.veeam.com/kb1184

When we attempt this test, we recieve the following error from vSphere --
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
Failed to lock the file
Cannot open the disk '/vmfs/volumes/94c7b227-c3a52d4b/DCVD-00114/DCVD-00114.vmdk' or one of the snapshot disks it depends on.
Failed to power on scsi0:1.
Failed to add disk scsi0:1.
He have been attempting to add a backup proxy to speed up backups, but it actually has increased our backup times and causes disruptions to our large VMs when removing the snapshot.

When we disable the proxy, the backup times decrease (are faster) and the snapshot removal is not an issue.

Can anyone shed some light on this issue?

Veeam B&R 8.0.0.2084
Win 2008 R2
4cpu 6GB ram

Proxy
Win 2008 r2
4cpu 6GB ram

Using B&R to backup 25ish VMs

More infrastructure specs upon request

Thanks

P.Tide
Product Manager
Posts: 5321
Liked: 468 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: HotAdd Test fails per KB

Post by P.Tide » Jan 14, 2019 4:45 pm

Hi,

How many hosts do you have and what kind of production storage do you use? Is it NFS by any chance?

UPDATE: Please update your VBR server to the latest version and see if the issue persists. The version that you use now is already out of support.

Thanks!

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 14, 2019 5:19 pm

Hi and thanks for the reply

We have 3 hosts and the storage is NFS

I understand that the version we are using is out of support, we are just cautious when doing upgrade on our production servers.

Is there any other advice besides updating our version of Veeam?

If not, where do i download the required version and what version is recommended?

Thanks

P.Tide
Product Manager
Posts: 5321
Liked: 468 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: HotAdd Test fails per KB

Post by P.Tide » Jan 14, 2019 5:37 pm 1 person likes this post

Well, you could check this KB article and double-check if the infrastructure meets the requirements. Also you can investigate what is locking the file.

You can find upgrade instructions here:

- User Guide
- Knowledge Base

Thanks!

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 14, 2019 5:54 pm

It appears our infrastructure supports the requirements.

What is strange is Veeam does not state "Failing over to network mode" so I believe it is functional via Veeam, it is just very slow (doubling backup times) and causes ICMP alarms/errors (can't be pinged for > 1min) on our large vms when removing the snapshots. (vms with multiple large disks // > 100Gigs)

I did see this vmware kb article, so I am confused on how Veeam actually preforms the "hotadd"...Maybe I need to enable scsi bus sharing on all my vms and set the setting to "Physical" (Virtual disks can be shared between virtuals machines on any server)?

https://kb.vmware.com/s/article/2001217

Thanks again

P.Tide
Product Manager
Posts: 5321
Liked: 468 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: HotAdd Test fails per KB

Post by P.Tide » Jan 14, 2019 6:02 pm

File is being locked by a consumer on host B with exclusive lock.
Since the message basically means that the vmdk file is locked, I'd suggest to start with figuring out what locks the file.

Maybe I need to enable scsi bus sharing on all my vms and set the setting to "Physical" (Virtual disks can be shared between virtuals machines on any server)?
Definitely no, you should not do that - VMware does not support snapshots of virtual machines configured for bus sharing.

Thanks!

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 14, 2019 9:27 pm

Interestingly enough, all of our VMs are locked? (.lck files on every vm)

I know this isn't a vmware forum, but have you guys seen this behavior?

Thanks again

Andreas Neufert
Veeam Software
Posts: 3921
Liked: 707 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HotAdd Test fails per KB

Post by Andreas Neufert » Jan 15, 2019 10:23 am

Maybe space related? How much free space do you have on the datastore?

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 15, 2019 1:40 pm

Thanks for the reply

We have 8TB free space on our NFS

I followed the vmware KB article for looking into .lck files, and nothing stood out to me as "why the file is locked"...

I did see that the storage volume was called out twice, once as the UUID and once as the "plain text" name you can see in vSphere client when running
# egrep -i DISKNAME.vmdk /vmfs/volumes/*/*/*.vmx
Not sure if vCenter "thinks" there are 2 of everything because of this..

I'll reach out to vmware and see if they can investigate the lck files.

Side note, we have 3 infrastructure locations, all with the same hardware and base config (3 separate facilities that function similarly) and all 3 experience the same lck files across the entire system.

All 3 systems also run veeam...

Thanks again

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 15, 2019 11:33 pm

The more I dig into this, the more confused I get.

Per this vmware kb, the lck (lock file) is always there when a VM is powered on - https://kb.vmware.com/s/article/1003857?lang=en_US

Maybe I am getting vmware "locked" file terminology crossed with veeam "locked" file terminology?

It appears from a vmware side of things, the "lck" file is ALWAYS there when a vm is powered on. So it appears to be a veeam config issue?

Is there a check list of items for HotAdd config? I reviewed the requirements link, and it appears we are fine in that sense.

Here are all 8 items from that link --

1. We are not using a VMFS 3 datastore (We are using NFS)
2. The proxy is a vm on a host that can access the datastore
3. We are not running 5.1
4. We are running 5.5 and do not have any vms that come close to 62 TB
5. We dont have any IDE stuff
6. VMware tools is installed and up to date on every machine
7. We have a 10TB data store with 8TB of free space
8. We are not on 6.0 or using SATA devices

Any additional input is appreciated

Thanks

More infrastructure notes (one of our sites, we have 3 similar)

ESXi 5.5 P10 – 4722766
10TB NFS datastore with 8TB free space (Shows up as Type: NAS)
3 host cluster (Nutanix 72 logical cpu, 384GB ram)
vCenter 5.5
HA datastore HB enabled
~30 VMs

foggy
Veeam Software
Posts: 18390
Liked: 1580 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HotAdd Test fails per KB

Post by foggy » Jan 16, 2019 12:50 pm

Are you positive that the correct proxy (the one residing on the same host where VM resides) is always selected? There's a registry value to address NFS locking issues during hotadd.

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 16, 2019 2:16 pm

Hi foggy thanks for the reply

So new update --

Veeam "hotadd" jobs do not fail (as stated in my 3rd post above)

The original reason for me going down this rabbit hole of troubleshooting was to find out why the new proxy is in essence doubling job times so I figured reading kbs and trying the steps would help.

I still can not "hotadd" vms in the esx side of things, so it is still somewhat confusing.

I am working with a veeam support rep behind the scenes from here (forum) so I will keep this updated as I progress.

Thanks

foggy
Veeam Software
Posts: 18390
Liked: 1580 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HotAdd Test fails per KB

Post by foggy » Jan 16, 2019 2:24 pm

The original reason for me going down this rabbit hole of troubleshooting was to find out why the new proxy is in essence doubling job times so I figured reading kbs and trying the steps would help.
What transport mode is being utilized by the jobs? What are the bottleneck stats they report?
I still can not "hotadd" vms in the esx side of things, so it is still somewhat confusing.
Do I understand right, disks can be hotadded when the job is running but you cannot hotadd them manually, using vSphere Client?

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 16, 2019 2:45 pm

Transport mode (when testing the hotadd features) is configured to use "Virtual Appliance" with the "Failover to network mode" unchecked.
Do I understand right, disks can be hotadded when the job is running but you cannot hotadd them manually, using vSphere Client?
This is correct, I can not run the veeam kb for manually testing hotadd, the adding of the hdd fails -- but when veeam runs the job, I can see hotadd is functional and is working (i can open the settings of the proxy vm and see that the drive is now connected to that vm...also the "Action" window of veeam states -- "Using backup proxy xxx for disk Hard disk 1 [hotadd]"

Running a test of 1 vm configured for hotadd using a proxy forced to virtual appliance mode and forced to use the proxy takes 21 mins
--Bottleneck is stated to be the source at 94% / proxy 22% / Network 14% / Target 11%

Same job with no proxy and using network mode takes 8 mins
--Bottleneck is stated to be the source at 99% / proxy 32% / Network 6% / Target 0%

Thanks for all the input

foggy
Veeam Software
Posts: 18390
Liked: 1580 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HotAdd Test fails per KB

Post by foggy » Jan 16, 2019 2:53 pm 1 person likes this post

Ok, please continue working with support engineers, they should be able to detect the actual phase that takes long.

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Jan 16, 2019 4:35 pm 1 person likes this post

Turns out our config of "Network mode" is recommended over "Virtual Appliance Mode" due to our 10Gb network setup.

Snapshot stunning appears to be more prevalent with the hotadd setup so we will continue to use Network mode.

No real improvement/gain for us to use the hotadd/extra proxy features.

:roll:

foggy
Veeam Software
Posts: 18390
Liked: 1580 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HotAdd Test fails per KB

Post by foggy » Jan 16, 2019 4:39 pm 1 person likes this post

Yes, network mode on 10Gb should rock. ;)

euscada
Influencer
Posts: 10
Liked: 2 times
Joined: Jan 14, 2019 2:22 pm
Full Name: EU SCADA
Contact:

Re: HotAdd Test fails per KB

Post by euscada » Feb 13, 2019 11:14 pm 1 person likes this post

Hi all,

Just to update this thread for anyone curious about this issue...

Not necessarily the "HotAdd Test Fail", but issues with overall speed with our Nutanix infrastructure and Veeam, please read this article.

https://www.veeam.com/nutanix-veeam-bac ... es_wpp.pdf

This was for some reason not noticed by Veeam techs, I had to pursue support via Nutanix.

Hope this helps anyone looking for issues utilizing Veeam with a Nutanix device.

Thanks

Post Reply

Who is online

Users browsing this forum: Bing [Bot], marcottt and 10 guests