HotAdd Test fails per KB

[euscada]

******-------****** Case #03359726 was created. ******-------******


Hi all,

We have been attempting to setup HotAdd mode in our environment and it appears we can not pass the test per this KB article from Veeam --

https://www.veeam.com/kb1184

When we attempt this test, we recieve the following error from vSphere --

File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
File is being locked by a consumer on host B with exclusive lock.
Failed to lock the file
Cannot open the disk '/vmfs/volumes/94c7b227-c3a52d4b/DCVD-00114/DCVD-00114.vmdk' or one of the snapshot disks it depends on.
Failed to power on scsi0:1.
Failed to add disk scsi0:1.

He have been attempting to add a backup proxy to speed up backups, but it actually has increased our backup times and causes disruptions to our large VMs when removing the snapshot.

When we disable the proxy, the backup times decrease (are faster) and the snapshot removal is not an issue.

Can anyone shed some light on this issue?

Veeam B&R 8.0.0.2084
Win 2008 R2
4cpu 6GB ram

Proxy
Win 2008 r2
4cpu 6GB ram

Using B&R to backup 25ish VMs

More infrastructure specs upon request

Thanks

[PTide]

Hi,

How many hosts do you have and what kind of production storage do you use? Is it NFS by any chance?

UPDATE: Please update your VBR server to the latest version and see if the issue persists. The version that you use now is already out of support.

Thanks!

[euscada]

Hi and thanks for the reply

We have 3 hosts and the storage is NFS

I understand that the version we are using is out of support, we are just cautious when doing upgrade on our production servers.

Is there any other advice besides updating our version of Veeam?

If not, where do i download the required version and what version is recommended?

Thanks

[PTide]

Well, you could check this KB article and double-check if the infrastructure meets the requirements. Also you can investigate what is locking the file.

You can find upgrade instructions here:

- User Guide
- Knowledge Base

Thanks!

[euscada]

It appears our infrastructure supports the requirements.

What is strange is Veeam does not state "Failing over to network mode" so I believe it is functional via Veeam, it is just very slow (doubling backup times) and causes ICMP alarms/errors (can't be pinged for > 1min) on our large vms when removing the snapshots. (vms with multiple large disks // > 100Gigs)

I did see this vmware kb article, so I am confused on how Veeam actually preforms the "hotadd"...Maybe I need to enable scsi bus sharing on all my vms and set the setting to "Physical" (Virtual disks can be shared between virtuals machines on any server)?

https://kb.vmware.com/s/article/2001217

Thanks again

[PTide]

File is being locked by a consumer on host B with exclusive lock.

Since the message basically means that the vmdk file is locked, I'd suggest to start with figuring out what locks the file.

Maybe I need to enable scsi bus sharing on all my vms and set the setting to "Physical" (Virtual disks can be shared between virtuals machines on any server)?

Definitely no, you should not do that - VMware does not support snapshots of virtual machines configured for bus sharing.

Thanks!

[euscada]

Interestingly enough, all of our VMs are locked? (.lck files on every vm)

I know this isn't a vmware forum, but have you guys seen this behavior?

Thanks again

[Andreas Neufert]

Maybe space related? How much free space do you have on the datastore?

[euscada]

Thanks for the reply

We have 8TB free space on our NFS

I followed the vmware KB article for looking into .lck files, and nothing stood out to me as "why the file is locked"...

I did see that the storage volume was called out twice, once as the UUID and once as the "plain text" name you can see in vSphere client when running

# egrep -i DISKNAME.vmdk /vmfs/volumes/*/*/*.vmx

Not sure if vCenter "thinks" there are 2 of everything because of this..

I'll reach out to vmware and see if they can investigate the lck files.

Side note, we have 3 infrastructure locations, all with the same hardware and base config (3 separate facilities that function similarly) and all 3 experience the same lck files across the entire system.

All 3 systems also run veeam...

Thanks again

[euscada]

The more I dig into this, the more confused I get.

Per this vmware kb, the lck (lock file) is always there when a VM is powered on - https://kb.vmware.com/s/article/1003857?lang=en_US

Maybe I am getting vmware "locked" file terminology crossed with veeam "locked" file terminology?

It appears from a vmware side of things, the "lck" file is ALWAYS there when a vm is powered on. So it appears to be a veeam config issue?

Is there a check list of items for HotAdd config? I reviewed the requirements link, and it appears we are fine in that sense.

Here are all 8 items from that link --

1. We are not using a VMFS 3 datastore (We are using NFS)
2. The proxy is a vm on a host that can access the datastore
3. We are not running 5.1
4. We are running 5.5 and do not have any vms that come close to 62 TB
5. We dont have any IDE stuff
6. VMware tools is installed and up to date on every machine
7. We have a 10TB data store with 8TB of free space
8. We are not on 6.0 or using SATA devices

Any additional input is appreciated

Thanks

More infrastructure notes (one of our sites, we have 3 similar)

ESXi 5.5 P10 – 4722766
10TB NFS datastore with 8TB free space (Shows up as Type: NAS)
3 host cluster (Nutanix 72 logical cpu, 384GB ram)
vCenter 5.5
HA datastore HB enabled
~30 VMs

[foggy]

Are you positive that the correct proxy (the one residing on the same host where VM resides) is always selected? There's a registry value to address NFS locking issues during hotadd.

[euscada]

Hi foggy thanks for the reply

So new update --

Veeam "hotadd" jobs do not fail (as stated in my 3rd post above)

The original reason for me going down this rabbit hole of troubleshooting was to find out why the new proxy is in essence doubling job times so I figured reading kbs and trying the steps would help.

I still can not "hotadd" vms in the esx side of things, so it is still somewhat confusing.

I am working with a veeam support rep behind the scenes from here (forum) so I will keep this updated as I progress.

Thanks

[foggy]

The original reason for me going down this rabbit hole of troubleshooting was to find out why the new proxy is in essence doubling job times so I figured reading kbs and trying the steps would help.

What transport mode is being utilized by the jobs? What are the bottleneck stats they report?

I still can not "hotadd" vms in the esx side of things, so it is still somewhat confusing.

Do I understand right, disks can be hotadded when the job is running but you cannot hotadd them manually, using vSphere Client?

[euscada]

Transport mode (when testing the hotadd features) is configured to use "Virtual Appliance" with the "Failover to network mode" unchecked.

Do I understand right, disks can be hotadded when the job is running but you cannot hotadd them manually, using vSphere Client?

This is correct, I can not run the veeam kb for manually testing hotadd, the adding of the hdd fails -- but when veeam runs the job, I can see hotadd is functional and is working (i can open the settings of the proxy vm and see that the drive is now connected to that vm...also the "Action" window of veeam states -- "Using backup proxy xxx for disk Hard disk 1 [hotadd]"

Running a test of 1 vm configured for hotadd using a proxy forced to virtual appliance mode and forced to use the proxy takes 21 mins
--Bottleneck is stated to be the source at 94% / proxy 22% / Network 14% / Target 11%

Same job with no proxy and using network mode takes 8 mins
--Bottleneck is stated to be the source at 99% / proxy 32% / Network 6% / Target 0%

Thanks for all the input

[foggy]

Ok, please continue working with support engineers, they should be able to detect the actual phase that takes long.

[euscada]

Turns out our config of "Network mode" is recommended over "Virtual Appliance Mode" due to our 10Gb network setup.

Snapshot stunning appears to be more prevalent with the hotadd setup so we will continue to use Network mode.

No real improvement/gain for us to use the hotadd/extra proxy features.

[foggy]

Yes, network mode on 10Gb should rock.

[euscada]

Hi all,

Just to update this thread for anyone curious about this issue...

Not necessarily the "HotAdd Test Fail", but issues with overall speed with our Nutanix infrastructure and Veeam, please read this article.

https://www.veeam.com/nutanix-veeam-bac ... es_wpp.pdf

This was for some reason not noticed by Veeam techs, I had to pursue support via Nutanix.

Hope this helps anyone looking for issues utilizing Veeam with a Nutanix device.

Thanks