Host-based backup of VMware vSphere VMs.
Post Reply
crond
Enthusiast
Posts: 41
Liked: 3 times
Joined: Jan 06, 2020 6:58 am
Full Name: Dmitry Becha
Contact:

Hotadd fails after ESXi host reboot.

Post by crond »

Hi All,

An ESXi 7.0.3 host is running VCSA and and latest VBR11 in a Windows 2019 VM.

Host went through reboot to install additional network card, nothing else has changed. But now all VM backup and replications jobs fail with

Code: Select all

Processing Error: Failed to open VDDK disk [[XYZ.vmdk] ( is read-only mode - [true] )
Failed to open virtual disk [YXZ.vmdk (flags: 4)
Logon attempt with parameters [VC/ESX: [Login];Port: 443;Login: [];VMX Spec: [moref=vm-];Snapshot mor: [snapshot-443];Transports: [nbd];Read Only: [true]] failed because of the following errors:
Failed to open disk for read.
Failed to upload disk.
Searching KB and forum leads to severa discussion and 2 KBs like https://www.veeam.com/kb1054
I verified UUIDs but other than that I don't think anything is applicable to my case.

Adding a new backup jobs with new source VM results in the same failure.
As a test veeam user was added to "Administrators" role on VCSA with the same results.
No errors in the VCSA or host logs. Snapshots are successfully created / removed.

Any recommendation how I can further localize the issue ?
Andreas Neufert
VP, Product Management
Posts: 6654
Liked: 1394 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Hotadd fails after ESXi host reboot.

Post by Andreas Neufert »

OK I would say the factor that you use "NBD" which uses the VMKernel interface and that you have installed a new network card are an indication that this is somehow related.

Can you share what you have done with the network card added to the ESXi? How did you configured it?

Check on the vcenter and klick on the ESXi host. What FQDN name is listed there (even when you have added the ESXi by IP, the backup APIs will use the FQDN. Check if you can PING this FQDN from Veeam Backup server and from the Veeam Proxy.

Restart the Veeam Service and the Veeam vcenter Cache Service (don´t know the exact name for now, but if in doubt reboot all Veeam* Services on the Backup server). This will force Veeam to build a new VMware topology in it´s cache.
PetrM
Veeam Software
Posts: 3216
Liked: 515 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Hotadd fails after ESXi host reboot.

Post by PetrM »

Hello,

Also, according to the report snippet, there is an attempt to open a disk for read in Network mode, but not in HotAdd:
"Snapshot mor: [snapshot-443];Transports: [nbd]"
So far, it's unclear how the issue can be related to HotAdd. I would check what Andreas has suggested and open a support case so that our engineers can analyze debug logs. Please don't forget to share a support case ID over here for our reference.

If you want to try HotAdd mode, make sure that all the requirements are met and you can HotAdd disks manually in vSphere client.

Thanks!
crond
Enthusiast
Posts: 41
Liked: 3 times
Joined: Jan 06, 2020 6:58 am
Full Name: Dmitry Becha
Contact:

Re: Hotadd fails after ESXi host reboot.

Post by crond »

Apologies for misleading snapshot of error logs.

To avoid any doubts

* All jobs are configured to use "HotAdd" and were working correctly for some time. This includes ESXi VM backup and replication jobs.
* VCSA, VBR VM and source VM are on the same esxi host and are using local (directly attached) storage
* All requirements (https://www.veeam.com/kb1054) are satisfied.

After reboot/HW changes "hot add source disk" no longer works but from Job perspective it is not considered to be a failure. I can see an information message that Job is "Unable to hot add source disk" and it falls back to network mode which in turn also fails.

I see successful login of veeam user account on VCSA side and no error messages.
Snapshot is taken successfully and there are several NFC messages related to operation performed by Veeam on a source VM/disks all market "Successful"

At this point I'm trying to understand how to debug "Unable to hot add source disk" part.
crond
Enthusiast
Posts: 41
Liked: 3 times
Joined: Jan 06, 2020 6:58 am
Full Name: Dmitry Becha
Contact:

Re: Hotadd fails after ESXi host reboot.

Post by crond »

Hmm looks like VBR sends set of commands to default default IP address (assigned to VMK with default TCP/IP stack) for ESXi host. At some point veeam starts to send commands to a different IP assigned to new VMK adapter (default stack with only vMotion service enabled) which VBR has no access to. All subsequent VDDK request never reach host.

So I guess it's borderline VMware/Veeam question specifically for VDDK. I.E how to force VBR to use a specific IP address to communicate with esxi host
crond
Enthusiast
Posts: 41
Liked: 3 times
Joined: Jan 06, 2020 6:58 am
Full Name: Dmitry Becha
Contact:

Re: Hotadd fails after ESXi host reboot.

Post by crond » 2 people like this post

Issue is resolved.
Removing backup NFC service from new VMK solves the problem.

Still not clear why Veeam switches form one IP address to a different one mid flight, but I suspect it can be more VMware/VDDK type of question
Andreas Neufert
VP, Product Management
Posts: 6654
Liked: 1394 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Hotadd fails after ESXi host reboot.

Post by Andreas Neufert »

Yes, this is a correct assumption. The VDDK (and so Veeam) will follow this setting. It was a PowerCLI configurable option since a longer time.
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 143 guests