HotAdd fails after 24-72 hours of B&R server uptime.

[iL1fe]

Hi,
Case #05421232
Have had an issue for long long time...going back to B&R v10 ESXi6 U3. B&R on Server 2012 R2.

Now on ESXi 7 u2d Vsphere 7 U3. Veeam 11 B&R Server 2022. Current Veeam build.

The hotadd fails after 24-72 hours of B&R server uptime.
Both with Replica jobs + standard backup jobs.

The replica jobs contain error.
5/3/2022 3:32:57 AM :: Processing windowserver.domain.LOCAL Error: Failed to open VDDK disk [[ESXi-SATA-4TB-RAID10] windowserver.domain.LOCAL_replica/FP-windowsserver.domain.LOCAL-000005.vmdk] ( is read-only mode - [false] )
Logon attempt with parameters [VC/ESX: [VcenterServer.domain..LOCAL];Port: 443;Login: [username@vsphere.local];VMX Spec: [moref=vm-56349];Snapshot mor: [snapshot-63244];Transports: [nbd];Read Only: [false]] failed because of the following errors:
Failed to open disk for write.
Failed to download disk 'vddkConnSpec>'.
Shared memory connecti

Replica jobs just fail all together without hotadd. Regular Backup jobs then failover to network mode.
B&R server reboot fixes it.
I've gone through the hotadd troubleshooting guide. Hotadd works. But then it would appear some authentication issue surfaces after brief uptime. When it happens - all jobs fail hotadd. When rebooted and working. All jobs hotadd works.

Since this issue been around for so long I believe its something with Vcenter / vddk?

Any suggestions welcome. Thank you.

[jorgedlcruz]

Hello,
Thanks for the support case. I would triple check the DNS configuration across the whole platform, meaning the Veeam Components, sounds like you have all in one with VBR/Proxy, make sure that can resolve all ESXi FQDNs, plus vCenter FQDN.
And the other way around, make sure, connecting over ssh, that all ESXis and vCenter can resolve Veeam Server fqdn.

If no DNS is in place, or you think it might be erratic, try adding all the names to the /etc/hosts and on the hosts file on Windows.

Besides that, looks to me something networking related.

Keep us posted

[iL1fe]

Hi - Thanks for feedback.
Yes they're two DCs. ESXi hosts + VC all have DNS resolution. Servers added to VC via DNS names and not IPs.

Environment is setup with just one B&R server. Which is also the proxy. Have two ESXi hosts. One host backups everything in network mode without a proxy on that host.
However the ESXi host in which B&R Server 2022 / proxy VM resides has few other VMs that can backup via hotadd.

ESXi hosts + VC reside on a management vlan. B&R server / proxy is on another vlan with access via L3 switch backplane.

If it was DNS / Network related wouldn't it just never work at all. There is something connected to server uptime of a day to ~3 and then hotadds + replicas fail.

[iL1fe]

As suggested I've tested and confirmed full DNS FQDN connectivity. Everything can resolve and ping all. Vcenter, host1, host2, primary dc/dns.
All seems well on both DNS and vlan routing.
Thanks.

[jorgedlcruz]

Hello,
It sounds very strange yes, meaning that it works for a while and then it stops working, not sure if it might be some vMotion, etc. The best course of action, for now, is to wait a bit from our Support Engineers.

I can see a lot of workarounds on the forums, and on the Internet, but would not recommend wasting much time trying million of things, and letting my colleagues pin the specific error.

Thank you

[iL1fe]

Ok. Thanks.
FYI this is small 2 host vsphere essentials is without any vmotion or SAN. Its all local RAIDs on PERC H730 controller.

Server currently not working for hotadd so was running some tests on standard job.
I was combing through one of the standard job logs that can leverage hotadd. Located the hotadd.attacher.log

Few tidbits of interest here:

Code: Select all

[05.05.2022 05:46:31.970] < 20400> cli      | >>  |Agent: Failed to process method {HotAdd.AttachDisks}: Failed to open VDDK disk [[SSD960-RAID1] SERVER.DOMAIN.LOCAL/SERVER.DOMAIN.LOCAL.vmdk] ( is read-only mode - [true] )
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |--tr:{
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |--tr:VDDK error: 13 (You do not have access rights to this file). Value: 0x000000000000000d
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |--tr:Failed to open virtual disk [SSD960-RAID1] SERVER.DOMAIN.LOCAL/SERVER.DOMAIN.LOCAL.vmdk (flags: 4)
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |--tr:Failed to execute VDDK command 'Open'
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |Logon attempt with parameters [VC/ESX: [VcenterServer.DOMAIN.LOCAL];Port: 443;Login: [username@vsphere.local];VMX Spec: [moref=vm-18241];Snapshot mor: [snapshot-63302];Transports: [hotadd];Read Only: [true]] failed because of the following errors:
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |--tr:}
[05.05.2022 05:46:31.970] < 20400> cli      | >>  |--tr:Failed to get physical drives for vddk disks

Later in log:

Code: Select all

[vddk] VixDiskLib: Resolve host.
[05.05.2022 05:47:28.486] < 22200> vdl      | [vddk] 2022-05-05T05:47:28.486-07:00 warning -[22200] [Originator@6876 sub=Default] Closing Response processing in unexpected state: 3
[05.05.2022 05:47:28.486] < 19200> vdl      | [vddk] 2022-05-05T05:47:28.486-07:00 warning -[19200] [Originator@6876 sub=vimaccess] cannot get thumbprint: SSL error code '151441516', exception: 'Wrong X.509 Certificate format'
[05.05.2022 05:47:28.548] < 19200> vdl      | [vddk] VixDiskLib: VixDiskLib_Disconnect: Disconnect.

Not sure why error 13 or what it is unhappy about certificate. The vcenter + hosts are just using the standard Vmware self signed. Which of course was accepted when adding vcenter to veeam. Scan rescan vcenter server in veeam without issue. No prompt to accept cert again. All this in my opinion points to some authentication issue after brief uptime - but why I wonder...I've also double checked all NTP config - all time is good.

Thank you.

[jorgedlcruz]

Very strange,
As said, I would let Support help you accordingly. In case you might want to try the next real quick - https://kb.vmware.com/s/article/2108294

I would double-check the service account as well you are using to connect to the vCenter, and perhaps restart the ESXi Management Agents - https://kb.vmware.com/s/article/1003490

But better for you to wait on Support. Thanks!

[FirstPage]

Hello,

did you manage to get an answer about why hotadd was failing after some time on your proxy server? It seems that I have the same issue as you (a reboot fixed the hotadd issue), but naturally a periodic reboot is definitely not the good solution, so I was wondering if you had an answer or a solution for this problem?

Many thanks

[Mildur]

Hi FirstPage

Please open a support case with veeam to analyze this issue in your environment.
As a workaround you can deploy a Linux proxy instead using the VBR server.

Thanks
Fabian