Hotadd not working

[perjonsson1960]

Folks,

Starting on Sep. 23rd, B&R is no longer able to do hotadd on most of the hard disks in the VMware VMs. Instead it is failing over to network mode. It worked fine before that date.
The vCenter server is v7.0 U2c and the hosts are 7.0 U2a.
Proxy servers: Windows Server 2019, virtual hardware is v19, and VMware Tools is v11333 (11.2.5)

One of my colleagues has upgraded virtual hardware and VMware Tools in the last few weeks, but he is not certain about the date(s) when he did that, so we don't know if that has something to do with it.

Case #05063056

PJ

[HannesK]

Hello,
support should find the reason by checking the logs.

As it makes little sense to guess on that kind of issues, please continue working with support.

As quick workaround, you could create a new proxy and see how it goes there.

Best regards,
Hannes

[perjonsson1960]

We just now saw in the VMware logs that virtual hardware in the virtual proxies was upgraded to v19 on Sep. 22, the day before the problems started. So there we might have the cause for the problem. I have informed the support about this. We'll see what they say about it.

PJ

[perjonsson1960]

Support says that there are no known problems with using virtual hardware 19 in the VMware proxies. Nevertheless, when we used a different VM, with virtual hardware 17 as proxy, it worked immediately! So we can conclude that virtual hardware 19 does not work in the VMware proxies, at least not in our environment.

PJ

[PetrM]

Hello,

I'd continue to investigate the issue in order to find the specific of your environment. By the way, did you test HotAdd manually with 19 version? I'd suggest to take some test VM for this experiment.

Thanks!

[perjonsson1960]

Yes, we did the manual hotadd, and it worked. We used the same VM and the same proxy in the manual hotadd and in a test backup job, and the manual hotadd worked, whereas the backup job did not. But, as I said, virtual hardware 17 worked like a charm in the backup job.

PJ

[PetrM]

Ok, it's up to you anyway. I suppose that the issue does not come from our code but I'd dig deeper into it to find the root cause, don't hesitate to ask our support engineers to examine debug logs in order to clarify the reason of this behavior.

Thanks!

[perjonsson1960]

On Friday we had a remote session, and the support engineer used a utility in the proxy called "vixDiskLib", and tried to do the hotadd with that, but that also failed.

[PetrM]

Hello,

Yes, this tool allows us to invoke the specific API calls in a similar way as Veeam does during backup. Most probably it would be necessary to share the results of the test with VMware support but I believe that our engineer will be able to figure out an appropriate action plan for troubleshooting.

Thanks!

[perjonsson1960]

He seems to be waiting for us to create new VMware proxies with an older virtual hardware, to verify that it works with that configuration. We will do that asap.

[perjonsson1960]

All the Hotadds worked now, when the proxies are using virtual hardware 17. No problems at all!

PJ

[perjonsson1960]

It seems that the support team is going to close this case now. I don't know if they will open a case with VMware or not, but I am going to drop this now since it works like a charm for us with virtual hardware 17 in the proxies. I should also point out that virtual hardware 19 is NOT a problem for us in the VMs being backup up, only when we are using it in the proxies.

PJ

[PetrM]

Hello,

I don't think that they will open a case with VMware if you decide to stop working on it, anyway they need some cooperation on your part to upload logs or to give the remote access etc. Glad to hear that the issue is sorted out!

Thanks!