Problem with automatic Transport Mode selection (NBD fallback instead of HotAdd)

[dbr]

Hi guys,

I noticed that some of the backup tasks don't use HotAdd even though they should. I don't know since when exactly, but we upgraded to VBR11 in early march. Unfortunately, session log is retained only 6 weeks back. So, I cannot say, whether we have the issues since upgrade to VBR11 or also before. Some VMs can be backed up with HotAdd and some not (similar issue: here). Veeam says that there's another host that has the same uuid (example of one job, other jobs have the same issue with different proxies):

HotAdd works not (names edited for privacy):

Code: Select all

[21.05.2021 08:24:20] <01> Info         [ViProxyVmSearcher] VM ProxySite02(vm-1212291) has the same bios uuid as proxy ProxySite01.domain.local in the DB and located on host EsxSite02.domain.local(host-1116671)

HotAdd works:

Code: Select all

[18.05.2021 08:47:59] <01> Info         [ViProxyVmSearcher] VM ProxySite01(vm-1212278) has the same bios uuid as proxy ProxySite01.domain.local in the DB and located on host EsxSite01.nordmilch.local(host-35908)

Latter does only work because ProxySite01 and ProxySite01.domain.local is exactly the same machine. We don't use cloning or anything else that could result in a same uuid. I'm wondering why Veeam is complaining a duplicate for the second log entry, because it's the same vm. Here are the uuids of the two mentioned proxies:

ProxySite01
- uuid.bios from vmx file: "42 13 42 d5 82 3b e8 36-c0 e2 ff d4 f3 07 9f 5d"
- uuid from (Get-WmiObject Win32_ComputerSystemProduct).UUID: D5421342-3B82-36E8-C0E2-FFD4F3079F5D

ProxySite02
- uuid.bios from vmx file: "42 13 1b 88 a1 4b fc 2d-98 d8 b4 89 a2 aa 64 a8"
- uuid from (Get-WmiObject Win32_ComputerSystemProduct).UUID: 881B1342-4BA1-2DFC-98D8-B489A2AA64A8

Is there a way to check which uuid Veeam compares? Are the uuids written to log file? I've already opened a support case (04817485), but at the moment we have no clue what's going on. Maybe someone else has an idea and can shed some light on it.

PS: We use VMware vSphere 7.0.1

Thanks,
Daniel

[foggy]

Hi Daniel, thanks for opening a case - that's definitely something that needs a thorough logs investigation. Let's wait for our engineers' reply once they review those.

[dbr]

Hi foggy,

the engineer already investigated logs, but I guess he's on the wrong way. However, I will give him some time to check further. Thanks.

[veremin]

What made you believe so? Have you tried to point your engineer to the right direction (if you truly believe it's the right direction)?
And remember you can always escalate the case whenever you feel unsatisfied with the level of support provided. Thanks!

[dbr]

More or less. The engineer pointed me to the log entries I've provided above. So, he's aware that the issue could be related to the uuid. I checked some other jobs an it seems that it is a general issue in our environment. Anyway, today I've already asked for escalation. He then wanted to have a look on the VeeamBackup database export first. In case he won't find something suspicious, he will escalate the case. So, I'm fine with it. I've opened the thread because I hoped that someone encountered a similar issue or has some hints to narrow down ist, so I can point the engineer in the right direction and we can speed up fixing the issue a bit. I will come back here once the cause of the issue has been found...

[veremin]

I've opened the thread because I hoped that someone encountered a similar issue or has some hints to narrow down ist, so I can point the engineer in the right direction and we can speed up fixing the issue a bit.

As you have pointed out correctly, the issue seems to be environment-specific, so there is no universal solution that comes to mind.

In case he won't find something suspicious, he will escalate the case.

Sounds reasonable.

Thanks!

[dbr]

Meanwhile the case has been escalated to second level. Quite strange what's going on here. There're several duplicated uuids in the table PhysicalHosts even though the ids are definitely unique when checking in VMware or Windows. This can be temporarily fixed by going through the proxy edit wizard or just running Get-VBRViProxy | Set-VBRViProxy until the host discovery strikes, which runs every four hours per default, or is run manually by rescan managed servers. The engineer told me that he has noticed another internal thread with a similar or maybe the same issue. I will keep you posted.

[dbr]

Follow up:

We have configured in-guest iSCSI to access the nas systems. Because the iSCSI network on every repository server isn't routed it wasn't necessary to use unique subnets across multiple repository servers. On a rescan all repository servers with the same iSCSI initiator ip adresse get a bios id of another repository server and it is written to the physicalhosts table. RnD is working on it.

[dbr]

Some weeks and a couple of private fixes later it seems to be fixed now. If we're lucky the fix will be implemented in version 11a. Otherwise I will get a private fix for that version, too.