Application aware backups with persistent guest agent fails after v13 upgrade

[RagnarJoensen]

Hi,

We recently upgrade our Veeam Backup & Replication from v12 to v13 (13.0.1.1071) and suddenly experienced a lot of VM backups were failing. Case #08010047

After investigating multiple log files, where most of the errors seemed to point and ssl or handshake errors. The issue revealed itself in the backup logs on the backup server, looking like this.

Code: Select all

Couldn't link Physical Host 06927f76-4fe0-47de-b630-37c614afcb23 to BObject 01dd32db-e6a4-4146-a5f4-20802496aff9, as Physical Host is already linked to 33606187-7375-4556-856f-8551ea881512

I located the issue to be that multiple VMs have identical BIOS UUID and this UUID is used to uniquely identify VMs. You can see the ID by running this inside the VM

Code: Select all

Get-CimInstance Win32_ComputerSystemProduct | Select-Object UUID

or or by querying SCVMM with

Code: Select all

(Get-SCVirtualMachine -Name "VM-Name").BiosGuid

I raised a case #08010047 with Veeam to see if my findings are correct. And to know if this is intentional or not, as I cannot seem to find any information regarding this in the Veeam HyperV documentation. Also in my opinion the BIOS UUID is a bad way of uniquely identifying a VM.

Solution
1. Disable "use persistent guest agent" the VM backs up just fine, if the necessary extra ports required for this are open.
2. Shutdown VM. Generate a new BIOS UUID. Power on the VM and install the Veeam deployment kit (don´t know if the entire veeam deployment kit is necessary, but it isn´t enough to just reinstall the veeaminstaller, and i didn´t test more options)

[PetrM]

Hi Ragnar and Welcome to Veeam R&D Forums!

Ok, let's wait for what our support team will find out.

Thanks!

[RagnarJoensen]

This is what I was informed by Veeam Support "With regards to the BIOS UUIDs, Veeam has always centered around that unique identifier as it provides cardinality, ensures backup-based operations are proceeding with a known and trusted entity, and to avoid licensing conflicts that could arise where there are duplicate entries that would otherwise consume licenses unfairly, etc."

But I don´t agree with this. This was not an issue with v12 and previous versions, everything worked as expected. It didn´t become a problem for us until we upgraded to v13. Now if Veeam has decided that this is the ID they are going to use forward, or they are using the ID in another manner than they used to before, then of course I will have to manage around that. But in my opinion the BIOS UUID is not a good identifier to use to uniquely identify VMs.

Or maybe I am totally missing something here, thats also a possibility. I was wrong once, this might be my second time

[RagnarJoensen]

Ok, got the final answer from support and here is the breakdown.

When running Veeam Backup & Replication v13, and have the options "Application-aware processing" and "Use persistent guest agent" enabled. This will require the usage of the "Deployment kit" on Windows VMs, and when using the deployment kit the BIOS UUID becomes the unique identifier of the VM.

So when deploying VMs in HyperV/VMware be careful, as some deployment methods result in duplicate BIOS UUIDs for VMs.

This is the only software we use, that actually uses the BIOS UUID to identify a VM/Host, so we never payed much attention to this, as it never was an issue. Lesson learned

[PetrM]

Hi Ragnar,

In v13, we use a certificate-based connection to interact with the persistent agent. We need to store information about a host in the configuration database, and the record is created based on the BIOS UUID. If there are duplicated BIOS UUIDs, we cannot establish a connection, as a different certificate will be used. I don't think there are any other identifiers that are reliable and safe enough to use. I am discussing this issue with the Dev team anyway.

How did multiple VMs end up with the same BIOS UUID? Did you clone them?

Thanks!