"Some hot add capable backup proxy VMs were skipped..."

[bushtor]

Hi,

I get this warning at the beginning of all my replication jobs. VMWare 6 latest update and Veeam 8 latest update. Is it something I should worry about?

How should I remedy the situation..?

Thanks for comments and help on this

regards Tor

[Dima P.]

Hello Tor,
Could you please confirm VMware Tools are installed and up-to-date on the Veeam server and any Veeam proxies? Thanks

[hoFFy]

Since we upgrade to 8.0 Update 2 we receive the following message at the beginning of our backup jobs:
"Some hot add capable backup proxy VMs were skipped due to having non-unique BIOS UUID"

Whats exactly the cause for this message? Consequence is that only the local proxy, the B&R server himself is used, not the proxies in the VMs on the other hosts.

[tdewin]

Could it be that you cloned the proxies?

[hoFFy]

Nope. Nothing changed.
Trying to delete and readding the proxies didn't work

[chrisdearden]

Duplicate BIOS ID's can cause problems with the hotadd process. It seems that we now trap this error instead of seeing a hotadd failure.

you might want to try and change the BIOS UUID on the proxy servers.

http://kb.vmware.com/selfservice/micros ... Id=1002403

[hoFFy]

The proxies have not been cloned, and they weren't converted, or restored, or something else. All went finde, until we installed Update 2. Prior to the update there were no error messages, or warnings, the proxies did their job well

[KOPFteam]

Hello forum,

after the update to Veeam B&R 8.0.0.2021 I find the (green!) line

Unable to leverage hot add processing mode: All suitable backup proxy VMs have non-unique BIOS UUID

in all job results (backup job and replication jobs).

We're using VMware 5.5 with 2 ESXi and we have 2 Veeam backup proxies in use:
One is the physically installed vCenter where Veeam is running on and one is running in a VM on the source ESX.
No, I didn't clone any VMs. I installed all VMs manually.

I didn't have this error(?) warning (?) before the update and the Veeam jobs seem to run fine, but I have a bad feeling...

Thanks,
Florian

[foggy]

Those warnings are caused by the fact that VDDK 6 (adopted in Veeam B&R v8 Update 2) does not allow hotadd for VMs with non-unique BIOS UUIDs.

[dellock6]

I wonder if it has anything to do with VMs deployed from templates...

[rcampbell]

I've noticed the same thing, not deployed via template, converted from physical etc etc.

I just went through every VM checking the .vmx file for the BIOS UUID. None of them are duplicated. The only duplicates are the corresponding replica.

Is Veeam not excluding _replicas when checking for duplicate BIOS UUID's?

[Dima V.]

Helo!

This is our new check for existence of any VM, which has the same BIOS UUID as proxy Vm,
That wasn't our improvement, but new limitation from Vmware VDDK6 (used both for vSphere 5.5 and vSphere 6.0).

To find out which vm's has the same bios uuid as your proxy:

1. go to VBR logs location. %ProgramData%\Veeam\Backup\%JobName%
2. open log file called Job.%JobName%.Replica.log
3. find the section corresponding to selecting your proxy, e.g. my proxy called ved-sata-proxy:

Code: Select all

Info     [ProxyDetector] Detecting storage access level for proxy [ved-sata-proxy]

then

Code: Select all

Info     [VimApi] FindAllByUuid, uuid: "42062b51-98c3-bf6c-8bdb-926d432863a4"

you will see all vms with duplicated bios uuid:

Code: Select all

Info     VM ved-sata-proxy(vm-34683) has the same bios uuid as proxy ved-sata-proxy in the DB
Info     VM ved-sata-proxy_replica.2018(vm-35020) has the same bios uuid as proxy ved-sata-proxy in the DB

If we found more than one VM with duplicated bios uuid we discard Hotadd proxy for that proxy.

[joergr]

Yep, also direct SAN is no longer possible if no unique BIOS UUID was found. In VDDK<6 the MoRef was taken into place to check this stuff. NOW with VDDK6 the BIOS UUID is the way. Now i think VDDK6 is ONLY used with ESXi6, 5.5 and 5.1, maybe only 6.0 and 5.5 (does someone here know this 100%? Then please jump in). So the previous ESX Versions should not be affected by this.

[hoFFy]

Thanks Dima,

this helped a lot! We cloned some of our VMs into a a lab. The backup proxies are also in the lab, which caused the error.
So then the only way to come around with it, is to change the BIOS-UUID of the VMs in the lab, right?
Is there any other way to create a clone of some of our VMs to use them in a seperate virtual network without the need to manually change the UUID?

[dellock6]

Now i think VDDK6 is ONLY used with ESXi6, 5.5 and 5.1, maybe only 6.0 and 5.5 (does someone here know this 100%? Then please jump in). So the previous ESX Versions should not be affected by this.

I need to check this internally to have a final word on this, but by looking at the available VDDK libraries in an installation, I suspect VDDK 6.0 is only used against vSphere 6.0, since the 5.5 is still there together with the 5.0. I've updated my post on this topic:

http://www.virtualtothecore.com/en/vddk ... plication/

and unless the update didn't removed the 5.5 version and so it's an issue with my lab, all three versions are there.

[joergr]

Thanks, Luca! It would be great to know 100% what exact VDDK is used by B+R for which ESXi Version.

[joergr]

To add this: I find this even more interesting because VDDK 6 is in general principle compatible to 5.1, 5.5 and 6.0.

But i would like the idea that VEEAM might not jump on the gun and use this immidiately for all these when there is still the possibility to use the VDDK 5.5.2 for ESXi 5.5 and 5.1 and the VDDK 5.0.4 for ESXi 5.0 because they have proven to be solid.

[dellock6]

Correct, if you read my post (sorry for the shameless plug...) the exact reason for having multiple versions of the libraries is exactly this one...

[joergr]

great, thanks.

So - can you 100% confirm VDDK 6.0 *at this time* is only used for ESXi 6.0 by B+R?

Best regards,
Joerg

[dellock6]

No, I'll let as said the final word to PM and Developers, mine is just an assumption, otherwise there will be no reason to keep both 5.5 and 6.0 in the same installation.

[joergr]

OK - could you reach out to the devs and confirm this 100%? Because it would brighten my day )) it really would

[dellock6]

Already forwarded internally and asked to follow-up, don't worry.

[joergr]

When carefully checking out this http://forums.veeam.com/veeam-backup-re ... ml#p147051 I think VDDK 6.0 is used at least also for ESXi 5.5. The question is now - is it also used for ESXi 5.1 / 5.0?

Best regards,
Joerg

[foggy]

VDDK 6.0 is used for ESXi 6.0 and ESXi 5.5.
VDDK 5.0 is used for ESXi 5.1.

VDDK 5.5 is not used (replaced by VDDK 6.0).

[joergr]

Great, thanks, Foggy. You did not mention ESXi 5.0 - my guess is VDDK 5 is also used for ESXi 5.0 - correct?

Best regards,
Joerg

[dellock6]

Correct, you can read the answer as "ESXi 5.1 and older".
So it seems vddk 5.5 is a leftover of the installation Alexander?

[joergr]

Thanks guys

[geofftx]

So does this mean as of 8.0U2 that Veeam is now uniquely identifying VMs based on UUID and not MoRef, or are MoRefs still used?

[joergr]

Only where VDDK 6 comes in place - so for ESXi 5.5 and ESXi 6.0. Theres nothing much VEEAM can do about that - it relies on a decision by VMware.

[foggy]

Veeam B&R still tracks VMs based on moref IDs. This is just a VDDK 6.0 limitation on hotadd transport mode.