[Bug] Swapping OS Linux <-> Windows on physical agent leads to licensing error during backup runs

[mikeely]

This is technically a VBR problem since the issue isn't with the agents themselves. I'll explain how to re-create the issue, which can also be found in Case #05389448 and subsequently Case #05399064 for those who have access:

• Set up a server with the Linux agent and back it up for a while
• Turn down that server but do not remove it from the Veeam inventory because you forgot
• Install Windows on that same hardware
• Attempt to back it up using the Veeam Windows agent

The backup will fail as follows:

Code: Select all

Error: The agent fqdn.of.retired.linux.server (12345678-1234-1234-1234-1234567890abcd) has invalid license platform id [1c1c1c1c-2a2a-1c1c-2a2a-1c1c1c1c1c1c], but expected [c1c1c1c1-d3d3-c1c1-d3d3-c1c1c1c1c1c1]  

(these are obviously fake UUIDs but in the general format of the ones in the actual error message)

The problem exists in dbo.Licensing.LicensedInstances. Apparently the agent uses the BIOS UUID to track license usage, which makes sense until one triggers this bug. In the above-mentioned tickets I first was able to fix the Windows agent that was failing by doing

Code: Select all

update [dbo].[Licensing.LicensedInstances]
set platform_id = 'c1c1c1c1-d3d3-c1c1-d3d3-c1c1c1c1c1c1'
where platform_id = '1c1c1c1c-2a2a-1c1c-2a2a-1c1c1c1c1c1c'

This, of course, broke all the Linux agent backups with the error showing the UUIDs reversed. I should have seen this coming. The solution ultimately was to redo the update statement in reverse, then drop the row containing the stale entry from the retired Linux server. At that point the Veeam agent for Windows registered itself and backed up happily, as did the Linux agents.

I can think of several different ways to fix this but they're probably wrong. Hopefully I have at least described the bug accurately enough that Veeam engineering can fix it the right way.

[HannesK]

Hello,
as far as I see, there is no bug number in the two cases. The easiest way to get a bug fixed is asking support to check whether it's a "bug or a feature". A bug is then confirmed with a bug number (6 digit number)

Yes, we use the BIOS UUID for license tracking.

Turn down that server but do not remove it from the Veeam inventory because you forgot

That's probably the reason and might be "works as designed". I will check, but to me it sounds logical to throw an error, if the machine to be backed up has a duplicate BIOS UUID. Unfortunately BIOS UUIDs are not unique. The BIOS IDs have duplicates / collisions more often than one might expect.

Best regards,
Hannes

[mikeely]

In this case though, we have the same physical hardware, which means the BIOS UUID should be expected to be the same, and the only difference here is that we've switched from one operating system to another. While in this case the switch was from Linux to Windows it would be reasonable to expect that the same would occur if one were to switch from Windows to Linux (or ESX etc) as well.

In either case, the problem is that while the hardware _should_ be licensed assuming enough VUL instances remain, instead an error is thrown referencing an obsolete system and the backup fails. This to me at least is very much incorrect behavior - in other words, a bug.

[HannesK]

which means the BIOS UUID should be expected to be the same

how should the software know what is expected and what is an unexpected duplicate BIOS UUID?

[mikeely]

Just saying, taking a server out of commission and installing a different OS on it shouldn't result in failed backups.

[mikeely]

Hello,
as far as I see, there is no bug number in the two cases. The easiest way to get a bug fixed is asking support to check whether it's a "bug or a feature". A bug is then confirmed with a bug number (6 digit number)

Just read this part closely. I wish I had known this before the tickets had been closed, or that there was a way to reopen tickets (seriously, reopening tickets is something that would help for a lot of things).

I'll open a whole new ticket just to get this issue declared to be a bug.

[HannesK]

tickets can be re-opened since some time as far as I know. For this specific one: no need to do anything. Discussions are going on, whether it's bug or a safety feature. I will update the thread once there is a final answer.

[mikeely]

If it is a safety feature it will be useful to document the behavior as well as any mitigation for people who are repurposing existing hardware onto a different operating system.

[AlexHeylin]

FWIW I've run into several problems caused by Veeam using the BIOD UUID and expecting it to be unique and never change. Both those things aren't always true. In the case of VMs on some hypervisors, the IDs are all the same. We've also seen at least two hardware servers where the BIOS UUID changed either as a result of a BIOS upgrade or just because it felt like changing and breaking all the backups. It's better than using the MAC address or some other awful thing, but Veeam would do well to consider that these IDs are not either truly UUID or cannot change. The fallout can be substantial.

[mikeely]

It seems like there are two problems with using BIOS UUID for licensing purposes: the UUID can change, and there can be duplicate UUIDs - Supermicro sends out a painful number of servers with the serial number set to "0123456789" and I wouldn't be at all surprised to see them do the same with the UUID.
So it's probably possible to avoid or at least reduce collisions by using a checksum value of the BIOS UUID and every MAC address on the system. The chances of that value occurring on two physical systems is vanishingly small - the only way I could see it is if there were two servers with duplicate BIOS UUID and no onboard network ports and somebody swapped NICs.
I can't think of anything to mitigate changes though. Maybe it's just better for Veeam to expose whatever's being used to track the license to the users in the VBR console so we don't have to resort to SQL queries to manage changes.

[HannesK]

conversations on how to handle this scenario are still going on. it can take some time, but I will come back once there is an answer.

[AlexHeylin]

So it's probably possible to avoid or at least reduce collisions by using a checksum value of the BIOS UUID and every MAC address on the system.

Unfortunately using "the" MAC creates a whole new set of problems. The first of which is what is "THE" MAC when a machine has multiple NICs in software, and quite possibly in hardware. Saying "The one with the default route" then gets bogged down if the default route is assigned to some form of bond or software interface (which may have a non-unique or changeable MAC). Some software makes a software NIC appear as the default route where all instances of that software (universally) use the same MAC. What happens when a NIC is removed or added to the machine, or simply a cable gets un/plugged, etc. It's a can of worms.

We use a system which many years ago in pre-hardware-UUID days chose to use the MAC to uniquely identify systems. It has been an ongoing issue for at least the last seven years that I've worked with that system. It mostly works, except when it doesn't and then it can break quite spectacularly. There's no great easy answer to this (which is why the hardware UUID was invented), but beware of assuming MAC is going to help unless you're going to do a LOT of testing of edge cases and code in exactly how to repeatably deal with each and every one.

Hashing various info together does increase the chance of real uniqueness - but also hugely increases the chance that the value will change.

[mikeely]

I agree with you on using "the" MAC which is why I suggested "checksum value of the BIOS UUID and every MAC address" although as I write this it occurs to me that you'd want to be sure to sort them reliably before checksumming. None of which of course solves the issue of a NIC being swapped out which will reliably blow up this scheme.

It's not an easy problem.

[AlexHeylin]

Oh I missed the "every MAC" bit. That could well be worse the using "the MAC", because adding / removing, or even re-MACing (yes, MACs can be dynamically assigned - Hyper-V vNICs for example) any NIC (physical, virtual, or software) would cause the ID to change and that's something VBR really can't cope with. As you say - something that sounds so simple, turns out to be very complicated when you really get into it. It makes you want to beat those who deviate from the "BIOS UUID should be truly universally unique and never change" rule with a big stick doesn't it...

[mikeely]

Can a MAC be dynamically reassigned on a physical NIC though? Not sure I'm ready for that level of anarchy. Anyhow this license issue wouldn't apply to a VM unless someone were using the agent to back those up for some weird reason.

[PTide]

Hi,

With VAL the simplest workaround would be to use your own ID file instead if UUID obtained from BIOS:

1. open /etc/veeam.ini
2. find "[core]" section and add this line:

agentIdFilePath = /etc/veeamagentid

After that the Agent will use the file instead of generating UUID based on HWID

Please try it and let me know how it works.

Thanks!

[AlexHeylin]

Can a MAC be dynamically reassigned on a physical NIC though? Not sure I'm ready for that level of anarchy.

Absolutely it can, even on a hardware NIC. This has been supported by most NICs for 20 years. I've rarely seen it done, but there have been cases it's been required. One case I heard about, but did not personally see, was an organisation ended up with two NICs with the same "hardware" MAC burned into them. It look them ages to find it - because in those days MACs were supposed to be globally unique. The don't even pretend that's the case now - just that it's unlikely that you'll end up with two NICs using the same MAC in a given L2 domain.

Anyhow this license issue wouldn't apply to a VM unless someone were using the agent to back those up for some weird reason.

That's not as weird as it sounds. Sometimes there's good reasons for doing that. I don't like the reasons, but sometimes they're logically sound.

[PTide]

One case I heard about, but did not personally see, was an organisation ended up with two NICs with the same "hardware" MAC burned into them. It look them ages to find it - because in those days MACs were supposed to be globally unique. The don't even pretend that's the case now

True. The same goes with HW ID - the main reason why we added the option to define VAL Agent ID manually is because there were cases when different MBs would come with identical BIOS UUID (sometimes all of them were just zeroes).

[AlexHeylin]

@PTide - that's interesting. Did you share that observation with the VAW and VBR teams? We've had problems with VBR because this ID they've assumed to be unique and immutable isn't. If you didn't share it, please can you because it still causes edge case problems (which usually require scrapping the VBR DB and building the whole config from scratch)? Thanks

[PTide]

I am pretty sure that the whole QA department at Veeam is aware of this HW problem, but I'll double check : )

Thanks!

[HannesK]

Hello,
I just wanted to come back with a final take from our side on this topic: For foreseeable future, support is the way to go. While the SQL statement from above works in many situations, there are more things to consider. We also consider improving things directly in the product.

Best regards,
Hannes