Problems with vCenter backup and VEEAM v12 - Failed to get guest OS path for newly attached disk

[arogarth]

Him since we updated to VEEAM VBR v12 we get some error for some backups like

> 09:38:38 Preparing backup proxy <PROXY> for disk Hard disk 1 [hotadd] 00:21
> 09:39:29 Error: Failed to get guest OS path for newly attached disk [<storage>] <vm>/<vm>.vmdk: there are 2 new disks with uuid XXXXXXX

We have two Proxies on this cluster, if I set proxy01 to this Job as fix - it seems to worked. With proxy02 it doesn't.

Both Proxies has same setup...

Did anyone has the same problem here?

AND yes, I opened a ticket

[Mildur]

Hi arogarth

AND yes, I opened a ticket

What is the case number?

I checked support cases from the last 2 weeks. I don't see similar issues.
I'll suggest keep working with support to solve this issue.

Update:
Found one other case. Solution was to redeploy the proxies. But not confirmed yet if it's related to V12 or not.
Before you try to redeploy the proxies, please discuss it with your support engineer.

Best,
Fabian

[arogarth]

I send the ticketnumber via PM to you

[arogarth]

I did not redeploy - but I removed the proxy and re-add

[Mildur]

Thanks for the case number.
Let's see what support will find out in your case.

Best,
Fabian

[arogarth]

I have three clusters with this problem - on one I will try to deploy completely new proxies.

[arogarth]

OK, this did not work - same problem happened so I must wait for support...

[JH-IT]

We updated our cloud connect servers today and are experiencing the same issue.
Till now only with replica's towards us, sometimes they run but most of the time they fail with the same error.
We switched to NBD processing to work around this issue and so far the performance is reasonable (much better than with Veeam V11).
And most importantly, it seems to be stable.

Ticket: 05924055

[arogarth]

OK, found the problem together with Support.

The Main Problem was on VMware Version 9.7 and below and if you deploy lot of VMs in short time out of a template. Then some UUIDs are duplicated.
See https://kb.vmware.com/s/article/2006865 for more details.

As Service Provider this is a desaster - we have to change lot of VMs now because Backup wont work with Version 12 now. For example, for one customer there are over 120 HDDs with same UUID - And this is only ONE customer on the infrastructucture.

It would be very nice to get a option to degrade this errors as warning, so that backups will work again - It worked in Version 11...

To find Disks with duplicate UUID you can use these PowerCLI Commandlet

Code: Select all

Get-VM | Get-HardDisk | Select @{N='VM';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}} | Group-Object -Property Uuid | ?{ $_.Count -gt 1 } 

[Mildur]

Hello

Thanks for the feedback @arogarth. Thanks for the case @JH-IT.

I will have a look at both cases and discuss it with the team if they are not already escalated to RnD.

Best,
Fabian

[arogarth]

Hi,
I create a Powershell script which changes the Disk UUID without the need to shutdown the VM.

https://github.com/claranet/ClaranetHub ... e-DiskUUID

Best regards,
Aro

[MG_rubinion]

Hi,

thanks for providing your script @arogarth, this one made our day

We had the same problem after upgrade to Veeam V12.
PS told us we had two sets of VM with duplicate UUIDs, one with three VMs and one with two VMs.
I executed the Change-DiskUUID.ps1 per VM but had still the same error.

The script does not change the UUID of the last VM in the set of duplicated UUIDs, so I forced a change by editing line 112 and hardcoded the duplicated UUID recognized earlier, like

Code: Select all

$duplicateDiskUUIDs = "6000C298-d581-xxxx-xxxx-0c5316c1c92c"

After that the backup job runs whithout error so far. We will see tomorrow when all jobs were done.

[arogarth]

Hi, thanks for Feedback. Does the Script triggered vMotion? The UUID is chnages only after a vMotion. I also updated the Script yesterday because chnaging UUID not work if the VM has a Snapshot.

[MG_rubinion]

Hi,

if the script discovers a duplicated UUID (or is forced to do ) vMotion was always triggered, all VMs had no snapshot.
All jobs with problem VMs I changed so far ran without error this night.

But when all other jobs ran this night two more VMs had the same error with duplicated UUID which was not discovered by the script.
I "forced" a recreating UUID an both VMs an will see if error occurs again.

[arogarth]

OK, found new issue on this Problem - Backup over HotAdd will not work with “Linked Clone” as the created in vCloud Director if enabled or VM created in Virtual Desktop Environment.

Reproduce

Code: Select all

$VM = Get-VM "<my vm>"
$Snapshot = $VM | New-Snapshot -Name "BASE"
New-VM -Name "$($VM.Name) - Linked Clone" -VM $VM -VMHost $VM.VMHost -LinkedClone -ReferenceSnapshot $Snapshot

Get-VM -Name "$($VM.Name)*"| Get-HardDisk | Select @{N='Name';E={$_.Parent.Name}}, @{N='Uuid';E={$_.ExtensionData.Backing.Uuid}}

[JH-IT]

@arogarth, thanks for the PowerCLI Commandlet and script!

We probably saw the same behavior as @MG_rubinion.
I just ran the PowerCLI Commandlet and I have found 3 VM's having the same disk UUID.
I ran the PowerCLI Commandlet against the environment containing the same VM's we had issues with directly after the upgrade to V12 and these 3 VM's are not the VM's we had issues with.

I can confirm, the output of your PowerCLI Commandlet shows the same disk UUID from our VMware Horizon environment.

There were no issues with Veeam V11 and except for the workaround with NBD, there are no other changes made but the upgrade to V12.

[JH-IT]

Another update from me.
I created a test case by disabling NBD and using hotadd on a VM which had issues before.
The steps I followed and the results:
1. Ran the job -> It failed with error "there are 2 new disks with uuid 6000C29c-1645-59f1-efa1-ed82d9a3f127"
2. Ran the Commandlet -> VM I tested with did not show up with a duplicate ID
3. Changed the Commandlet so it would show all disk UUID's and searched for the duplicate ID -> Disk UUID is only on the VM I tested with
4. Ran the job again but listed the UUIDs while there was a snapshot on the VM (same as what Veeam does) -> No duplicate UUID found and the job failed with the same error as in step 1

As this is not a VM I can easily vMotion currently, I have not tested to change the UUID but I'm most certain this will work.

Other thing outside of LinkedClones no able to backup, the Commandlet showed duplicate UUID's on VM's deployed from template last week on vSphere 7 (Build 20842708).

This VM is located on vSAN but other VM's that are working are also located on vSAN.

[MOBO]

am i the only one who change-diskUUID.ps1 script isnt working after copying ?
it seems to be missing some coding , like line 60 missing something before ? and line 124 missing ` at end

[JH-IT]

On request of support I deployed another proxy as we are using Ubuntu 20.04 LTS with LVM but this time without LVM.
Turns out, the VM that failed processing without a duplicate ID was succesful processed on this new proxy.

Do you use also use linux proxies?

[Mildur]

Thanks JH-IT

I asked our internal teams today if there were changes between V11 and V12. I will keep you updated.

Best,
Fabian

[sarapinho]

Hello,

I had the same problem, I did the process of removing the proxy (ubuntu 22.04) and adding it again solved it for some vms and failed for others.

I ran the @arogarth script that removed the duplicate IDs, after that I retryed the job and it worked.

Thanks

[TitaniumCoder477]

Same problem here for multiple clients.

Client A: Replica
1. Deleted the replica (a small VM and a local replica so easy to replicate again)
2. Confirmed using PowerCLI no duplicates for it
3. Created a new replica job and started it
4. Experienced the issue
5. Used PowerCLI to view duplicates and saw that proxy w/ attached base disks was showing as the duplicate now!
6. This duplicate disappeared once the job failed and cleaned up

Client B: Backup
1. Regenerated a UUID on one endpoint that I could power off any time, following https://kb.vmware.com/s/article/1021189
2. Confirmed no duplicate UUID via PowerCLI
3. Tried a quick backup and got the same failure
4. PowerCLI still showed no duplicate, so I suspect it was same as #5 above where duplicate that Veeam was unhappy about was the base disk attached to the proxy itself

[Artaheri]

2. Confirmed no duplicate UUID via PowerCLI
3. Tried a quick backup and got the same failure
4. PowerCLI still showed no duplicate, so I suspect it was same as #5 above where duplicate that Veeam was unhappy about was the base disk attached to the proxy itself

This is the issue I am having. We updated to V12 a few days ago and since then most backups and replicas are failing, using Linux proxies.

PowerCLI shows no duplicate UUIDs until the job is running, then if I check while the job is running it highlights duplicates on both the original VM and the proxy when it is mounted.

I have ticket 06064453 open with Veeam support about this.

[TitaniumCoder477]

Sorry, I had to work extensively with support. This was the solution:

1. Logon to the proxy
2. sudo systemctl disable multipathd && sudo systemctl disable multipath-tools
3. Reboot

After that worked fine.

[Mildur]

Hello

I‘m discussing the issue since a few weeks with support and QA. We think, it‘s related to a known issue with multipath on V12 Linux proxy server. And not with duplicated disk UUIDs.
If you have multipath enabled on your Linux proxy, it‘s worth a try to disable multipath and check again.

@TitaniumCoder477
Thank you James, you were faster

Best,
Fabian

[michele.berardo]

Same identical issue here, thank you very much TitaniumCoder477 for sharing solution with us.
Our Case # 06057682 was not going on in the right way.
We have fixed disabling multipathd and multipath-tools

[Artaheri]

Perfect, this worked for us, too! Thanks @TitaniumCoder477 for the fix, and @Mildur for further details. Our support ticket was also going in a different direction.

[michele.berardo]

Just a question: can we consider this behavior and the multipath disable as part of a Linux Virtual Appliance standard deployment from now and on? Or should it be considered a workaround waiting for a permanent fix?

[Novox]

Code: Select all

1. Logon to the proxy
2. sudo systemctl disable multipathd && sudo systemctl disable multipath-tools
3. Reboot

Also fixed the issue for me. Should we be disabling multipath on all linux proxies? Any idea why this would suddenly become an issue? I have opened a case as well.

Thank you!

[Mildur]

Hello Novox

Yes, please disable multipath. A support case is not required. We released a kb article for this particular issue:
https://www.veeam.com/kb4460

Best,
Fabian