Unable to Hot Add ... after upgrade to 9.5

[dominicbatty]

For information ... I am seeing problems with errors relating to proxy servers being unable to [hot add] drives immediately following the upgrade to v9.5. This was working fine in v9.0 and the first and subsequent backups after the upgrade are reporting ...

"Unable to hot add source disk, failing over to network mode"

the only configuration changes we have made were to upgrade the software to v9.5 so it does look to be related to this. We are using VMWARE.

I am currently progressing this through support on case 01985225 but wanted to post this here for the community in case others are seeing the same problem or have not realised it's dropping over to network mode.

I have already checked the following and both are correct.

https://www.veeam.com/kb1054
https://www.veeam.com/kb1184

Regards, Dominic.

[foggy]

Hi Dominic, thanks for posting, this doesn't look as a known issue so working with your engineer to find out the reasons of this behavior is recommended.

[josephstratus]

We are having the same issue and it's only been since upgrading to 9.5. We've also had half of our backups failing since upgrading to 9.5 and support is taking days of a single email a day to get back to us.

[dominicbatty]

Our problem is a bit odd, the second the upgrade went in none of the proxies we have would hot add disks. I've been through the mill a bit with support testing a whole variety of issues unrelated to moving to v9.5, for me everything was working fine, I installed 9.5 and the proxies stopped hot adding, I thought it was pretty obvious. However, after some effort I've now been told that I have duplicated UUID numbers in my infrastructure and that this is not compatible with v9.5. Equally there is also an entry in "Known Issues" for v9.5 that states this but not in quite such an obvious way hence why I missed it before upgrading.

The reason I have duplicate UUID numbers (these are the numbers that identify the virtual machine in the VMWARE infrastructure) is that I have VEEAM replicas of some of my proxy machines in the same infrastructure but these machines are also used as VEEAM proxies. I use these machines for VEEAM proxies as generally my infrastructure is quiet overnight hence why not use the spare capacity of a production server to facilitate the proxy requirements instead of having to implement additional windows servers which require additional capacity, licensing and maintenance.

So for example I have ...

Physical VMWARE1
VirtualMachine1
VirtualMachine2
VirtualMachine3_replica1
VirtualMachine4_replica1

Physical VMWARE2
VirtualMachine3
VirtualMachine4
VirtualMachine1_replica1
VirtualMachine2_replica1

VirtualMachine1/2/3/4 are all proxies but because they all have replicas none of them will hot add.

The current reasoning behind this is ...

"Veeam Software calls VMware API and components of VDDK. In you particular infrastructure vddk sees same uuid-numbers of proxies and replicas, so hotadd failing. Veeam 9.0 worked with old version of vddk, 9.5 works with new one. And issue is here. So you do not need to implement more proxies, you need to remove replicas of them".

However, the replicas of my proxies are actually replicas of my production servers. I can't yet establish if this is a known problem which will be fixed or a known problem I'll just have to get on with. It appears to be an issue in VMWARE based on the above response from support. Either way it's a bit of a problem, I only have two physical servers so might have to implement two additional windows servers but some might have many more than this.

Cheers, Dominic.

Cheers, Dominic.

[dominicbatty]

Interestingly I just found this ... https://www.veeam.com/kb2159

However, this issue started for me when upgrading from 9.0 to 9.5 and not when I went to 9.0 which I have been using pretty much since it was released?

[foggy]

v9 worked for you since we addressed this issue in Update 2a, however, this was somewhat broken again in v9.5 and will be fixed in the first update to it.

[bigbicy]

it seems that we have the same issue. Hot add won't work after upgrade to 9.5.

[foggy]

Please wait for the v9.5 Update 1 that will address this issue.

[$username]

Is there an ETA for when 9.5 U1 will hit?

[Shestakov]

Update 1 is going to be available in 1-2 weeks. Please contact the support team to get it as soon as possible.

[foggy]

You can actually request the supported pre-release code of Update 1 even today.

[omerlin98000]

Hello all.
Found the same problem here, after a 9.5 upgrade.
Two of my 3 VM proxies had replicas, so my jobs, when managed through these two proxies, were falling back to network mode since 9.5.
Case is #01991700, Il'ya was a VERY helpful and skilled engineer to detect this problem on 1st December.
Just after deleting my two proxies replicas, jobs started to work in hot-add mode again
Happy to read that there will be a patch!

BTW, here is a way to find if you have duplicate UUIDs:
- getting Bios UUID Inside a VM (PowerShell): (Get-WmiObject Win32_ComputerSystemProduct).UUID, looks like : DAE32E42-2750-3619-C366-B871E3EB9EF5
- getting UUID VM in vSphere/ESXi: download/open the .vmx file, search for Something like uuid.bios = "42 2e e3 da 50 27 19 36-c3 66 b8 71 e3 eb 9e f5"
These two information are to confirm you have the right information about the right VM.
Note that some bytes are reversed, DAE32E42 is the same as "42 2e e3 da", same for the next 4 bytes

Now, to check your vCenter database, go to https://myvcenterserver/mob, login and go to query ServiceContent/SearchIndex/FindAllByUUID.
The search UUID string have to be written precisely in that manner: 8symbols+dash+4symbols+dash+4symbols+dash+4symbols+dash+all_the_rest.
That search immediately show me that I had duplicated ID on my Veeam proxies VMs (even if I can see obviously my VM replicas in vCenter):



Again, many thanks to Il'ya (Veeam support) for his outstanding mastering

[Mike Resseler]

Olivier,

Thanks for the detailed information on how it got solved with you. I'm sure it will be helpful for many others. I will also make sure that the Ilya will get the praise he deserves

Thanks
Mike

[bulkas]

I am see the same problem on 2 sites after upgrade from vsphere 5.5+ VB&R 8.0 to vsphere 6.7u1 + VB&R 9.5u3a . After registry tweak jobs are working but all jobs unable to [hot add] drives. We checked combination of vc 6.7U1 and host version 6.5U1 and have same problem.
Backup server and backup proxy are the same server. Manually disk adding to proxy from KB is working. We have only one proxy and it dont have duplicated UUID in vcenter.

we dont know what to do.

[Shestakov]

Hello Andrey and welcome to the community!
Have you contacted Veeam support team? If yes, please provide the support case number.

[bulkas]

Yes, we contact veeam support over portal, case 03358811.

[wishr]

Hi Andrey,

Thanks for letting us know. Our support engineer will reach you back for investigation - log analysis is required since the root cause may vary from case to case.

BR,
Fedor

[soteris84]

I wanted to chip in and say that we too face this issue for a long time using VBR 9.5U4. We also have an open case with Veeam, Case #03427994

Thank you for your help

[morciodm]

is there any update on this issue.. we have same problem since upgrade to 9.4 u4

[wishr]

Hi Debra,

Please raise an independent support case to get it sorted out.

Thanks