Comprehensive data protection for all workloads
Post Reply
dominicbatty
Novice
Posts: 3
Liked: never
Joined: Oct 02, 2014 1:43 pm
Contact:

Unable to Hot Add ... after upgrade to 9.5

Post by dominicbatty » Nov 25, 2016 10:24 am

For information ... I am seeing problems with errors relating to proxy servers being unable to [hot add] drives immediately following the upgrade to v9.5. This was working fine in v9.0 and the first and subsequent backups after the upgrade are reporting ...

"Unable to hot add source disk, failing over to network mode"

the only configuration changes we have made were to upgrade the software to v9.5 so it does look to be related to this. We are using VMWARE.

I am currently progressing this through support on case 01985225 but wanted to post this here for the community in case others are seeing the same problem or have not realised it's dropping over to network mode.

I have already checked the following and both are correct.

https://www.veeam.com/kb1054
https://www.veeam.com/kb1184

Regards, Dominic.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by foggy » Nov 25, 2016 11:09 am

Hi Dominic, thanks for posting, this doesn't look as a known issue so working with your engineer to find out the reasons of this behavior is recommended.

josephstratus
Lurker
Posts: 1
Liked: never
Joined: Dec 15, 2016 7:36 pm
Full Name: Joseph Turley
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by josephstratus » Dec 15, 2016 7:47 pm

We are having the same issue and it's only been since upgrading to 9.5. We've also had half of our backups failing since upgrading to 9.5 and support is taking days of a single email a day to get back to us.

dominicbatty
Novice
Posts: 3
Liked: never
Joined: Oct 02, 2014 1:43 pm
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by dominicbatty » Dec 19, 2016 8:49 am

Our problem is a bit odd, the second the upgrade went in none of the proxies we have would hot add disks. I've been through the mill a bit with support testing a whole variety of issues unrelated to moving to v9.5, for me everything was working fine, I installed 9.5 and the proxies stopped hot adding, I thought it was pretty obvious. However, after some effort I've now been told that I have duplicated UUID numbers in my infrastructure and that this is not compatible with v9.5. Equally there is also an entry in "Known Issues" for v9.5 that states this but not in quite such an obvious way hence why I missed it before upgrading.

The reason I have duplicate UUID numbers (these are the numbers that identify the virtual machine in the VMWARE infrastructure) is that I have VEEAM replicas of some of my proxy machines in the same infrastructure but these machines are also used as VEEAM proxies. I use these machines for VEEAM proxies as generally my infrastructure is quiet overnight hence why not use the spare capacity of a production server to facilitate the proxy requirements instead of having to implement additional windows servers which require additional capacity, licensing and maintenance.

So for example I have ...

Physical VMWARE1
VirtualMachine1
VirtualMachine2
VirtualMachine3_replica1
VirtualMachine4_replica1

Physical VMWARE2
VirtualMachine3
VirtualMachine4
VirtualMachine1_replica1
VirtualMachine2_replica1

VirtualMachine1/2/3/4 are all proxies but because they all have replicas none of them will hot add.

The current reasoning behind this is ...

"Veeam Software calls VMware API and components of VDDK. In you particular infrastructure vddk sees same uuid-numbers of proxies and replicas, so hotadd failing. Veeam 9.0 worked with old version of vddk, 9.5 works with new one. And issue is here. So you do not need to implement more proxies, you need to remove replicas of them".

However, the replicas of my proxies are actually replicas of my production servers. I can't yet establish if this is a known problem which will be fixed or a known problem I'll just have to get on with. It appears to be an issue in VMWARE based on the above response from support. Either way it's a bit of a problem, I only have two physical servers so might have to implement two additional windows servers but some might have many more than this.

Cheers, Dominic.

Cheers, Dominic.

dominicbatty
Novice
Posts: 3
Liked: never
Joined: Oct 02, 2014 1:43 pm
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by dominicbatty » Dec 19, 2016 8:52 am

Interestingly I just found this ... https://www.veeam.com/kb2159

However, this issue started for me when upgrading from 9.0 to 9.5 and not when I went to 9.0 which I have been using pretty much since it was released?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by foggy » Dec 23, 2016 11:14 am

v9 worked for you since we addressed this issue in Update 2a, however, this was somewhat broken again in v9.5 and will be fixed in the first update to it.

bigbicy
Lurker
Posts: 1
Liked: never
Joined: Dec 29, 2016 8:14 am
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by bigbicy » Dec 29, 2016 8:19 am

it seems that we have the same issue. Hot add won't work after upgrade to 9.5.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by foggy » Dec 29, 2016 12:57 pm

Please wait for the v9.5 Update 1 that will address this issue.

$username
Veeam ProPartner
Posts: 67
Liked: 6 times
Joined: Dec 09, 2014 4:28 pm
Full Name: Gary Busey
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by $username » Dec 29, 2016 10:19 pm

Is there an ETA for when 9.5 U1 will hit?

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by Shestakov » Dec 30, 2016 9:32 am

Update 1 is going to be available in 1-2 weeks. Please contact the support team to get it as soon as possible.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by foggy » Dec 30, 2016 10:24 am

You can actually request the supported pre-release code of Update 1 even today.

omerlin98000
Veeam ProPartner
Posts: 31
Liked: never
Joined: Aug 27, 2014 9:29 am
Full Name: Olivier Merlin
Location: Monte-Carlo, Monaco
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by omerlin98000 » Jan 04, 2017 10:43 am

Hello all.
Found the same problem here, after a 9.5 upgrade.
Two of my 3 VM proxies had replicas, so my jobs, when managed through these two proxies, were falling back to network mode since 9.5.
Case is #01991700, Il'ya was a VERY helpful and skilled engineer to detect this problem on 1st December.
Just after deleting my two proxies replicas, jobs started to work in hot-add mode again :)
Happy to read that there will be a patch!

BTW, here is a way to find if you have duplicate UUIDs:
- getting Bios UUID Inside a VM (PowerShell): (Get-WmiObject Win32_ComputerSystemProduct).UUID, looks like : DAE32E42-2750-3619-C366-B871E3EB9EF5
- getting UUID VM in vSphere/ESXi: download/open the .vmx file, search for Something like uuid.bios = "42 2e e3 da 50 27 19 36-c3 66 b8 71 e3 eb 9e f5"
These two information are to confirm you have the right information about the right VM.
Note that some bytes are reversed, DAE32E42 is the same as "42 2e e3 da", same for the next 4 bytes ;)

Now, to check your vCenter database, go to https://myvcenterserver/mob, login and go to query ServiceContent/SearchIndex/FindAllByUUID.
The search UUID string have to be written precisely in that manner: 8symbols+dash+4symbols+dash+4symbols+dash+4symbols+dash+all_the_rest.
That search immediately show me that I had duplicated ID on my Veeam proxies VMs (even if I can see obviously my VM replicas in vCenter):

Image

Again, many thanks to Il'ya (Veeam support) for his outstanding mastering :D

Mike Resseler
Product Manager
Posts: 5662
Liked: 593 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by Mike Resseler » Jan 04, 2017 10:45 am

Olivier,

Thanks for the detailed information on how it got solved with you. I'm sure it will be helpful for many others. I will also make sure that the Ilya will get the praise he deserves :-)

Thanks
Mike

bulkas
Lurker
Posts: 2
Liked: never
Joined: Jan 14, 2019 8:55 am
Full Name: Andrey I
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by bulkas » Jan 14, 2019 9:11 am

I am see the same problem on 2 sites after upgrade from vsphere 5.5+ VB&R 8.0 to vsphere 6.7u1 + VB&R 9.5u3a . After registry tweak jobs are working but all jobs unable to [hot add] drives. We checked combination of vc 6.7U1 and host version 6.5U1 and have same problem.
Backup server and backup proxy are the same server. Manually disk adding to proxy from KB is working. We have only one proxy and it dont have duplicated UUID in vcenter.

we dont know what to do.

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by Shestakov » Jan 14, 2019 10:40 am

Hello Andrey and welcome to the community!
Have you contacted Veeam support team? If yes, please provide the support case number.

bulkas
Lurker
Posts: 2
Liked: never
Joined: Jan 14, 2019 8:55 am
Full Name: Andrey I
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by bulkas » Jan 14, 2019 12:32 pm

Yes, we contact veeam support over portal, case 03358811.

wishr
Veeam Software
Posts: 878
Liked: 84 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by wishr » Jan 14, 2019 12:54 pm

Hi Andrey,

Thanks for letting us know. Our support engineer will reach you back for investigation - log analysis is required since the root cause may vary from case to case.

BR,
Fedor

soteris84
Lurker
Posts: 1
Liked: never
Joined: Sep 03, 2018 8:15 am
Full Name: Soteris Karagioris
Contact:

Re: Unable to Hot Add ... after upgrade to 9.5

Post by soteris84 » Feb 27, 2019 6:26 pm

I wanted to chip in and say that we too face this issue for a long time using VBR 9.5U4. We also have an open case with Veeam, Case #03427994

Thank you for your help

Post Reply

Who is online

Users browsing this forum: Bing [Bot], dstasinski and 20 guests