Unable to Hot Add ... after upgrade to 9.5

Availability for the Always-On Enterprise

Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby dominicbatty » Fri Nov 25, 2016 10:24 am

For information ... I am seeing problems with errors relating to proxy servers being unable to [hot add] drives immediately following the upgrade to v9.5. This was working fine in v9.0 and the first and subsequent backups after the upgrade are reporting ...

"Unable to hot add source disk, failing over to network mode"

the only configuration changes we have made were to upgrade the software to v9.5 so it does look to be related to this. We are using VMWARE.

I am currently progressing this through support on case 01985225 but wanted to post this here for the community in case others are seeing the same problem or have not realised it's dropping over to network mode.

I have already checked the following and both are correct.

https://www.veeam.com/kb1054
https://www.veeam.com/kb1184

Regards, Dominic.
dominicbatty
Novice
 
Posts: 3
Liked: never
Joined: Thu Oct 02, 2014 1:43 pm

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby foggy » Fri Nov 25, 2016 11:09 am

Hi Dominic, thanks for posting, this doesn't look as a known issue so working with your engineer to find out the reasons of this behavior is recommended.
foggy
Veeam Software
 
Posts: 14741
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby josephstratus » Thu Dec 15, 2016 7:47 pm

We are having the same issue and it's only been since upgrading to 9.5. We've also had half of our backups failing since upgrading to 9.5 and support is taking days of a single email a day to get back to us.
josephstratus
Lurker
 
Posts: 1
Liked: never
Joined: Thu Dec 15, 2016 7:36 pm
Full Name: Joseph Turley

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby dominicbatty » Mon Dec 19, 2016 8:49 am

Our problem is a bit odd, the second the upgrade went in none of the proxies we have would hot add disks. I've been through the mill a bit with support testing a whole variety of issues unrelated to moving to v9.5, for me everything was working fine, I installed 9.5 and the proxies stopped hot adding, I thought it was pretty obvious. However, after some effort I've now been told that I have duplicated UUID numbers in my infrastructure and that this is not compatible with v9.5. Equally there is also an entry in "Known Issues" for v9.5 that states this but not in quite such an obvious way hence why I missed it before upgrading.

The reason I have duplicate UUID numbers (these are the numbers that identify the virtual machine in the VMWARE infrastructure) is that I have VEEAM replicas of some of my proxy machines in the same infrastructure but these machines are also used as VEEAM proxies. I use these machines for VEEAM proxies as generally my infrastructure is quiet overnight hence why not use the spare capacity of a production server to facilitate the proxy requirements instead of having to implement additional windows servers which require additional capacity, licensing and maintenance.

So for example I have ...

Physical VMWARE1
VirtualMachine1
VirtualMachine2
VirtualMachine3_replica1
VirtualMachine4_replica1

Physical VMWARE2
VirtualMachine3
VirtualMachine4
VirtualMachine1_replica1
VirtualMachine2_replica1

VirtualMachine1/2/3/4 are all proxies but because they all have replicas none of them will hot add.

The current reasoning behind this is ...

"Veeam Software calls VMware API and components of VDDK. In you particular infrastructure vddk sees same uuid-numbers of proxies and replicas, so hotadd failing. Veeam 9.0 worked with old version of vddk, 9.5 works with new one. And issue is here. So you do not need to implement more proxies, you need to remove replicas of them".

However, the replicas of my proxies are actually replicas of my production servers. I can't yet establish if this is a known problem which will be fixed or a known problem I'll just have to get on with. It appears to be an issue in VMWARE based on the above response from support. Either way it's a bit of a problem, I only have two physical servers so might have to implement two additional windows servers but some might have many more than this.

Cheers, Dominic.

Cheers, Dominic.
dominicbatty
Novice
 
Posts: 3
Liked: never
Joined: Thu Oct 02, 2014 1:43 pm

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby dominicbatty » Mon Dec 19, 2016 8:52 am

Interestingly I just found this ... https://www.veeam.com/kb2159

However, this issue started for me when upgrading from 9.0 to 9.5 and not when I went to 9.0 which I have been using pretty much since it was released?
dominicbatty
Novice
 
Posts: 3
Liked: never
Joined: Thu Oct 02, 2014 1:43 pm

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby foggy » Fri Dec 23, 2016 11:14 am

v9 worked for you since we addressed this issue in Update 2a, however, this was somewhat broken again in v9.5 and will be fixed in the first update to it.
foggy
Veeam Software
 
Posts: 14741
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby bigbicy » Thu Dec 29, 2016 8:19 am

it seems that we have the same issue. Hot add won't work after upgrade to 9.5.
bigbicy
Lurker
 
Posts: 1
Liked: never
Joined: Thu Dec 29, 2016 8:14 am

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby foggy » Thu Dec 29, 2016 12:57 pm

Please wait for the v9.5 Update 1 that will address this issue.
foggy
Veeam Software
 
Posts: 14741
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby $username » Thu Dec 29, 2016 10:19 pm

Is there an ETA for when 9.5 U1 will hit?
$username
Enthusiast
 
Posts: 60
Liked: 5 times
Joined: Tue Dec 09, 2014 4:28 pm
Full Name: Gary Busey

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby Shestakov » Fri Dec 30, 2016 9:32 am

Update 1 is going to be available in 1-2 weeks. Please contact the support team to get it as soon as possible.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby foggy » Fri Dec 30, 2016 10:24 am

You can actually request the supported pre-release code of Update 1 even today.
foggy
Veeam Software
 
Posts: 14741
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby omerlin98000 » Wed Jan 04, 2017 10:43 am

Hello all.
Found the same problem here, after a 9.5 upgrade.
Two of my 3 VM proxies had replicas, so my jobs, when managed through these two proxies, were falling back to network mode since 9.5.
Case is #01991700, Il'ya was a VERY helpful and skilled engineer to detect this problem on 1st December.
Just after deleting my two proxies replicas, jobs started to work in hot-add mode again :)
Happy to read that there will be a patch!

BTW, here is a way to find if you have duplicate UUIDs:
- getting Bios UUID Inside a VM (PowerShell): (Get-WmiObject Win32_ComputerSystemProduct).UUID, looks like : DAE32E42-2750-3619-C366-B871E3EB9EF5
- getting UUID VM in vSphere/ESXi: download/open the .vmx file, search for Something like uuid.bios = "42 2e e3 da 50 27 19 36-c3 66 b8 71 e3 eb 9e f5"
These two information are to confirm you have the right information about the right VM.
Note that some bytes are reversed, DAE32E42 is the same as "42 2e e3 da", same for the next 4 bytes ;)

Now, to check your vCenter database, go to https://myvcenterserver/mob, login and go to query ServiceContent/SearchIndex/FindAllByUUID.
The search UUID string have to be written precisely in that manner: 8symbols+dash+4symbols+dash+4symbols+dash+4symbols+dash+all_the_rest.
That search immediately show me that I had duplicated ID on my Veeam proxies VMs (even if I can see obviously my VM replicas in vCenter):

Image

Again, many thanks to Il'ya (Veeam support) for his outstanding mastering :D
omerlin98000
Veeam ProPartner
 
Posts: 30
Liked: never
Joined: Wed Aug 27, 2014 9:29 am
Location: Monte-Carlo, Monaco
Full Name: Olivier Merlin

Re: Unable to Hot Add ... after upgrade to 9.5

Veeam Logoby Mike Resseler » Wed Jan 04, 2017 10:45 am

Olivier,

Thanks for the detailed information on how it got solved with you. I'm sure it will be helpful for many others. I will also make sure that the Ilya will get the praise he deserves :-)

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3151
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], law999, Majestic-12 [Bot] and 41 guests