Host-based backup of VMware vSphere VMs.
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

CBT and Hot Add failing after Instant VM Recovery

Post by rawtaz »

Hi,

I am trying out Veeam B&R (version 6 with patch 2) very briefly so far, I've only added one backup job containing one Windows 2008 R2 VM. The VM has two 50GB VMDKs and the backup job is set to include both of them, backing up to a CIFS repository, reverse incremental, dedupe, no compression, no VMware Tools quiescence, *use CBT* and do application-aware image processing. The ESXi host storage is local DAS.

I added it one or two days ago, and for the first eight runs the backup completed just fine. I then tried an instant VM recovery, by having the backup controller publish it the VM from the backup via its own NFS (vPower NFS I think?). This worked fine too, the backed up VM was now running on the ESXi host. At first I ran it without having it connected to the network, and along side the "real" one, but then I shut down the real one and connected this backup one to the network to test it a little. So far so good. The backup job complained that it could not back up the VM anymore due to some files being locked, which I think is fine.

So after having tried this I then right-clicked the Instant Recovery VM and selected "Stop publishing". Veeam shut down the backup VM on the ESXi host, all fine. However on the following backup runs of the "real" VM, I got warnings: One saying

Code: Select all

2012-01-27 05:34:41 :: Hot add is not supported for this disk, failing over to network mode...
and the other saying

Code: Select all

2012-01-27 05:34:45 :: Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''
. These two warnings occurred once per VMDK, even for the first VMDK.

I looked in the forum and found some reports on the CBT issue, for example this. While this does seem relevant, it describes an issue where the CBT problem happens only on the second and later VMDKs, not the first one which I am seeing.

In any case I did shut down and restart the VM a couple of times, and after that I no longer have both of the above warnings, but only the Hot Add warning. To clarify; the current backup runs that are made for the VM only give the "Hot add is not supported for this disk" warning, but still for both VMDKs. I have tried to make a snapshot of the VM, and then deleted all of them (though there's only one) as I saw mentioned in another thread, but the warnings persist (and indeed the Hot Add does not work, it backs up over network now).

There is one more thing to note however, namely that after I stopped the publishing of the backed up VM and Veeam shut it down, I saw some heavy disk activity for it on the ESXi host, for quite some time. I do not know what that was about, as I had not asked Veeam to finalize/migrate the backed up VM to the ESXi host. However, the write cache during the Instant Recovery run was set to the ESXi host local storage, so could have something to do with that. I do presume, however, that the changes I made to the backed up VM during its test run were not saved back to the actual backup of it.

Any ideas why this might be and how to fix it without a new full backup (something I can do now but would like to not do in the future when there's a lot more data)? The general observation is that it's started since I did a simple Instant Recovery test run.

I know that Veeam are working on a workaround for a CBT problem that apparently is in ESXi/VMware/vSphere, but I'm not sure if it 1) will also fix this problem I was seeing for *both* VMDKs, and 2) has to do with the Hot Add problems as well.
S.Sandstrom
Lurker
Posts: 2
Liked: never
Joined: Dec 07, 2011 12:29 pm
Full Name: Scott Sandstrom
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by S.Sandstrom »

Rawtaz: Have same symptoms here. 36 backups ran without Hot Add error, now EVERY disk has that error. I'll be interested to hear what the solution is as well!
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

Been searching the forum now and looking through known limitations for Hot Add and I can't see anything that says this shouldn't work. And well, it did work before I did an Instant Recovery test so I guess that pretty much settles it :D

Hopefully someone has some pointers on how to get Hot Add running again or on what resources to look at to debug why it isn't working. I've been digging a bit in the logs on the ESXi server, but there's a lot of information in there so at first glance its hard to know what to spot, if there's anything in there.

Thanks!
no32656
Lurker
Posts: 2
Liked: never
Joined: Nov 11, 2010 11:10 am
Full Name: Roy Ingebrigtsen
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by no32656 »

Hi!
I have the same issues with the error: Hot add is not supported for this disk, failing over to network mode...
It suddenly started 2 days ago on all disk on all VM´s that I backup. There haven`t been any changes on the system.
I haven´t started any restore jobs that could have triggered this...


Kind Regards

Roy
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by Vitaliy S. »

Roy, for all technical issues it is always better to open a support case, as it is not possible to assist with technical problems via forums.
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

no32656 wrote:Hi!
I have the same issues with the error: Hot add is not supported for this disk, failing over to network mode...
It suddenly started 2 days ago on all disk on all VM´s that I backup. There haven`t been any changes on the system.
I haven´t started any restore jobs that could have triggered this...
Roy, have you tried rebooting the Veeam backup proxy/controller that handles the backups?
smile_dav
Enthusiast
Posts: 44
Liked: 5 times
Joined: Sep 26, 2011 9:47 am
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by smile_dav »

I have been experiencing the same issues here...
I think the issue resides somewhere with the proxy as a repair or a reinstall solves the problem. It also happens when I run the first backup after I've performed Instant VM recovery runs. However, this is the third time it has happened and Veeam support doesn't seem to know what the issue is, my boss aint happy getting backup warning emails :roll:
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

smile_dav wrote:I have been experiencing the same issues here...
I think the issue resides somewhere with the proxy as a repair or a reinstall solves the problem. It also happens when I run the first backup after I've performed Instant VM recovery runs. However, this is the third time it has happened and Veeam support doesn't seem to know what the issue is, my boss aint happen to get backup warning emails :roll:
Interesting, same trigger as I had then (a 100% basic Instant Recovery run). Did you open a support case with Veeam about it? Surely they must be able to figure it out from logs or something. It's not acceptable not to have Hot Add working in the long run.
smile_dav
Enthusiast
Posts: 44
Liked: 5 times
Joined: Sep 26, 2011 9:47 am
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by smile_dav »

rawtaz wrote: Interesting, same trigger as I had then (a 100% basic Instant Recovery run). Did you open a support case with Veeam about it? Surely they must be able to figure it out from logs or something. It's not acceptable not to have Hot Add working in the long run.
They told me to try resetting the CBT, but the jobs still failed.
As I needed the backup up and running without any errors for a demonstration, I just reinstalled Veeam with the same database and the issue went away but not for long...
I should also point out that in my case, jobs that failed were using the default proxy. Jobs that used other proxies didn't experience any problems, hence my conclusion that it has something to do with the default proxy. The issue also occurs after I've performed 10+ successful backups for each job, and it occurs on EVERY .vmdk in that job!
My Veeam Backup Server is a VM running Windows 2008 R2, with no other applications installed.
smile_dav
Enthusiast
Posts: 44
Liked: 5 times
Joined: Sep 26, 2011 9:47 am
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by smile_dav »

I've managed to temporarily solve the problem by removing all the proxies, then uninstall Veeam (all components) and install it again with the existing DB and it works fine.
However, I'm pretty sure this problem would appear again in the future... I hope Veeam looks at this (please).
no32656
Lurker
Posts: 2
Liked: never
Joined: Nov 11, 2010 11:10 am
Full Name: Roy Ingebrigtsen
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by no32656 »

Yes, I have tried to reboot. I`m working with Veeam Support on the error now, but it`s not solved yet. I believe this is some kind of bug.

rawtaz wrote: Roy, have you tried rebooting the Veeam backup proxy/controller that handles the backups?
acam8
Lurker
Posts: 1
Liked: never
Joined: Feb 06, 2012 12:29 pm
Full Name: Alessandro Polenta
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by acam8 »

I have the same exact issue after trying an instant recovery - Veeam B&R (version 6 with patch 2). Are there some other workarounds, other than reinstalling everytihng?

Thank you
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

The funny/interesting thing is that for me, the HotAdd started working again after a reboot of the Veeam controller VM (Windows 7 Professional). The CBT problems solved itself started working on its own as well when i fiddled around with things. It was so fuzzy that I didn't bother with it as it was so early in my testing. If it happens again I will look into it further. After that time, I've done Instant Recoveries with CBT and Hot Add working afterwards just fine.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by Vitaliy S. »

Alessandro, our support team might know other workarounds/fixes for that. Please contact them directly. Thanks!
smile_dav
Enthusiast
Posts: 44
Liked: 5 times
Joined: Sep 26, 2011 9:47 am
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by smile_dav »

I have installed the new patch on a new Veeam server, with the exact same configurations, but it didn't take long for this error to come back and haunt me... Please help, this is giving me nightmares and my boss is unhappy.
smile_dav
Enthusiast
Posts: 44
Liked: 5 times
Joined: Sep 26, 2011 9:47 am
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by smile_dav »

After some testing, I can say that the Hot Add issue is caused by a File Level Restore or a Instant Recovery run. However, when I reboot the server after a FLR or IR run, the backup jobs are fine...
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by dellock6 »

I had the same problem with a customer today, yesterday we did both a instant recovery and a file level restore, and now I'm receiving warning log complaining they had to failover to network mode, while they usually run in hotadd mode.

My customer is using Veeam Backup v6 with patch #3. Anyone else with this behaviour? Maybe we can submit a bug.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

dellock6 wrote:I had the same problem with a customer today, yesterday we did both a instant recovery and a file level restore, and now I'm receiving warning log complaining they had to failover to network mode, while they usually run in hotadd mode.

My customer is using Veeam Backup v6 with patch #3. Anyone else with this behaviour? Maybe we can submit a bug.
This is what triggered it for me too. I did my first Insant Recovery just to try it out, and right after that the VM started not being able to do HotAdd. A reboot of the Veeam controller solved it.

So to clarify; You say that they were using patch #3, and that is the latest one, right? The one that was released just recently and was meant to address this problem? Or is the latest patch not #3?
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by dellock6 »

Yes, I'm using patch #3, the last one published some days ago. I installed it right before using Instant Recovery.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by Gostev »

dellock6 wrote:Maybe we can submit a bug.
Yes, and please post the support case ID in this topic.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by dellock6 »

Hi Anton,
case opened, assigned ID#5174618.
I'm available to do all tests we need to discover the cause of the issue.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
peter.dyer
Enthusiast
Posts: 28
Liked: 2 times
Joined: Feb 10, 2010 5:05 pm
Full Name: Peter Dyer
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by peter.dyer »

I can confirm that a reboot of the Veeam server resolved these issues for me.

I cannot confirm the issues were caused by an Instant Restore. In my case I suspected a SureBackup job (basically the same as an Instant Restore). However, since the reboot I've been unable to reproduce the issues.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by dellock6 »

For now, I did not rebooted my customer's server so we can still see the issue.
Backup are failing over to network mode and they did complete, so we can stay for some days in this situation. Hopefully it would help in the resolution of the problem; I sent all the logs to support, waiting for other tests to do.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
smile_dav
Enthusiast
Posts: 44
Liked: 5 times
Joined: Sep 26, 2011 9:47 am
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by smile_dav »

Anyone got any news from Veeam Support?
I have been asked to check whether there are any mounted disks on the Veeam Backup Proxy and whether Change Block Tracking was enabled...
Just wanted to know if other people are at a more advanced stage at getting a resolution.

I know the obvious workaround is to change the transport mode for the proxy to network, but Virtual Appliance mode is much faster.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by dellock6 »

I did some tests with Support and they all failed, I exported further logs and send to them; now I'm still waiting for a showback from them.
I agreed with this customer to let the jobs failover to network mode, so we did not rebooted the Veeam Server, because seems that rebooting the server sometimes makes the error non reappearing even redoing a Instant Recovery, so we will not be able to eventually analyze the bug, if it's a bug.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
vmwarektnx
Novice
Posts: 7
Liked: never
Joined: Jan 26, 2010 8:42 am
Full Name: Roger Knutsen

Re: CBT and Hot Add failing after Instant Recovery test run

Post by vmwarektnx »

Same error, reboot fixed the problem
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by dellock6 »

Quick update: I was able to recreate the error by doing a reboot and then another File Level Restore, without an Instant Recovery. So the error is lying somewhere there...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

At least it's a very good thing that a reboot "fixes" it, I mean as opposed to having to redo the backups or jobs or whatever :)
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by Gostev »

Thank you all for your patience. The research has been completed today, we now fully understand what happens.

Indeed, this has nothing to deal with Instant VM Recovery, but only File Level Restore. The FLR process reserves a block of mount points from the system (100 of those), making hot add to use those above 100, and there is a bug in vStorage API that makes hot add go crazy in such case. The immediate hotfix will probably be to reduce the number of reserved mount point from 100 to 20 or something. As far as the current workaround, I suggest disabling local VMware backup proxy on your backup server, and using another VM as your hot add proxy server.

Thanks!
rawtaz
Expert
Posts: 100
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT and Hot Add failing after Instant Recovery test run

Post by rawtaz »

Gostev; Is it correctly understood that 1) Veeam will issue a hot fix/patch for this in the near future; 2) Until then, one may either do the workaround you suggest, OR simply not do File Level Restores and the problem shouldn't appear at all?
Post Reply

Who is online

Users browsing this forum: No registered users and 36 guests