Comprehensive data protection for all workloads
Post Reply
aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

2008R2 VM boot on recovery console during instant recovery

Post by aweiss »

Hi All
I have a VM at customer site that is backed up using last version of Veeam B&R (.203 x64) on a physical server, FC SAN attached to clariion, in SAN mode
The backup is successful, VSS calls inside the VM generate no error ... the Veeam job ends successfully.
When i try to run the VM during Instant Recovery session or inside a sure backup job, the VM always boots on the Recovery Console Wizard asking the language, local admin password and lets you launch CLI, Mem test etc ...
The VM is an Exchange 2010 SP1 server under 2008R2 EN with French MUI, VSS aware is enabled in the JOB with an account with enough rights inside the VM. I have rebuild a new backup job from scratch : same pb.
My customer still need to try a full restore when he will have free some space on its LUN
Case number at veeam support : 5147260
If anyone already experienced such behavior ... let me know
Other VM on 2008 R2 work fine. I also tried to deploy all templates used to create VM and they can be well backed up...
Thks for your help
Rgds
Alex
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by Vitaliy S. »

Hi Alex,

Sounds like something might be wrong with this particular VM, I would appreciate if you could update this topic once you resolve this issue with our support team.

Thanks.
aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by aweiss »

Hi Vitaly
I've just sent an update by email to support : my customer made the full vm restore test on a separate lun with no luck : the VM still boots in recovery mode.
Perhaps it's an OS related issue ... I've also sent log to support. Let's wait to see if they find something.
Alex
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by Gostev »

I am willing to bet that if your customer rebooted production server, it would boot in recovery mode as well. Sounds like the issue is already there on production server, and Veeam Backup is just grabbing the VM image with the issue.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by tsightler »

I was thinking the same thing. Have they rebooted the host to prove it does not have some type of issue? Either that, or disable VSS, which means that Veeam is not even touching the VM before the snap.
aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by aweiss »

They already rebooted the source VM with success. Hic Will it play without VSS as There is an exchange 2010 server inside the VM ? Should Tools Quiescence Will Be enough to get a consistent snap ?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by Vitaliy S. »

VMware Tools Quiescence will be enough only when pre-freeze and post-thaw backup job scripts are specified, otherwise Exchange backup image will not be in a consistent state.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by tsightler »

aweiss wrote:They already rebooted the source VM with success. Hic Will it play without VSS as There is an exchange 2010 server inside the VM ? Should Tools Quiescence Will Be enough to get a consistent snap ?
I was suggesting to test it without VSS just to see if the problem is caused only when Veeam VSS is used. If you don't use Veeam VSS then the VM is not touched at all by Veeam before it is snapped so a backup taken without VSS shouldn't be any different than if you powered off the VM at that instant. So I was simply suggesting using this as a test to narrow down the conditions under which the issue occurs.

Does the system happen to be a DC/GC? I know some people still install Exchange 2010 on a DC. If so, then booting into "Safe Mode" is actually normal (actually, the system reboots into Domain Services Repair Mode). It will take a varying amount of time (typically 15-30 minutes) to actually restore the DS before it reboots back into "normal" mode.
aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by aweiss »

Hi all
I have made many test with no luck

- Add system disk presented by Veeam to another VM : disk looks fine and there are file in it (same volume detected)
- Make a clone of the production server to one lun and backup : instant recovery is fine
- Dispatching the disks of the cloned server to 2 different LUN with the same vmdk name and backup again : instant recovery is fine
- Instant recovery from the production server without start. Connect the disk to a fresh new virtual machine : VM boots on recovery console
- Deployement of a new template, addition of a new disk with the same name as the first one on a different lun then backup up. Instant recovery is fine.
- new full backup from the clone directly with dispatched disk on two lun… : instant recovery is fine
- vmotion of the VM to another ESX and try the backup again : Instant recovery is fine

Another tets i am currently performing is booting AD + my "bad" VM into a virtual lab to be able to start exchange server inside the VM and try to backup again with exchange online. That is the only difference with my clone where Exchange is always offline
I have also update the support team by email
Alex
aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by aweiss »

VEEAM SUPPORT found the solution :)
CBT table was corrupted on this particular VM.
I tried to run the job without enabling use of CBT inside the Veeam job and all went fine.
I then, powered of production server, disabled CBT in the advanced VM properties, power on, power off and power on again. I did a new backup job with CBT enable in Veeam Job and the VM is able to but in Instant recovery mode.
Thks All
Alex
ovatandas
Lurker
Posts: 2
Liked: 1 time
Joined: Oct 04, 2011 1:05 pm
Full Name: Onur Vatandas
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by ovatandas »

Also worked for us, thanks
aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by aweiss »

Glade to see that ! The good thing should be to find how this CBT table gets corrupted
Alex
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by m.novelli »

I've enabled SureBackup for a customer and I've got the same issue: the original VM works fine, the replica and the SureBackup VM starts with Recovery Console (the OS is Windows 2008 R2)

I'll try the CBT reset trick to check if solve the problem

Cheers

Marco
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by m.novelli »

I confirm that resetting CBT solved my issue.

Now I'm wondering about all my other customers (about 25-30) with Veeam Standard Edition with all those "Success" backups... how many of them are really recoverable after two or three years from first installation? :shock:

Marco
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by veremin »

In order to be on the safe side you can perform manual verification on selective VMs, using Instant VM Recovery. Just don't forget to untick "connect VM to network" box, so that, restored VM will be isolated from production. Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by foggy »

Good point here is that you're starting using SureBackup to test your customers' backups. At least it was SureBackup who pointed out the possible issue with CBT (and further recoverability), starting VMs in the lab in recovery mode.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: 2008R2 VM boot on recovery console during instant recove

Post by m.novelli »

foggy wrote:Good point here is that you're starting using SureBackup to test your customers' backups. At least it was SureBackup who pointed out the possible issue with CBT (and further recoverability), starting VMs in the lab in recovery mode.
Yes

Would be good to have full SureBackup and SureReplica features in Veeam Standard Edition

Marco
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 135 guests