2008R2 VM boot on recovery console during instant recovery

[aweiss]

Hi All
I have a VM at customer site that is backed up using last version of Veeam B&R (.203 x64) on a physical server, FC SAN attached to clariion, in SAN mode
The backup is successful, VSS calls inside the VM generate no error ... the Veeam job ends successfully.
When i try to run the VM during Instant Recovery session or inside a sure backup job, the VM always boots on the Recovery Console Wizard asking the language, local admin password and lets you launch CLI, Mem test etc ...
The VM is an Exchange 2010 SP1 server under 2008R2 EN with French MUI, VSS aware is enabled in the JOB with an account with enough rights inside the VM. I have rebuild a new backup job from scratch : same pb.
My customer still need to try a full restore when he will have free some space on its LUN
Case number at veeam support : 5147260
If anyone already experienced such behavior ... let me know
Other VM on 2008 R2 work fine. I also tried to deploy all templates used to create VM and they can be well backed up...
Thks for your help
Rgds
Alex

[Vitaliy S.]

Hi Alex,

Sounds like something might be wrong with this particular VM, I would appreciate if you could update this topic once you resolve this issue with our support team.

Thanks.

[aweiss]

Hi Vitaly
I've just sent an update by email to support : my customer made the full vm restore test on a separate lun with no luck : the VM still boots in recovery mode.
Perhaps it's an OS related issue ... I've also sent log to support. Let's wait to see if they find something.
Alex

[Gostev]

I am willing to bet that if your customer rebooted production server, it would boot in recovery mode as well. Sounds like the issue is already there on production server, and Veeam Backup is just grabbing the VM image with the issue.

[tsightler]

I was thinking the same thing. Have they rebooted the host to prove it does not have some type of issue? Either that, or disable VSS, which means that Veeam is not even touching the VM before the snap.

[aweiss]

They already rebooted the source VM with success. Hic Will it play without VSS as There is an exchange 2010 server inside the VM ? Should Tools Quiescence Will Be enough to get a consistent snap ?

[Vitaliy S.]

VMware Tools Quiescence will be enough only when pre-freeze and post-thaw backup job scripts are specified, otherwise Exchange backup image will not be in a consistent state.

[tsightler]

They already rebooted the source VM with success. Hic Will it play without VSS as There is an exchange 2010 server inside the VM ? Should Tools Quiescence Will Be enough to get a consistent snap ?

I was suggesting to test it without VSS just to see if the problem is caused only when Veeam VSS is used. If you don't use Veeam VSS then the VM is not touched at all by Veeam before it is snapped so a backup taken without VSS shouldn't be any different than if you powered off the VM at that instant. So I was simply suggesting using this as a test to narrow down the conditions under which the issue occurs.

Does the system happen to be a DC/GC? I know some people still install Exchange 2010 on a DC. If so, then booting into "Safe Mode" is actually normal (actually, the system reboots into Domain Services Repair Mode). It will take a varying amount of time (typically 15-30 minutes) to actually restore the DS before it reboots back into "normal" mode.

[aweiss]

Hi all
I have made many test with no luck

- Add system disk presented by Veeam to another VM : disk looks fine and there are file in it (same volume detected)
- Make a clone of the production server to one lun and backup : instant recovery is fine
- Dispatching the disks of the cloned server to 2 different LUN with the same vmdk name and backup again : instant recovery is fine
- Instant recovery from the production server without start. Connect the disk to a fresh new virtual machine : VM boots on recovery console
- Deployement of a new template, addition of a new disk with the same name as the first one on a different lun then backup up. Instant recovery is fine.
- new full backup from the clone directly with dispatched disk on two lun… : instant recovery is fine
- vmotion of the VM to another ESX and try the backup again : Instant recovery is fine

Another tets i am currently performing is booting AD + my "bad" VM into a virtual lab to be able to start exchange server inside the VM and try to backup again with exchange online. That is the only difference with my clone where Exchange is always offline
I have also update the support team by email
Alex

[aweiss]

VEEAM SUPPORT found the solution
CBT table was corrupted on this particular VM.
I tried to run the job without enabling use of CBT inside the Veeam job and all went fine.
I then, powered of production server, disabled CBT in the advanced VM properties, power on, power off and power on again. I did a new backup job with CBT enable in Veeam Job and the VM is able to but in Instant recovery mode.
Thks All
Alex

[ovatandas]

Also worked for us, thanks

[aweiss]

Glade to see that ! The good thing should be to find how this CBT table gets corrupted
Alex

[m.novelli]

I've enabled SureBackup for a customer and I've got the same issue: the original VM works fine, the replica and the SureBackup VM starts with Recovery Console (the OS is Windows 2008 R2)

I'll try the CBT reset trick to check if solve the problem

Cheers

Marco

[m.novelli]

I confirm that resetting CBT solved my issue.

Now I'm wondering about all my other customers (about 25-30) with Veeam Standard Edition with all those "Success" backups... how many of them are really recoverable after two or three years from first installation?

Marco

[veremin]

In order to be on the safe side you can perform manual verification on selective VMs, using Instant VM Recovery. Just don't forget to untick "connect VM to network" box, so that, restored VM will be isolated from production. Thanks.

[foggy]

Good point here is that you're starting using SureBackup to test your customers' backups. At least it was SureBackup who pointed out the possible issue with CBT (and further recoverability), starting VMs in the lab in recovery mode.

[m.novelli]

Good point here is that you're starting using SureBackup to test your customers' backups. At least it was SureBackup who pointed out the possible issue with CBT (and further recoverability), starting VMs in the lab in recovery mode.

Yes

Would be good to have full SureBackup and SureReplica features in Veeam Standard Edition

Marco