Host-based backup of VMware vSphere VMs.
Post Reply
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

Not sure if this is a veeam issue or a generic O/S issue so will ask here before logging a support call.

Having done extensive testing of a VM which wouldn't boot in our first semi-serious DR test I've found that a Veeam-backed up version of the VM isn't bootable.

Initially I thought there were perhaps issues with the replica version of the VM so I attempted to boot it from the backup that the replica was seeded from. This version had the same issue so I tried a few different restore points with the same results.

So, I tried an instant recovery from the local backup (which was used to see the DR site backup which, in turn,, seeded the DR replicas) and this one wouldn't boot either, again from various restore points.

To prove one way or the other I just completed a standalone backup of this VM (a 500gb one so good job I'm patient!) and then attempted an instant restore from this new backup, with identical results, freezes on boot.

So, I'm currently unable to recover this particular VM to a bootable state and the fact that it's a DC worries us some.

Exact symptoms are:

Boots to Windows Server 2003 logo splash screen, then after a few iterations the scrolling progress bar freezes & the VM proceeds no futher but still goes mad consuming CPU resources.
Booting into safe mode shows it getting as far as loading (or attempting to load) acpitbl.dat and gets no further.
Searching the symptoms tends to point to dodgy raid or FC controller but that's obviously not the case here.
Another suggestion was to boot into repair mode & do a chkdsk /r c: which completed with no errors & didn't resolve the issue.

We then became concerned that this issue may only arise when the VM is rebooted so were trying to figure out what, if anythign, we should do to the original VM if it has an underlying issue with reboots waiting to hit us, but when checking the reboot history in the eventlog I was surprised to find that, for whatever reason, this VM appears to be on a reboot schedule on every 3rd Wednesday of the month, with the latest reboot being 12:01 AM this morning! So, the /original/ server appears to reboot just fine, but the Veeam backup version of it simply won't boot.
I can confirm that the last windows updates applied to this VM were an embarrasingly long time ago so no recent changes
Just for kicks I also tried:
Doing an isolated instant recovery of the VM from the latest backup I just took, but /not/ powering it on
Creating a second "shell" VM and adding the DCs boot volume as an existing disk
Boot this temp shell VM from this disk
However, this seems to yield the exact same results.

I'm pretty much out of ideas now but can't figure out, as I said, whether it's a VM or Veeam issue (in fairness to Veeam, EVERY other VM I've tried this with has worked 100% as expected) but I can't see what can change from the original to a backup VM other than the wizardry that Veeam do to DCs to prep them for recovery (not sure if that happens at backup time or restore time).

Any suggestions very much appreciated!

Paul
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by veremin »

Hi, Paul. Might be a silly question, but I have to ask - do you have Application Aware Image Processing option enabled in the settings of backup job?

Thanks.
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

No such thing as a silly question in my books :) I do have it enabled, both in the original large job and in my single VM test job.
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

Anyone have any ideas?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by Vitaliy S. »

pkelly_sts wrote:Creating a second "shell" VM and adding the DCs boot volume as an existing disk
Boot this temp shell VM from this disk. However, this seems to yield the exact same results.
Did you create this VM manually and copied the OS disk from production VM? If this is the case, then it shows that the source VM is also affected by this behavior, however I'm not sure what can be causing this as you had a successful VM reboot two days ago.
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

Sorry for late reply, I'm not getting notifications for some reason plus I've been on holiday since last post...

Yes, I manually created a new VM with similar Memory/CPU specs. I then added the "existing" VMDK to it, in this case pointing it to the .VMDK that was mounted in the vPower_NFS whilst the VM was "Instant Recovered" but not powered up.

Agreed the VM reboot really scuppers things somewhat! It's due another reboot this Wednesday so will keep an eye on it anyway but no reason to doubt the consistency of the original VM.

The only other thing that springs to mind is that there's a change of CPU architecture involved (original is on AMD, DR is on Intel) but this is the exact same for all other VMs. I'm thinking I can rule it out by doing a local replica and see what happens there. Something else likely worth trying (though I'll have to do it out of hours) is to copy just the C: drive (second drive is 550gb) with the VM temporarily powered off & then trying to construct a new offline version that way which would rule out anything funky with Veeam's method of copying out data but TBH I'm clutching at straws here.

I'm wondering if logging a call with VMware would just have then sending me back to you guys?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by Vitaliy S. »

Paul,

If you could do a local replication on the same host, then it would help us to rule out the DR host's hardware from the question. I've seen a lot of successful replications between AMD/Intel chips, so I doubt it could be the reason of the initial issue.

BTW, since this a domain controller, can you please add it to the SureBackup job, select the DC role for this VM and start the verification process. During this process Veeam will do some manual steps required to boot a DC VM. If these steps help, then we'll focus on the recovery options you should follow to restore a DC from the backup.

Please let me know the results. You can check that by opening a VM console in vSphere Client.

Thanks!
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

I decided to try a local replication anyway after my last post and can confirm that the symptoms remain identical so that rules out different CPU architecture, plus it's a fresh replication from the original VM (i.e. not seeded from any backup).

Have just kicked off a Surebackup job with this VM ticked as DC, will report back shortly, well, "shortly" in Surebackup timescales :)
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

Exactly the same results via Surebackup job :( Freezes part way through splash screen...

FWIW there was a slight typo in my original post - in safe mode, the point it freezes is loading "acpitabl.dat", not "acpitbl.dat".

Scratching head now. I'm half tempted to stay up & watch Wednesdays reboot remotely but then I'm afraid I'll jinx it :-/

Swiftly running out of ideas here...

Paul
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by Vitaliy S. »

Hmm...ok, then I guess it's time to involve our support team into further investigation. Please open a support case and let me know your case number. Also I wonder if there are any error messages in the VMware VM log files. These files can be found in the VM home folder on the datastore hosting the VM config file.
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

OK, will open a ticket & report back.

Just to confirm, do you mean log files within the original VM folder, or in a replicated/restored copy of the VM?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by Vitaliy S. »

You need to look through the logs of the restored VMs, as original VM seems to be working fine ;)
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Backups of 1 VM unbootable - Veeam or OS issue?

Post by pkelly_sts »

I guessed as much but wanted to check :)

Now logged as Case ID: 00445724
Post Reply

Who is online

Users browsing this forum: No registered users and 111 guests