BSOD after B&R Migration of a running "instant Recovered VM"

[Fiskepudding]

Hello

I am currently evaluting Veeam B&R. Instant VM recovery is important for us. So I have done some testing.
It works fine, meaning that the VM starts on another host directly from the backup file.

Then I understood there were a few options when moving the VM (running in "recovery mode") back to a production environment.

The first i tested was Migrating from the backup host to a new host with "Migrate..." from vCetner. (not using Veeam for it) It took some time, but it was working.
The VM booted at in the production environment, and I was able to log on and use it normaly.

I guessed Veeam might do it better,
Just had to try the same with Migration from B&R.
So I deleted the just restored VM from the production server and "instant restored" the VM again, and Migrated from B&R while the VM was running, the process felt faster (without using clock)


The VM booted, and it the Win2008 R2 VM can stand "ready for login" forever, but as soon as I log in I get the BSOD.

IMG:



The Migration with vCenter was done with the VM powerd off.(I dont have shared storage for the test so this was my only option)

The Migration from Veeam B&R was first done with the VM running.(witch is one reason for using it...)

As a last test i did a B&R migration with the VM powered off, the VM was working fine after migration
(this time it felt like it took the same time as running migration it from vCetner)


Does this mean that its not possbile to use B&R Migration on a running VM?

Any ideas..?

Thanx

[Gostev]

Migration was designed primarily with this use case in mind (migrating a VM running of vPower NFS). I have not heard similar issues reported before, but may be there is some rare issue we do not know about? Could you please shoot the logs for this migration jobs to our support for investigation, and let me know the support case number, because I want to track it. Thanks!

[Fiskepudding]

Hello..

Sorry...
I just realised ther was a patch, and have patched server and proxy. The migration of the running VM was now successfull and I was able to log in.

Was this expected?.
I se another differnce. The host where the VM was migrated FROM, is stil holds the "instant restored VM" (before the patch the VM was deleted from soruce HOST) now it is powerd off with a suffix _migrated (Probalby the way it should be)

Good this was not a production server. (Because every changes done while the "instant restored VM" was running would have been lost, after the failed migration (since the source was deleted from disk AND not a working target VM. On they other hand they migt reside on the vPower NFS??



To summarise everyting seems to be working after the patch.

[Gostev]

Yes - as per the patch release notes, the patch did not change or fix anything around actual migration, but only added preservation of source VM in case migration fails (even though the job thinks it is successful), and you need to go back. Feel free to delete the _migrated VM manually. Actually, it should be deleted automatically as soon as heartbeat is detected from the migrated VM (but may be your test VM has no VMware Tools, or something).

But it is strange that your BSOD issue disappeared after the patch (there were no changes in the actual migration job).

[Fiskepudding]

Aarh.. Jumped the gun..

Did the same thing again. Just to se if it was really the fixed.
The VM on the backlup host is still there with "_migrated" suffix. Likt it was after the patch.. So my guess was that the migrated VM would sork fine.
I got the BSOD after login. Will try and migrate the now powered off VM that has _migrated suffix.

[Fiskepudding]

The VM i am testing with has VM tools installed (becaue i have one paravirtual disk and also has 10 GbE NIC so use VMXNET 3.

Well a hartbeat from the finished migrated VM shold by desing delete the _migrated one?
Then I would be in troble (since i have a migrated VM where the OS is acting up).
Now when the _migrated VM is NOT deleted, I at least have a chance to Migrate it when it is powered of.
Maybe that should be the default?.. Manual delete the _migrated VM? Or in my particular case "stop publish" the instant restored VM... Just a thought, I dont know

If it only were consitant.. basicly it have worked 1 out of 4 migrations (with power on), and 100% when power off.

Migrating the powered of "TEST_migrated" VM worked fine, as expected.
And the source VM "TEST_migrated" was then Deleted for the HOST.

[Fiskepudding]

Altho I feel like I am talking to myself here.
I think its only fair to update this post, if anyone takes a look.

I can no longer reproduce this problem.

To put it short. I deleted all my backups and did an brand new backup of the same VM, and i have successfully, Instant recovered the VM and migrated it while it was powered on to another ESX host.
I did this successfully 8 of 8 times. This was done with two differennt backup job settings. 1 that was identical with the one that gave med these BSOD...

However, the _migrated VM was NOT deleted in anny of my migrations, as suggested by Gostev.
Witch I think is just great, then I have the posiblity to test the new VM and delte _migrated when I feel like it.
But it mekes me wonder if this bevaviour can be expected in the futere, or if it is a bug.

OR is it becase it is an "instant recovered VM" And Veeam Migration tools wil not do that.
And that it only can be removed with the "Stop publishing" option in Veeam B&R.

And yes, VM tools are running on the new VM (production)

EDIT:The images was cropped to much when embedded
Here they are as URL:
http://www.cia.as/www/Veeam1.png
http://www.cia.as/www/Veeam2.png

[Gostev]

Hi Espen, we are researching this from our side, and we will keep in touch with you over support case you have open to share our findings when we have updates or have some questions. Thanks!

[jeepdude123]

What is the expected and optional detailed behavior in V6 patch 3 for source VM cleanup using Veeam migration?

I want to migrate a vm, but leave the source VM INTACT post migration. I want to manually clean it up ALWAYS.

Patch 2 notes and this conversation eludes to this _migrated vm naming, but what is the exact the cleanup behavior? Hopfully more than (IF Vmtools heartbeat on destination VM = alive, delete source machine?)

Again I want to be able to fire up the destination vm, install vm tools, upgrade virtual hardware etc and NOT have Veeam cleanup (delete) the source. I'm migrating from esxi 4.1 to 5, new hosts and datastores - I feel there is much more QA that needs to be done on the migrated destination machine before removing the source - think application consistency of critical SQL and heavy transaction workloads.

[Fiskepudding]

I totaly agree!

The deletion of the migrarated (source) VM should not be automatic.

This original issue with BSOD is still status "Research/Testing" and there have not been any updates on my ticket since 2012-02-27.
And regarding the migrated vm and deletion they answered:

"Regarding to your second question where original VM has not been deleted upon the successful migration - that was unexpected behavior and that information has been forwarded to our QA team for investigation."

[foggy]

Currently the behavior is exactly as it was described by Anton in his post above: the source (_migrated) VM is deleted automatically as soon as heartbeat is detected from the target VM.

[jeepdude123]

I tested this again with a esxi 4.1 to esxi5 migration. What is the "Heartbeat" you are looking for? I thought it was a vmtools heartbeat and was thinking that since the vmtools in a migration to vsphere 5 is sorta "broken" it would magically NOT cleanup, but it did. Any work arounds not a total hack to bypass this behavior for the time being?

[Gostev]

No workaround. What is the problem with removing "old" VM when the new one is running fine? VMware Tools heartbeat means that OS has booted up fine, and all system services are running normally.

[jeepdude123]

How does Veeam know that my multi-tier, transactional application is running "fine" after a migration. So if an OS boots up fine, that means the application is fine too? Serious? VM heartbeat is the all knowing application health monitor? Not a chance. Veeam and Vmware may be "happy" but it simply does not mean windows or the complex applications running on top are just magically working after the migration. How do you know I didn't change the network environment during the migration and it may not be at all trivial to get the application to "work" afterwords.

I just vconverted an exchange dag server last night for example. Let me tell you this is not trivial, not to be taken lightly, not something I would ever want go wrong.

It took 2 hours to get it to work right once it finished the 8 hour copy! (a combination of 450gb of vmdks!) When I fired the machine up vmtools was hosed I had no working virtual nic, multiple problems do to exchange firing up without working nic (had to change bindings to nic within application etc) and just weird behavior - all related to the machine being a vm AND being transferred to a new environment etc. At some point I may have had to make the call to fire up the "old" machine (remember my machine had already been down for 10 hours during the vconvert) and retry migration later due to my downtime windows and SLA. When you delete the "old" machine, I am now forced to restore and that is a whole other story that could be VERY involved, VERY risky etc. All because you deleted the "old" machine. Why would I take all this risk? I won't. I'll continue to use Vconverter!

I've migrated several critical machines before that had windows and application problems that couldn't be fixed and had to re-convert. Also when migrating a vm 4.1 to 5.0 I'm sure the post migrate virtual hardware upgrade is not 100% (if it was vmware wouldn't pop up a "please back this machine up before you convert to v8 hardware" - another case where if that process went bad, wouldn't it be nice to fire up that source machine instead of going into emergency restore mode.

I'm sure this must work all like magic in your QA lab, but as a brand new customer (I just purchased Veeam) I guess I'm telling you that this migrate and auto delete feature is incomplete. You don't need to research anymore. =)

Please offer a simple UI checkbox and workflow to simply "DON'T DELETE" my source VM post migrate. If it makes you happy, make it an "uncheck" to keep the source machine. =)

Thank you!

[Fiskepudding]

Great post jeepdude123. You have my vote for, at least, a checkbox.
But I wold say that it would even be better to change default behaviour to NOT delete.

Not that it is directly related to this issue, last week I had to use VMConverter to move a physical macine to an ESX.
This was an old terminal server with an old EDI softaware on it, that we rely on.
All other software but this worked after the convert. We put quite some time into fixing this, but this had to work before 07:00.
Time ran out and I had to power up the old physical macine again.
Its always good to know you have a someting to fall back on even if you dont need it, its reasuring.

So if there were te be a checkbox I wold like for it to be called "Delete VM after migration" and leave it UNCHECKED.