THANK YOU Instant VM Recovery

[tuscani]

One of our clients main server is a Server 2008 R2 which also is a DC and Exchange 2010 server ( I know I know.. but only ten users.. ). An attempt to apply Exchange 2010 SP2 botched exchange entirely.. and too make matters worse the local person did a restore but then cancelled it midway though which of course resulted in incomplete VMDK files and VM wouldn't power on. We had the VM back online in just a few minutes with instant recovery. *high five*. The backups are actually stored on USB drive attached to the receptionists machine and the VM is still running and performing great.. of course again only ten users. Still impressive though.

My question now is the best method to get it back into production.. Manual says (yes some still read those. ha) I can simply right click and restore to production.. However, would I want to stop publishing the VM first? Do I need to delete folder for the original VM on the ESXi host prior?

Thanks!

[foggy]

However, would I want to stop publishing the VM first?

No. You need to migrate the VM to production first. Then you can remove the instantly recovered VM by opening the Backup & Replication > Instant Recovery node, right-clicking the necessary VM and clicking Stop publishing.

Do I need to delete folder for the original VM on the ESXi host prior?

You can explicitly specify the target location for the migrated VM so there's no relation to the original VM at all.

[tuscani]

Excellent, so really there is no downtime here right.. I can do the migration anytime?

[Vitaliy S.]

Yes, you can. Potential downtime period depends on how you're going to migrate your VM back to production, see first question in this section of FAQ topic
VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

[tuscani]

Thanks.. I kicked off the migration to production about five hours ago and I am at 57% complete.. The task still says "Processing VM".. does that mean it is actually copying files from the USB drive back to the ESXi host at this stage? I am a bit confused here as the processing and processed still shows 0kb.

I also disabled backups temporarily, wasn't sure if this is required but figured it was for the best. I also decided not the delete source VM once migration is complete.. again, just being paranoid. Once the migration is complete are there any specific cleanup post migration tasks still required other than stop publishing? If I have to manually stop publishing does the new VM sit in powered off mode until I spin it up?

[veremin]

Does that mean it is actually copying files from the USB drive back to the ESXi host at this stage.

Assuming you’ve chosen a Quick Migration option as a finalizing step, it means that the VM is being restored from the backup file on the production server. Once it has been completed, the VM will be suspended for a little while in order to move all changes and consolidate them with the VM data.

Once the migration is complete are there any specific cleanup post migration tasks still required other than stop publishing?

As far as I’m concerned, you won’t even have to stop publishing. Once the migration is finished, it stops automatically.

Hope this helps.
Thanks.

[tuscani]

The job finished and publishing did stop automatically. There were a few additional tasks I had to perform.

-Unmount the NFS datastore
-Remove the SERVERNAME_guid VM and add the server back into inventory from the SERVERNAME_1 folder
-Remove the existing disks from the VM and re-add (the disks showed up as 0mb and I received an error about parent\child disks being modified)

The other other thing I have noticed at this point in my inbox items from yesterday are missing.. however, sent and deleted items are present. I am having the users verify everything ion their side. My worry is any modified files in shares that users made yesterday would have been lost.

I see I have a delta disks no as well for each VMDK.

[tuscani]

Spoke too soon.. it looks like the quick migration job did fail unexpectedly.. hmm.

[foggy]

Justin, I suggest contacting our technical support so they could review your logs for possible reason of the migration failure.

[tuscani]

Thanks will do

[tuscani]

case 00200634

As I feared.. users are reporting the work they had saved on the server yesterday is gone.. argh.

What's weird is for email the sent\deleted items from yesterday are present.. but nothing from yesterday in the inbox folder itself.. will see what support says

[Gostev]

All data that was created after the lastt backup will be lost as expected... unless it was synced to offline copy of user mailbox, in which case it should get synced back and appear in the mailbox once the user logs on. Thus, different results from user to user, and from folder to folder is to be expected... all depends on the specific user's activity (both online and offline) during the past day.

[tuscani]

But I thought the migrate to production job was suppose to merge any changes that took place while the recovery VM was mounted and in use? Or does the recovered VM just become the production VM (no syncing with original)? Regardless, neither explains why we lose a day of data.

Also, Support has the logs but they have no idea why the migrate to production job failed.

[Gostev]

Sure, everything that was created while the instantly recovered VM was mounted and in use will remain.

I was talking about data loss for the different period. What I was explaining is the data loss behind the definition of RPO (Recovery Point Objective) - in other words, the amount of data you are willing to lose in case of disaster. With daily backups, you can lose up to 24 hours of data (should disaster strike 1 second before the next daily backup runs). There is no magic here, no one can restore the data that was not backed up... lower RPO can only be achieved by more frequent backups.

Support has the logs but they have no idea why the migrate to production job failed.

Correction, sounds like one specific tier 1 engineer has no idea, but this only means your support case will require escalation to the higher support tier. Don't worry, we never close support cases with "Support has no idea" resolution. Would be really hard for our support to achieve 99% satisfaction ratio if we were to practice this

[tuscani]

"Sure, everything that was created while the instantly recovered VM was mounted and in use will remain."

That's what I thought too... And RPO doesn't apply here as I am talking about loss of data from the point Instant Recovery VM came online and the point where the Quick Migration failed.

But if that was the case I wouldn't be here trying to figure out what happened whether it be the product or process followed to migrate the VM back to production. Just doesn't make since.. sure the quick migration job failed.. but I was still able to remove the 0mb disks and add the correct ones (see above) and boot the VM just fine.. it appears again eventhough the quick migration job failed the VM did come up just missing a day of work (other than sent\deleted items in email which also makes no sense)!

Here is a timeline of events...

4/22 @ 11pm – The server failed to install a service pack update for Exchange and essentially broke Exchange

4/23 @ 2am - Restored the server from the 4/22 10pm nightly backup via Instant Recovery 

4/23 @ 8am to 5pm users worked all day off of the Instant Recovery VM without issue

4/23 @ 5:30pm I kicked of a “Migrate to Production” job.

4/25 @ 3:34 AM Quick Migration job fails unexpectedly

[foggy]

sure the quick migration job failed.. but I was still able to remove the 0mb disks and add the correct ones (see above) and boot the VM just fine..

Could you please clarify, what are the correct disks you are referring to?

[Tobias_Elfstrom]

I would suspect that since the quick migration failed they snapshot chain of the vmdk files failed somehow. When you then mounted the vmdk files without the snapshots you actually mounted the restored vmdk files but lost the data that was being saved to the snapshot files during the day that your users worked.

[tuscani]

Tobias.. that makes since, thanks.

Foggy.. the disks on the VM showed 0mb.. so I had removed them and readd the disks which were the correct size, but as Tobias pointed out these we apparently just the VMDKs from the 4/22 backup.

Again.. still would be nice to understand why the QM job failed

[Gostev]

If Quick Migration failed, your VM should still be published through Instant VM Recovery, is it the case?

[tuscani]

If Quick Migration failed, your VM should still be published through Instant VM Recovery, is it the case?

It was.. but we stopped publishing once we manually brought the production VM back online.

[Gostev]

OK, that explains. You should not stop publishing until you successfully migrated the current state of instantly recovered VM back to the production storage with Storage VMotion or Quick Migration.

[tuscani]

Yeah.. *facepalm*

As you said.. we should have left the user’s running on that until completing a successful Quick Migration. Definitely a learning experience. We just had assumed the job finished fine since the VM was present back on the ESXi host not realizing there was an issue until trying to power it on and then noticing 0mb disks. But by that time we had already stopped publishing the Instant Recovery VM.

But again, really would like to understand why the QM failed.

[Vitaliy S.]

But again, really would like to understand why the QM failed.

Our support team should help you with this question.

[tuscani]

We have a webex scheduled for Monday

[tuscani]

Quick follow-up.. I did find the culprit on the QM failure on the workstation that runs VEEAM.. A DAMN WINDOWS UPDATE REBOOTED THE MACHINE!! Argh! Nice timing huh?

Date: 4/24/2013 3:04:25 AM
Event ID: 22
Task Category: Automatic Updates
Level: Information
Keywords: Reboot
User: SYSTEM
Computer:
Description:
Restart Required: To complete the installation of the following updates, the computer will be restarted within 15 minutes:
- Security Update for Windows 7 (KB2840149)

Source: MSSQL$VEEAMSQL2008R2
Date: 4/24/2013 3:17:59 AM
Event ID: 17147
Task Category: Server
Level: Information
Keywords: Classic
User: N/A
Computer:
Description:
SQL Server is terminating because of a system shutdown. This is an informational message only. No user action is required.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSSQL$VEEAMSQL2008R2" />

[Gostev]

Wow, what an unfortunate coincidence indeed
Thanks for the follow up though.

[tuscani]

Sure.. what are the odds huh?