Comprehensive data protection for all workloads
Post Reply
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

THANK YOU Instant VM Recovery

Post by tuscani » 1 person likes this post

One of our clients main server is a Server 2008 R2 which also is a DC and Exchange 2010 server ( I know I know.. but only ten users.. :P). An attempt to apply Exchange 2010 SP2 botched exchange entirely.. and too make matters worse the local person did a restore but then cancelled it midway though which of course resulted in incomplete VMDK files and VM wouldn't power on. We had the VM back online in just a few minutes with instant recovery. *high five*. The backups are actually stored on USB drive attached to the receptionists machine and the VM is still running and performing great.. of course again only ten users. Still impressive though.

My question now is the best method to get it back into production.. Manual says (yes some still read those. ha) I can simply right click and restore to production.. However, would I want to stop publishing the VM first? Do I need to delete folder for the original VM on the ESXi host prior?

Thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: THANK YOU Instant VM Recovery

Post by foggy »

tuscani wrote:However, would I want to stop publishing the VM first?
No. You need to migrate the VM to production first. Then you can remove the instantly recovered VM by opening the Backup & Replication > Instant Recovery node, right-clicking the necessary VM and clicking Stop publishing.
tuscani wrote:Do I need to delete folder for the original VM on the ESXi host prior?
You can explicitly specify the target location for the migrated VM so there's no relation to the original VM at all.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Excellent, so really there is no downtime here right.. I can do the migration anytime?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: THANK YOU Instant VM Recovery

Post by Vitaliy S. »

Yes, you can. Potential downtime period depends on how you're going to migrate your VM back to production, see first question in this section of FAQ topic
VMware : [FAQ] FREQUENTLY ASKED QUESTIONS
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Thanks.. I kicked off the migration to production about five hours ago and I am at 57% complete.. The task still says "Processing VM".. does that mean it is actually copying files from the USB drive back to the ESXi host at this stage? I am a bit confused here as the processing and processed still shows 0kb.

I also disabled backups temporarily, wasn't sure if this is required but figured it was for the best. I also decided not the delete source VM once migration is complete.. again, just being paranoid. Once the migration is complete are there any specific cleanup post migration tasks still required other than stop publishing? If I have to manually stop publishing does the new VM sit in powered off mode until I spin it up?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: THANK YOU Instant VM Recovery

Post by veremin »

Does that mean it is actually copying files from the USB drive back to the ESXi host at this stage.
Assuming you’ve chosen a Quick Migration option as a finalizing step, it means that the VM is being restored from the backup file on the production server. Once it has been completed, the VM will be suspended for a little while in order to move all changes and consolidate them with the VM data.
Once the migration is complete are there any specific cleanup post migration tasks still required other than stop publishing?
As far as I’m concerned, you won’t even have to stop publishing. Once the migration is finished, it stops automatically.

Hope this helps.
Thanks.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

The job finished and publishing did stop automatically. There were a few additional tasks I had to perform.

-Unmount the NFS datastore
-Remove the SERVERNAME_guid VM and add the server back into inventory from the SERVERNAME_1 folder
-Remove the existing disks from the VM and re-add (the disks showed up as 0mb and I received an error about parent\child disks being modified)

The other other thing I have noticed at this point in my inbox items from yesterday are missing.. however, sent and deleted items are present. I am having the users verify everything ion their side. My worry is any modified files in shares that users made yesterday would have been lost.

I see I have a delta disks no as well for each VMDK.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Spoke too soon.. it looks like the quick migration job did fail unexpectedly.. hmm.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: THANK YOU Instant VM Recovery

Post by foggy »

Justin, I suggest contacting our technical support so they could review your logs for possible reason of the migration failure.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Thanks will do
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

case 00200634

As I feared.. users are reporting the work they had saved on the server yesterday is gone.. argh.

What's weird is for email the sent\deleted items from yesterday are present.. but nothing from yesterday in the inbox folder itself.. will see what support says
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: THANK YOU Instant VM Recovery

Post by Gostev »

All data that was created after the lastt backup will be lost as expected... unless it was synced to offline copy of user mailbox, in which case it should get synced back and appear in the mailbox once the user logs on. Thus, different results from user to user, and from folder to folder is to be expected... all depends on the specific user's activity (both online and offline) during the past day.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

But I thought the migrate to production job was suppose to merge any changes that took place while the recovery VM was mounted and in use? Or does the recovered VM just become the production VM (no syncing with original)? Regardless, neither explains why we lose a day of data.

Also, Support has the logs but they have no idea why the migrate to production job failed.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: THANK YOU Instant VM Recovery

Post by Gostev »

Sure, everything that was created while the instantly recovered VM was mounted and in use will remain.

I was talking about data loss for the different period. What I was explaining is the data loss behind the definition of RPO (Recovery Point Objective) - in other words, the amount of data you are willing to lose in case of disaster. With daily backups, you can lose up to 24 hours of data (should disaster strike 1 second before the next daily backup runs). There is no magic here, no one can restore the data that was not backed up... lower RPO can only be achieved by more frequent backups.
tuscani wrote:Support has the logs but they have no idea why the migrate to production job failed.
Correction, sounds like one specific tier 1 engineer has no idea, but this only means your support case will require escalation to the higher support tier. Don't worry, we never close support cases with "Support has no idea" resolution. Would be really hard for our support to achieve 99% satisfaction ratio if we were to practice this ;)
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

"Sure, everything that was created while the instantly recovered VM was mounted and in use will remain."

That's what I thought too... ;) And RPO doesn't apply here as I am talking about loss of data from the point Instant Recovery VM came online and the point where the Quick Migration failed.

But if that was the case I wouldn't be here trying to figure out what happened whether it be the product or process followed to migrate the VM back to production. Just doesn't make since.. sure the quick migration job failed.. but I was still able to remove the 0mb disks and add the correct ones (see above) and boot the VM just fine.. it appears again eventhough the quick migration job failed the VM did come up just missing a day of work (other than sent\deleted items in email which also makes no sense)!

Here is a timeline of events...

4/22 @ 11pm – The server failed to install a service pack update for Exchange and essentially broke Exchange

4/23 @ 2am - Restored the server from the 4/22 10pm nightly backup via Instant Recovery 

4/23 @ 8am to 5pm users worked all day off of the Instant Recovery VM without issue

4/23 @ 5:30pm I kicked of a “Migrate to Production” job.

4/25 @ 3:34 AM Quick Migration job fails unexpectedly

Image
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: THANK YOU Instant VM Recovery

Post by foggy »

tuscani wrote:sure the quick migration job failed.. but I was still able to remove the 0mb disks and add the correct ones (see above) and boot the VM just fine..
Could you please clarify, what are the correct disks you are referring to?
Tobias_Elfstrom
Enthusiast
Posts: 84
Liked: 8 times
Joined: Jul 04, 2012 6:32 am
Full Name: Tobias Elfstrom
Contact:

Re: THANK YOU Instant VM Recovery

Post by Tobias_Elfstrom »

I would suspect that since the quick migration failed they snapshot chain of the vmdk files failed somehow. When you then mounted the vmdk files without the snapshots you actually mounted the restored vmdk files but lost the data that was being saved to the snapshot files during the day that your users worked.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Tobias.. that makes since, thanks.

Foggy.. the disks on the VM showed 0mb.. so I had removed them and readd the disks which were the correct size, but as Tobias pointed out these we apparently just the VMDKs from the 4/22 backup.

Again.. still would be nice to understand why the QM job failed
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: THANK YOU Instant VM Recovery

Post by Gostev »

If Quick Migration failed, your VM should still be published through Instant VM Recovery, is it the case?
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Gostev wrote:If Quick Migration failed, your VM should still be published through Instant VM Recovery, is it the case?
It was.. but we stopped publishing once we manually brought the production VM back online.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: THANK YOU Instant VM Recovery

Post by Gostev »

OK, that explains. You should not stop publishing until you successfully migrated the current state of instantly recovered VM back to the production storage with Storage VMotion or Quick Migration.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Yeah.. *facepalm*

As you said.. we should have left the user’s running on that until completing a successful Quick Migration. Definitely a learning experience. We just had assumed the job finished fine since the VM was present back on the ESXi host not realizing there was an issue until trying to power it on and then noticing 0mb disks. But by that time we had already stopped publishing the Instant Recovery VM. :(

But again, really would like to understand why the QM failed.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: THANK YOU Instant VM Recovery

Post by Vitaliy S. »

tuscani wrote:But again, really would like to understand why the QM failed.
Our support team should help you with this question.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

We have a webex scheduled for Monday :)
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani » 1 person likes this post

Quick follow-up.. I did find the culprit on the QM failure on the workstation that runs VEEAM.. A DAMN WINDOWS UPDATE REBOOTED THE MACHINE!! Argh! Nice timing huh?

Date: 4/24/2013 3:04:25 AM
Event ID: 22
Task Category: Automatic Updates
Level: Information
Keywords: Reboot
User: SYSTEM
Computer:
Description:
Restart Required: To complete the installation of the following updates, the computer will be restarted within 15 minutes:
- Security Update for Windows 7 (KB2840149)

Source: MSSQL$VEEAMSQL2008R2
Date: 4/24/2013 3:17:59 AM
Event ID: 17147
Task Category: Server
Level: Information
Keywords: Classic
User: N/A
Computer:
Description:
SQL Server is terminating because of a system shutdown. This is an informational message only. No user action is required.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="MSSQL$VEEAMSQL2008R2" />
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: THANK YOU Instant VM Recovery

Post by Gostev »

Wow, what an unfortunate coincidence indeed :(
Thanks for the follow up though.
tuscani
Enthusiast
Posts: 62
Liked: 3 times
Joined: Dec 28, 2012 8:00 pm
Full Name: Justin Durrant
Contact:

Re: THANK YOU Instant VM Recovery

Post by tuscani »

Sure.. what are the odds huh? :)
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Gostev, jp.verlande, Semrush [Bot] and 162 guests