Data LOSS - Undo Failover

[rambrosio]

I was very eager to try the Failover feature on one of my Replicated VM, which happens to be a Linux Network Stats server,
collecting stats from 130 Cisco Switches every 5 minutes so that customers can gauge their network utilization.
After reading further, I noticed at the end of page 74 of the User Guide, in small print the following sentence:

"Note: At undoing failover, any changes made to the replicated virtual machine since it was last powered on will be lost"

What? Does this mean that all the data that was collected for the past 4 hours will be lost when I Undo Failover and start the
original VM? What good is that and how can I prevent this data loss now that I'm 4 hours into this setup??? Help.

[Gostev]

Rob, undo failover restores replica VMDKs to the original state so that you can continue replication job after restoring your source VM from backup (this is the main scenario for this functionality).

If you'd like to keep the replica data, use VMware Infrastructure Client to remove the snapshot on replica to commit all changes data into VMDK. After this, your replica becomes "normal" VM, which you can do anything with. So you can either keep it running on the target host, or move it to the source host by creating new replication job back to the source host.

Thank you.

[rambrosio]

Gostev, I want to make sure I understand what I need to do as I've been collecting data on this particular server for two days now and don't want to loose it.

If I go to VMware Infrastructure Client on my destination VM and navigate to Snapshots for STATS-replica, I have the following entry: VEEAM BACKUP RUNNING SNAPSHOT, whit this description: Please do not delete this snapshot. It is being used by Veeam Backup. Are you saying that if I want to keep all collected data on this VM, I need to delete this Snapshot.

[Gostev]

Correct. Really, we are talking about regular snapshot removal operation here, nothing specific to Veeam replica.

Exact same happens each time you perform VM backup:
- snapshot is created to make VMDKs "static"
- VMDKs are backed up by backup application, while all VMDK changes are redirected into snasphot file
- when backup is complete, snapshot is removed and during this process all data from snapshot file is applied back into VMDK file, so you don't loose any VMDK changes happened while backup was running

[rambrosio]

Gostev, thanks. I deleted the Snapshot and now the VM has all the collected data committed to the VMDK, which is exactly what I wanted.
My weekend is going to be very enjoyable now that this has been taken care of )). Enjoy your.

[Gostev]

Thanks Rob

[rambrosio]

Anton, I have one more Server to Migrate, Exchange 2003. I have migrated 6 server to this point and they are all running smoothly.
While doing some final checking on Exchange I noticed the Snapshot Manager shows EXCHANGE_20AUG2009, with a description: First Exchange Snapshot.

I've been replicating this VM to a backup ESX for a few weeks now (last rep was this weekend). I also rebooted this VM as recently as a week ago.
Are Snapshots performed on the Source VM during Backup/Replication. I don't understand why I still have a Snapshot in place after the Source Exchange has been successfully replicated.

[Vitaliy S.]

Hello Rob,

Let me try to answer your question. Are you talking about snapshot on the source VM (Exchange Server)? During the backup/replica job, as Anton have already stated above, we do create the snapshot to be able to backup/replicate source VMDK files. This snapshot is called "VEEAM BACKUP TEMPORARY SNAPSHOT", and after the job finishes, the snapshot is being deleted and all the changes that happened to source VM during the job run are committed to source VMDK file. As for the snapshot EXCHANGE_20AUG2009, I believe this snapshot was manually created on August, 20. And manually created snapshot are not deleted from the source VM during the job runs, that's why you see it in the Snapshot Manager.

However the replicated VM will not have that snapshot presented as in vStorage API as well as in VCB modes, all snapshots are committed to the VMDK file, it's VMware expected behavior. But If you still want to backup your current snapshot tree, you should use Network mode (3rd RadioButton).

Thank you!

[rambrosio]

Vitaly, thanks for your reply.

Yes, the snapshot is on my Source Exchange VM. There are no Repl jobs running at this moment but the Snapshot is still there.
The destination Exchange VM also shows the same Snapshot.

As I explained above, the odd thing is that both Source and Destination VM's have been recently rebooted (last week).
If the EXCHANGE_20AUG2009 snapshot was manually created, and there have been subsequent and frequent Repls, why is that manual Snapshot still
showing.

I guess my most pressing question is: Can I delete the Snapshot on both Source and Destination VMs?

[Gostev]

Rob, our product will not touch any existing VM snapshots, so it is expected that backups or replication did not delete the existing VM snapshots.

Concerning EXCHANGE_20AUG2009 snapshot, yes you can do anything you want with this one, as this is not something our product created. By the way, judging on the date, it was created very long ago.

[rambrosio]

Vitaliy, thanks. I did use network mode for Repl.

Can you please confirm these steps going forward and completing the migration of this Exchange server:

1.) I will delete the Snapshot on my source.
2.) Delete Snapshot on destination.
3.) Delete the Repl job and create new one (I fear that destination VM db is inconsistent at this point)
4.) After new Repl has completes successfully, Failover to destination VM and test exchange db consistency

[Gostev]

Rob, the procedure looks correct, but you should completely delete replica VM at step 2, not just its snapshot.

Please note that we are not in position to give advices on operation (1), because it is regular VMware snapshot management operation of manually created snapshot, and thus is completely unrelated to Veeam Backup. So, if something goes wrong with it, we cannot be held responsible. I just wanted to explicitly state this because since snapshot is so old (almost half-year old), it is probably huge and thus will take very long time to be deleted. This may potential result in deletion timeouts and other issues. Potentially, it can even corrupt your source VM disks (we had this situation once internally when deleting very large forgotten snapshot on our Exchange server). I recommend that you start this procedure during off hours, and if you have some issue deleting this snapshot, be prepared to open support case with VMware.

[rambrosio]

Thanks Anton.. I'll delete the snapshot after hours tonight.

I still have one gnawing question regarding these snapshot.
If the source exchange Server was shut down last week and then rebooted (the Server was off line for about 30 minutes), am I correct in thinking that the Snapshot should have been deleted during that operation and subsequent Repl jobs would have created their own Snapshot?

[Gostev]

Rob, your understanding is not correct. Shutting down or rebooting the VM does not affect VM snapshots anyhow. Veeam Backup & Replication also does not "touch" existing VM snapshots during its operations, instead we create our own temporary snapshot (as Vitaly explained above), which we then automatically remove as soon as replication cycle completes.

[mhypolite]

Hi All, I have read the posts here, I just wanted to be clear on this as I have a problem similar to rambrosio.

so after I fail over to a less powerful backup server (Because of hardware power failure on the Primary), and my failed over vms have been running for a week, I obviously want to keep the data, so I delete the snapshot and my replica becomes a normal VM, when my primary esx server comes online, do I now have to delete my original VMS and then create another Replication Job to get my VMs back on my original hosts?

What happens to my original job that is stuck in the Failed over state, do I just delete that job? Or can I reuse it (I have over 20 VMS and deleting and recreating 2 jobs anytime there is a failure or DR test can become tedious at best)

[Gostev]

Michael, your above summary is 100% correct.

Job cannot "stuck in Failed over state", as failover is per VM rather than per job. So you can definitely reuse the job, just remove the deleted source VM from it, then add new source VM once replicate back from DR site completes and VM start running on the source host.

Also, I wanted to note that replica DR test does not include performing failover with Veeam Backup UI, see this thread for the process.