Best option for Exchange Recovery

[benbour]

Good morning all. We had our exchange server blow up and am currently looking for the best method of getting it back. It's currently in Instant Recovery mode so our users can access email during production hours. I have the go-ahead to turn the server off at the end of the day and proceed with 1 or 2 options:

1) Finish the Instant recovery by turning off the VM and performing a cold migration.

I am a little hesitant with this method as I cannot guarantee that it will be completed by the next morning. I also did a hot migration Sunday and had it fail at the end:
9/18/2016 9:30:55 PM :: Copying snapshot files Error: Failed to get object properties. Object snapshot-7901
9/18/2016 9:45:06 PM :: Failed to process VM Mail1.nbpsdhu.ca at 2016-09-18T214506 Error: Failed to get object properties. Object snapshot-7901

2) Create a Replication job for the VM that is currently running on my Veeam Backup server, and later failover permanently to it. My only reservation with this is basically I am unfamiliar with the process. Either way, I am going to shut this thing down at the end of the day and proceed with one of the above.

Veeam recommended the Replication route as it is the quicker and safer option. Worst case scenario I can power on the Mail server in the morning if it hasn't finished by then without issue. The question I have is this:

After the replication job completes, I'll have to create a failover job to now use the replica as the new production VM. Since this VM is seen on the host, will Veeam automatically power it off and switch them over? Or how does that work when failing over to a replica? This email server is also a DC, will it have an issue when it comes up? will the DC "know" it's been cloned and fail to replicate to other DCs?

Thanks for your help as we are trying to recover from this disaster. I just wanted to post my scenario here to gain as much info as possible going forward.

[foggy]

Hi Ben, look at the planned failover scenario, looks like what you need.

[benbour]

Thanks for the reply. Would I not be looking at a Permanent Failover though? First things first I will need to replicate this server. Is it ok to perform this now, during production? or should I wait until after hours? Also, should I shut the VM down before performing the replication or is it OK to leave on?

Thanks.

[foggy]

You can first perform planned failover (which includes final synchronization of data after the VM is turned off, to avoid any data loss) and then make it permanent, if everything is fine. Replication can be performed during production hours.

[benbour]

Alright so I've created the replication job and started it. I went to make a failover plan but I don't have any replicas as of yet so I'll make it after the first replication is done. Thanks.

[foggy]

Just to correct a bit, not failover plan, but planned failover, since these are a bit different things.

[benbour]

Ahh yes, I see. I guess I only saw "failover plan" as I don't have a replica as of yet.

So to confirm:

1) Create replica of VM (can be done live even though VM is currently in instant recovery?)
2) Right click on the replica in ready state and select "planned failover"
3) After failover is complete, right click the active replica and select "make permanent"

[benbour]

Just a quick update. our mail VM is currently running in instant recovery mode, and Veeam is at the stage "preparing guest for hot backup" in the replication job:

9/19/2016 10:34:59 AM :: Preparing guest for hot backup

It's been there for just over an hour now. Should I just replicate this thing when it's off later? will that make a difference in speed?

Also, If it's still replicating tomorrow morning when email needs to be back, am I OK to turn it back on?

Thanks again for the info. I just want to be sure of every detail as this is production exchange.

[foggy]

The steps look good. The fact it takes long most likely explained by the fact it is running from the backup file. Replicating it in off state wouldn't require this particular step, indeed. What is your backup storage, btw?

Yes, turning it on while replication is still running is ok.

[benbour]

It has finished preparing the guest, and has began to replicate. I am finding that it has halted exchange to a crawl and many user's have frozen outlook clients. My backup storage is physical disks on a HP storage works server. I still plan on shutting down the server after production hours to speed up the replication, if that's ok.

[foggy]

Yep, this seems to be a reasonable step.

[benbour]

Well, It's all done, and we're back up and running. Thanks for help with this. I can confirm that a replication job with planned failover was the best option for this scenario.

[sofyane.bousalhih]

Hello,
your mail server is the same Domain controller ?, if yes you had a chance this time but never clone your DC or restore it that will cause an issue with the other domain controler and you may occur a big issue in the replication .

[foggy]

Sofyane, what issues do you mean? Domain controller backup and recovery with Veeam B&R is pretty straightforward and doesn't incur any issues if done properly (and is automatic in most cases).

[sofyane.bousalhih]

you backup your DC just to recover the AD objects removed by error but never restore all the DC VM .technically in Veeam you can do that BUT in Microsoft It could create a catastrophic Resume Producing Event.

if you have just one DC controller in your company maybe there is no impact but if you have multiple DC in your forest and you restore all the VM of the secondary DC or Primany DC , you create a problem in the infrastrucure beacause the Update Sequence Number (USN) of replication changes and this parameter is the unique key between the DC. .

it's like you told me that you take a snapshot of your DC to keep a copy of it.never,never,never,never do it , ever.

Here’s what happens:

At some point, you create a snapshot of the Domain Controller.
After taking that snapshot, the DC then going about processing is usual changes to the AD database. Those changes are then replicated to other DCs in the forest. With each change is an increase to the USN, which serves as a sort of up-to-date measurement answering the question of, “Who’s got the most recent version of the database?”.
Sometime later, you restore the DC by using the snapshot you took earlier. This restore also restores the value of the USN to its previous number. After the restoration, the DC then goes about making changes again to its database, incrementing the USN. But, this time the USN isn’t correct. Its the old USN.
Two things can happen at this point.

First, if by the next replication cycle the value of the USN on the restored DC hasn’t reached the value it had before the restore operation, any inbound replication partner will detect that the DC was improperly restored. It will signal to the forest to quarantine the DC. This is done to protect the forest.

Second, if the USN has surpassed that previous value, the DC will replicate only those changes that correspond to the USN values between the value stored in its up-to-dateness vector and the new value. This creates a situation where data on different DCs is now different, even though replication believes it to be the same. Due to the way replication is handled through USN numbers and up-to-dateness vectors, the differences in data cannot be detected, and likely cannot be corrected. In short, you’ve got an exceptionally big problem that you might not be able to fix!

[foggy]

You speak about valid things, however, do not take into account the fact that Veeam B&R handles those issues automatically, taking the appropriate steps during its application-aware processing to ensure that USN rollback does not occur. That's why we always recommend enabling application-aware image processing when backing up a DC (and in fact any Windows VM).