Issues with powering up DR replica

[jreid]

Hi,

we are having intermittant issues when we run tests of our DR environment.

Production environment is vSphere 4.0 with 2 hosts and 7VMs
All Vms are set to Hardware 7
Veeam server (WIndows 2003) is located within production environment as standalone (non domain) server

DR environment is vSphere 4.0 with 2 hosts
One host is for replication target only
One host is for running powered up DR replicas

Link speed between production and DR site is 50Mbps
Replication jobs configured to use Changed Block Tracking and VSS

We perform regular failover tests of the DR environment as follows.
Disable replication jobs.
VMotion DR replica servers from replication target host to second DR host (as replication target host does not have sufficient resources to run DR replicas)
During VMotion VMs remain on same data store, just host change
Power up VMs on DR host server
Test environment - OK
Power down VMs on DR host server
VMotion VMs back to replication target host
Re-enable replication jobs

I am seeing issues which seem to point to disk corruption, chkdsk running at startup, registry hive errors on boot, Exchange databases in 'dirty shutdown' state even though VSS in logs reports successfull with no errors......??

Is this linked to how we are doing the DR tests???
If I power up the replica and run tests the replca VM disk changes.
Does CBT deal with this??
Im thinking that as the vmdk mask is on the source/production environment it may not as it will replcate what it thinks has changed which will now be out of sequence with the replica vmdk.
If this is the case is there a way around this, first replca after test to disable CBT so Veeam does a comparison?

Note. We are not using the Veeam console for failover as in a real DR situation the Veeam console would be dead with the rest of the production environment.

Many thanks

Jason

[Vitaliy S.]

Jason,

Your tests are all fine except for one thing which causes this issue. The difference between powering on replicated VMs using Veeam console and doing it manually is the snapshot creation. When you perform a failover using Veeam console we create a snapshot to protect replicated VMDK from user's changes, so that you are able to revert snapshot and continue replicating after the testing has been completed.

However, when you just power on VM manually using vSphere Client, the replica virtual disks are being changed by the running replica VM, making the disks go out-of-sync with the source VM disks. The following incremental job run will not be aware about these changes, assuming that the virtual disk is still in the state it was left during the previous incremental run. So the job will continue to update blocks changed on the source VM, but blocks changed during replica testing will remain different with the corresponding source blocks. Thus, your replica VM will remain inconsistent until you delete it and replicate over from scratch.

So, if you do want to check VMs state using vSphere client do not forget to make a snapshot beforehand and then remove it (without committing all the changes), so you incremental job runs do not corrupt disks. Hope it makes sense.

Thanks!

[jreid]

Many thanks Vitaliy,

my next question was going to ask if creating a snapshot before power on would work.
Makes so much sense when you sit back and think about.

Kind regards

Jason

[jreid]

Sorry,

can I just confirm if I change the job and disable CBT will that then bring the replica back into sync (as without cbt Veeam will compare the source and target vmdk and replicated any differences).
Just want to be 100% about fixing the issue I have now created on our DR replicas.

Thanks

Jason

[Vitaliy S.]

I'm afraid it won't fix that, you need to remove your replicated VMs and perform a replication job one more time to have it in consistent state.

[Gostev]

Here is the correct procedure of replica testing which won't corrupt replica disks during the testing:
http://www.veeam.com/forums/viewtopic.p ... ing#p10941

[joergr]

now i have a question here: lets say some customer simulates and tests his replica by booting it and forgets to revert the taken snapshot or even don´t take a snapshot at all. so alright, replication is no more secure then because the changes tracking is screwed up. But what i would find very important: Will VEEAM B+R actually KNOW that there is something wrong or will it just continue replicating and telling the user everything is all right?

best regards,
Joerg

[mmonroe]

Ok, lets take it a step farther..

I am testing and trying to understand the replication process and need some clarifiation. I am not finding much in my doc with regards to my thoughts below.

So far, we have discussed making the replications on a "DR" environment and then testing them to make sure they boot and whatnot - either via the Veeam console to do a failover or without it. When you take a look at the DR environment and browse the datastore, you can see where Veeam is creating the VBK and VRB files.

Lets say that your main environment has a catatrophic failure. The main Veeam VM server was in that environment so you basically just have to go to the replicated DR area and turn on the VM's go get things back online. So far, so good - your VM's are running and back into semi-production on DR hardware.

Some time passes and now you have the main environment "fixed", "rebuilt", etc. You new need to get the replicated VM back to the main environment. They need to be clean afterwards - ie no VBK or VRB files taking up space and whatnot.

So how do you go about doing it? and what if you need to "turn-on" a specific replica copy versus the latest sysnthetic one?

Create a Veeam VM server in that environment and then use the "copy" function to get them moved? Use VMware to simply export them to the new environment? How are the VBK and VRB's going to get cleaned up?

Just getting the VM's running in the DR area is half of the battle, you still have to get everything back into the live production environment at some point and I can't find any doc that details ways of doing this.

Thanks in advance,

Mark

[Vitaliy S.]

Hi Mark,

I would recommend you to use hot VM copy or create replication job to move your VMs back to production site. VBK and VRB files won't be copied/replicated and you can easily remove them from the DR site after that. By the way have you had a chance to look through these discussions that cover a lot of questions about replication/failover: What to do after a "disaster" and Veeam Server DR

Hope it helps!

[rhnb]

Can I just chip in here with something we've noticed.
It's to do with the Network Adapter's MAC address.

We've found that with something running as say a licence server then if you bring the replica up (different MAC address) then things can go awry.
It's 'fixable' but requires manual editing.
I'd assume that this could cause issues if you tried restoring a Windows Domain Controller from a replica too? (Though luckily at the moment I'll never have to try that

I' sure there's some very good reason why the MAC address of the replica is not the same as the original? (ie trying to bring up the replica when the original VM is still running - but that causes issues anyway with IP addresses etc).

Cheeers... Allan.

[tsightler]

Can I just chip in here with something we've noticed.
It's to do with the Network Adapter's MAC address.

We've found that with something running as say a licence server then if you bring the replica up (different MAC address) then things can go awry.
It's 'fixable' but requires manual editing.
I'd assume that this could cause issues if you tried restoring a Windows Domain Controller from a replica too? (Though luckily at the moment I'll never have to try that

I' sure there's some very good reason why the MAC address of the replica is not the same as the original? (ie trying to bring up the replica when the original VM is still running - but that causes issues anyway with IP addresses etc).

This isn't really a "Veeam" issue per se. VMware dynamically assigns MAC addresses using an algorithm. A replica VM is still a different VM (thus the different name and UUID) so when you power it on VMware assigns a new MAC address. You should be able to work around this my manually assigning MAC addresses, which you really should for for "license" servers anyway, just to guarantee that they don't change in the future. For example, we've had MAC addresses changes after major ESX/vCenter upgrades (when switching to a new vCenter server) or in cases where we've moved VM's around using manual tools and unregister/register from the command line. Assigning a manual MAC address to those servers eliminates this issue. We use a scheme where the last three octects of the manual MAC address is the HEX equivalent of the last three octets of the IP address. This helps eliminate the possibility of duplicates (it's still possible to have duplicates globally, but unlikely on the same LAN, which is all that really matters for MAC addresses).

[mmonroe]

There really needs to be a complete section in the documentation that discusses a DR restores and the various issues that may need to be dealt with. It could be a work in progress that gets added to as various issues with SQL, AD, Exchange, MAC address, license servers, Citrix and whatnot are encountered.

The formus are great, however, there needs to be a single point that documents the various considerations -and- there are clearly a bunch of them based upon this single forum thread alone.

The whole intent with a product like Veeam is to be able to restore systems and their files. As such the documenation should detail how those process work and their limitations.

[cmdrriker]

I just wanted to post a reply in support of the statement by mmonroe regarding the lack of documentation. This has been my major frustration with Veeam. I expect casual forum style support for a free, open source product, but not for one that is commercially sold. There needs to be more comprehensive formal documentation regarding DR configurations and issues.

[Vitaliy S.]

Hello Will, thank you for the feedback about how to improve our documentation. We do pay attention to the requests of our community members.