Need help with virtual lab & exchange DAG

[adelatorre]

I am having a problem getting my virtual lab working. I have a 2 server DAG for exchange 2010.

Subnet addresses are 10.0.3.61, 10.0.3.62 for mb1 & mb2 respectively.

DAGNetwork02 uses 10.12.3.61, 10.12.3.62.

vmware shows the DAG network ip addresses as the addresses for the machines.

So I get this error when the virtual lab tries to power up:

Code: Select all

6/21/2012 6:52:08 PM          Getting virtual lab configuration
6/21/2012 6:53:23 PM          Starting virtual lab routing engine
6/21/2012 6:53:41 PM          FirstADS - Publishing
6/21/2012 6:53:47 PM          FirstADS - Reconfiguring
6/21/2012 6:53:50 PM          FirstADS - Registering
6/21/2012 6:54:27 PM          FirstADS - Configuring DC
6/21/2012 7:02:49 PM          FirstADS - Powering on
6/21/2012 7:02:51 PM          FirstADS - Heartbeat test
6/21/2012 7:03:06 PM          FirstADS - Running ping test(s)
6/21/2012 7:03:12 PM          FirstADS - Application initialization
6/21/2012 7:03:24 PM          xMB1 - Publishing
6/21/2012 7:03:28 PM          xMB1 - Reconfiguring
6/21/2012 7:03:33 PM          xMB1 - Registering
6/21/2012 7:10:40 PM Fail     xMB1 - Powering on
6/21/2012 7:10:40 PM Fail     Error: No destination network for IP address '10.12.3.61'
6/21/2012 7:10:40 PM          Sending snmp notification
6/21/2012 7:10:41 PM          Sending email report
6/21/2012 7:10:54 PM          xMB1 - Powering off
6/21/2012 7:10:56 PM          xMB1 - Unregistering
6/21/2012 7:10:56 PM          xMB1 - Cleaning up redo logs
6/21/2012 7:10:58 PM          xMB1 - Unpublishing
6/21/2012 7:11:07 PM          FirstADS - Powering off
6/21/2012 7:11:08 PM          FirstADS - Unregistering
6/21/2012 7:11:09 PM          FirstADS - Cleaning up redo logs
6/21/2012 7:11:10 PM          FirstADS - Unpublishing
6/21/2012 7:11:17 PM          Stopping virtual lab routing engine
6/21/2012 7:11:17 PM Fail     Error: No destination network for IP address '10.12.3.61'
6/21/2012 7:11:20 PM          Job finished

ONe other question I had:

Do I need to restore both mailbox servers to do item level restore, or will having just one of them in the application group suffice?

[rbrambley]

Hello,

it appears that your DAGNetwork02 is not properly configured in your vLab. Make sure you create an isolated portgroup in the vLAB network settings that matches the production portgroup. Set the internal vLab interface to match the proper gateway, etc that those vnics in your VMs are already set to use. This is all better explained in the PDF linked in this blog post.

As far as your other question about using both nodes - I believe an Exchange DAG allows a single node to boot isolated and mount all the stores. Assuming this is the case, all you will need is your DC(s) and one DAG member. In fact, if you think in terms of how a DAG is supposed to function when there is a server failover, this is essentially what you will create in the isolated vLab. The difference is you are cold booting only one member instead of failing over. This should cover you for Surebackup Verification and even U-AIR restores. For file level restores it is the same because each DAG member should have a replicated copy of the stores, logs, etc.

Think like this for the backups too. You really only need to backup one node. In fact, backup the passive node so you won't impact prod Exchange with a VMware snapshot and commit too. Exchange 2010 even lets you restore from any member in the DAG, so even "real" restores can be covered by a backup of the passive node.

[Bunce]

In a 2 node dag you need a filesharewitness.

Where is this installed? if only bringing up one node you'll need it to form a quorum, otherwise you'll need to perform some additional steps..

[rbrambley]

In a 2 node dag you need a filesharewitness.

Where is this installed? if only bringing up one node you'll need it to form a quorum, otherwise you'll need to perform some additional steps..

True. I forgot about the Witness. I'm not an everyday Exchange admin, obviously.

But then that answers the question on "do I need the whole DAG to run in a vLab?" In a 2 node DAG yes - unless you perform the additional restore steps, but what about larger DAGs? You probably just need 1 DAG and the Witness. Or choose to config the vLab to leave the single DAG VM running so you can perform the additional restore steps.

Of course a file level restore does not need a vLab or a powered on VM, so backing up a single passive node still makes sense for that job type. The point is that the backup is at the VM level, so admins can choose up front whether to change the application design / vm architecture or change the restore process to account for the application's needs.

[Cokovic]

As far as i remeber from my Exchange DAG tests the filesharewitness is not really mandatory for your lab. Instead a working DC, DNS, a CAS/Hub and at least one mailbox server is needed if you want to get UAir working for example.

[Bunce]

It depends on a number of things and config settings such as DAC mode but upshot is that a dag copy generally won't mount unless it can form a quorum. This is to prevent 'split brain' scenarios where both ends of a disconnected network continue to function independent of each other during an outage. It can be manually overridden though.

Fileshare witness is required wherever there is an even number of nodes, where it acts as the deciding vote.

[rbrambley]

Bunce,

I just came to the same realization with the Exchange 2010 help on Technet.

Correct me if I misunderstand the copied paragraphs below and your obvious Exchange expertise, but, in terms of minimizing the headaches of backing up Exchange DAGs and doing restores with Veeam (vPower, vLabs, U-AIR, etc) odd node DAGs appear to be the way to go. So, therefore backing up a single, passive node of a 3 member DAG would allow the VM to boot and mount by itself (assuming AD, DNS, etc is there too).

A 5 member DAG would require 2 members to be backed up and restored.

Even node DAGs require a witness VM too so

2 member DAG needs 1 VM, 4 needs 2, etc + the witness VM

Architecting the DAG so that at least 1 VM has passive copies of the stores (for 3 node DAGs) then creates the ability to minimize impact to the production mailboxes. Be sure to put the active DAG members on different datastores, hosts, racks / power (failure domains).

Did we just come up with a best practice?

DAGs with an even number of members use the failover cluster's Node and File Share Majority quorum mode, which employs an external witness server that acts as a tie-breaker. In this quorum mode, each DAG member gets a vote. In addition, the witness server is used to provide one DAG member with a weighted vote (e.g., it gets two votes instead of one). The cluster quorum data is stored by default on the system disk of each member of the DAG, and is kept consistent across those disks. However, a copy of the quorum data isn't stored on the witness server. A file on the witness server is used to keep track of which member has the most updated copy of the data, but the witness server doesn't have a copy of the cluster quorum data. In this mode, a majority of the voters (the DAG members plus the witness server) must be operational and able to communicate with each other to maintain quorum. If a majority of the voters can't communicate with each other, the DAG's underlying cluster loses quorum, and the DAG will require administrator intervention to become operational again.

DAGs with an odd number of members use the failover cluster's Node Majority quorum mode. In this mode, each member gets a vote, and each member's local system disk is used to store the cluster quorum data. If the configuration of the DAG changes, that change is reflected across the different disks. The change is only considered to have been committed and made persistent if that change is made to the disks on half the members (rounding down) plus one. For example, in a five-member DAG, the change must be made on two plus one members, or three members total.

Quorum requires a majority of voters to be able to communicate with each other. Consider a DAG that has four members. Because this DAG has an even number of members, an external witness server is used to provide one of the cluster members with a fifth, tie-breaking vote. To maintain a majority of voters (and therefore quorum), at least three voters must be able to communicate with each other. At any time, a maximum of two voters can be offline without disrupting service and data access. If three or more voters are offline, the DAG loses quorum, and service and data access will be disrupted until you resolve the problem.

[Bunce]

Correct me if I misunderstand the copied paragraphs below and your obvious Exchange expertise

No expert here - just come across it before Did a DR simulation last month so the process for DAG switchovers is quite familiar.

in terms of minimizing the headaches of backing up Exchange DAGs and doing restores with Veeam (vPower, vLabs, U-AIR, etc) odd node DAGs appear to be the way to go. So, therefore backing up a single, passive node of a 3 member DAG would allow the VM to boot and mount by itself (assuming AD, DNS, etc is there too).

I think the number of copies in a DAG should really be based of uptime requirements & capacity sizing - as well as site topology. While an odd number of copies does do away with the FSW requirement, it really only has relevance in a DR scenario and doesn't add alot of additional complexity to the switchout steps required given everything else that would be going on at the time

A 5 member DAG would require 2 members to be backed up and restored.

Even node DAGs require a witness VM too so

2 member DAG needs 1 VM, 4 needs 2, etc + the witness VM

Almost. a 5 Member DAG would require 3 available copies to form a quorom (majority). 2 wouldn't cut it as XGE would assume the other 3 are online, talking to each other and have formed a quorum - so it would dismount these 2 to avoid Split Brain.

Upshot is the the number of available copied required, for various DAG configs:

• 8 Node DAG -> Requires 4 copies + FSW
• 7 Node Dag -> Requires 4 copies
• 6 Node Dag -> Requires 3 copies + FSW
• 5 Node Dag -> Requires 3 copies
• 4 Node Dag -> Requires 2 copies + FSW
• 3 Node Dag -> Requires 2 copies
• 2 Node Dag -> Requires 1 copies + FSW
• 1 Node Dag -> Is a happy chappy.

So to be able to bring up a live environment automatically using Veeam (vPower etc), you need the number of servers listed above to form the quorum- BUT thats even only possible if DAC mode is disabled (not recommended).

Even with DAC mode disabled it may still not work given Veeam brings the servers up sequentially - I'm not sure of the tolerance period, but if the copies are not all up with network connectivity to each other within a short period then Exchange will assume they are isolated and dismount..

Again, there are manual steps that can be run to force a DAG failover after bootup, so I assume the powershell commands could be innjected into a lab.

To be honest - I've never really tried it. We have a 3 DAG cluster (across two sites), and only backup a single copy (plus the CAS). Its an active copy, but backed up at night, so there's no impact on production as long as it doesn't overlap with the XGE maintenance window (default between 1am and 5am from memory).

For restores I just bring up this copy and use other tools to mount the EDB (for the time being anywy ) Even this is rarely required as the built in exchange retention policy (aka dumpster) provides us sufficient short term retention to cover 99% of helpdesk calls..

So - after all that.. What the best practice? My view is that DAG's themselves effectively give you a DR solution built-in. By definition you have an active/up-to-date copy - presumably at another locaton, ready to go -> one of those (few) areas that Veeam doesn't proove to be as useful for replication

For retrieval of longer term items - Veeam is still a must, but you're going to struggle to get a live environment automatically restored and DB's mounted into a lab (for Surebackup etc) in anything but a standalone or simple DAG environment, unless you're willing to backup the majority (quorum) of your DAG copies.

In that case it doesn't really matter wether you have an odd or even number of copies as the 'deciding' vote is still effectively an additional server to bring up -albeit the FSW or (n/2+1)'th DAG copy - unless you have your FSW on a server that is already being bought up in the lab (eg DC or CAS).

[rbrambley]

So - after all that.. What the best practice? My view is that DAG's themselves effectively give you a DR solution built-in. By definition you have an active/up-to-date copy - presumably at another locaton, ready to go -> one of those (few) areas that Veeam doesn't proove to be as useful for replication

For retrieval of longer term items - Veeam is still a must, but you're going to struggle to get a live environment automatically restored and DB's mounted into a lab (for Surebackup etc) in anything but a standalone or simple DAG environment, unless you're willing to backup the majority (quorum) of your DAG copies.

In that case it doesn't really matter wether you have an odd or even number of copies as the 'deciding' vote is still effectively an additional server to bring up -albeit the FSW or (n/2+1)'th DAG copy - unless you have your FSW on a server that is already being bought up in the lab (eg DC or CAS).

Well said! You really have to think about your layout of your DAG in order to use in a Veeam Surebackup job or vLab.

[NiallDES]

I am experiencing the same problem as this, i.e. a 3 member DAG split accross 3 geographical areas. Each DAG member is being backed up by a seperate installation of Veeam B&R 6.1 - therefore I can only bring 1 DAG member back in each location and as a result can't get the mailbox database to mount.

Could anyone shed any light on how to get around this issue? Is it something that is addressed in version 6.5? thanks

[tsightler]

In 6.5 we will have the ability to restore individual mail items directly from the backup without requiring a virtual lab so this architecture won't be an issue.

[NiallDES]

Great, thanks!