Exchange 2010 DAG backup ?

[janatecx]

We backup both Nodes as the customers requires a full vm backup at both sites.

We did however adjust the backup schedule. One node backup starts at 8P and the other around midnight. Just to make sure both vm's aren't being snapshot at the same time.
For Database backup, one node should suffice.

[dellock6]

If you want to be sure the two jobs do not overlap, you can also schedule only the first, and have the second one chained to the first, so it will only start when the first is completed.

Luca.

[veremin]

you can also schedule only the first

Or make corresponding changes with the number of concurrent tasks that can be processed by repository or backup proxy. Thanks.

[janatecx]

Ok, After running a week without any problem.

The problem occured again last night. Does anyone have more info on this issue, or why this could happen suddenly?

Many thanks in advance.

Jan

[Andreas Neufert]

Some ideas and tips from another discussion about DAG backup here in the Forum. Maybe they are helpful here as well.

My very personal list of tips to optimize Exchange DAG backups at this point are:

1) Increase the DAG heartbeat time (no reboot needed)
cluster /prop SameSubnetDelay=2000:DWORD
cluster /prop CrossSubnetDelay=4000:DWORD
cluster /prop CrossSubnetThreshold=10:DWORD
cluster /prop SameSubnetThreshold=10:DWORD
2) Use new Veeam Storage Snapshot Feature (Lefthand/3PAR/VSA) if you can (after v7 release)
3) Backup only an DAG member that hold only inactive databases (no cluster failover because of no active databases) (Logfile Truncation will be replicated by Exchange in whole DAG)
4) Try to avoid any changes at the backup time window (User, Background processes, Antivirus, ....). Also try to avoid that on all LUNs on the storage System itself (faster writes at snapshot commit).
5) Use Forward Incremental or if you Need space Forward incremental with daily transform into rollbacks. Do not use Reverse Incremental (Reverse Incremental took a bit longer than the other backup methods)
6) Use new parallel Prozessing with enought ressources to backup all of your disks at the same time (after v7 release)
7) Use Direct SAN Mode with minimal needed disks connected at selected Proxy. If not possible use NBD mode with 10GbE. (Do not run Proxy in Autoselect mode). Disable VDDK Logging for Direct SAN Mode if your backups themself run stable (ask support for the registry key and consequences).
8 ) Use actual VMware Versions (newest VADP/VDDK Kits with a lot of updates in it) and actual Veeam Versions (newer VDDK Integration)
9) Use at Minimum VMware vSphere 5.0 because of changes in the snapshot places and Background things.
10) User Fast disks for all of the VM disks (also the OS disk)
11) Less disks can help to reduce snapshot creation (and commit?) time.
12) Check your vcenter load and optimize it (or use direct ESX(i) Connections for Veeam VM selection.
13) Maybe another Option (unsupported from VMware) is to change VM Setting snapshot.maxConsolidateTime = "1" (see discussion above).

Hope that this informations can be helpful.

CU Andy

[janatecx]

Hi All,

The above steps have resolved the issue.
I changed the job this weekend from reversed incremental to forward incremental, and the job finishes almost 50% faster, en the DAG cluster remains online.

Thank you for the info.

[Andreas Neufert]

My updated list of tips, if you run into Exchange 2010 backup challenges:
1) Increase the DAG heartbeat time (no reboot needed) to avoid cluster failover
cluster /prop SameSubnetDelay=2000:DWORD
cluster /prop CrossSubnetDelay=4000:DWORD
cluster /prop CrossSubnetThreshold=10:DWORD
cluster /prop SameSubnetThreshold=10:DWORD
2) Use new Veeam Storage Snapshot Feature (Lefthand/3PAR/VSA) if you can (after v7 release) => Reduces Snapshot Lifetime to some seconds => No load and problems at commit because of less data.
3) If you have problems with cluster failover at Backup, one option is to backup DAG member(s) that hold only inactive databases (no cluster failover because of no active databases) (Logfile Truncation will be replicated by Exchange in whole DAG)
4) To reduce Snapshot commit time (and to reduce data in the snapshot), try to avoid any changes at the backup time window (User, Background processes, Antivirus, ....). Also try to avoid that on all LUNs on the storage System itself (faster writes at snapshot commit).
5) If you can not avoid many changes on block level at your backupwindow? Use Forward Incremental or if you need space Forward incremental with daily transform into rollbacks. Reverse Incremental took a bit longer than the other backup methods => longer snapshot livetime => more changes in the Snapshot to commit
6) To reduce Snapshot lifetime and reduce amount of data to snapshot commit, use new parallel processing with enought ressources to backup all of your disks at the same time (after v7 release)
7) To reduce backup time window and snapshot lifetime, use Direct SAN Mode with minimal needed disks connected at selected Proxy. If not possible use NBD mode with 10GbE. (Do not run Proxy in Autoselect mode). Disable VDDK Logging for Direct SAN Mode if your backups themself run stable (ask support for the registry key and consequences).
8 ) Use actual VMware Versions (newest VADP/VDDK Kits with a lot of updates in it) and actual Veeam Versions (newer VDDK Integration)
9) Use at Minimum VMware vSphere 5.0 because of changes in the snapshot places and Background things.
10) If you still facing problems, use faster disks for all of the VM disks (also the OS disk!!!)
11) Less VM disks can help to reduce snapshot creation (and commit?) time.
12) To avoid VSS timeouts (hardcoded 20 seconds at Exchange VSS writer), Check your vcenter load and optimize it (or use direct ESX(i) Connections for Veeam VM selection, so that the snapshot creation took less time.
13) If you facing VM downtime because of Snapshot commit, maybe another Option (unsupported from VMware) is to change VM Setting snapshot.maxConsolidateTime = "1" (in sconds) (see discussion above).
14) If you see Exchange VSS Timeout EventLog 1296 => Change Log setting => Set-StorageGroup -Identity "<yourstoragegroup>" -CircularLoggingEnabled $false
15) VSS Timeout Problems => Add more CPU/Memory to the VM

[Peejay62]

Very interesting thread this is. I am currently preparing everything to bring our exchange 2010 environment under Veeam. Already did various tests on a very small test-environment and that all works fine. But the production environment is of another magnitude. We have a 6 node DAG. Each node holding appx 9 tb of data, based on 1 and 1.5tb luns. The nodes each have 24gb mem and 4vcpu's. we run vsphere 5.1 and Veeam 7.
I don't have any reference in determining us having a small or big environment and I am trying to put all the things talked about in this thread into perspective. Obviously, i cannot do any testing in our live production environment so I am kind of swimming dry.Anybody with a similar environment wants to share on this...

[Jonathan]

You could also go really wild and not back up the mailbox stores at all. Just configure your deleted items retention times to whatever period you normally keep your backups. This enables your users to restore any item they deleted themselves. Microsoft supports circular logging for DAGs of 3 members or more, so log files will be truncated automatically so long as the databases are in sync.

We've been doing this since (the year) 2010 and so far it has worked flawlessly.

[chrisdearden]

You could also go really wild and not back up the mailbox stores at all. Just configure your deleted items retention times to whatever period you normally keep your backups. This enables your users to restore any item they deleted themselves. Microsoft supports circular logging for DAGs of 3 members or more, so log files will be truncated automatically so long as the databases are in sync.

We've been doing this since (the year) 2010 and so far it has worked flawlessly.

I assume each node is on a different storage / site?

[tsightler]

I'm also curious if you have any lagged nodes in the DAG group?

[Jonathan]

I assume each node is on a different storage / site?

Yup. Two VM nodes on a fibre channel array, one VM node in a separate site half a mile away on a different vSphere cluster existing of heavy Supermicro machines with local storage (SATA/SSD). All three nodes run CAS, Mailbox and Hub transport roles. Client access through a loadbalancer.org R16 cluster (MAPI) and a TMG 2010 cluster (ActiveSync\OWA\Outlook Anywhere).

[Jonathan]

I'm also curious if you have any lagged nodes in the DAG group?

No, but I'm considering it for my new EX2013 setup.

[hscox030]

I just virtualized my two Exchange 2010 mailbox servers so I'm setting up my first Veeam Exchange backup job and I have a question. I have two mailbox, CAS, Hub servers with 5 databases per server. Each server has a database copy of the other servers databases so each has 5 activated DB's and 5 DB Copies.

So my question is when I setup the job do I exclude the DB Copy drives (databases/logs) or do I include them in the backup job of one server? I plan to backup each of the two Exchange Servers. I am currently using Backup & Replication 6.5.

Thank you

[Vitaliy S.]

I believe it's fine to backup both databases, please review this topic for a detailed information on Exchange backup best practices.

[janatecx]

You can also upgrade to Veeam 7 with the new patch installed:

Added ability for application-aware processing logic to detect passive Microsoft Exchange DAG database present on the VM, and process it accordingly.
Added support for Exchange CCR clusters.

The complete list of improvements you can find here and also here you can download the Patch itself: http://www.veeam.com/kb1831

[atta]

Hello,

We have 1 site with 2 DAG members. Now if the one of them crush how can I restore full server and make sync after restoring. Does Veeam know like in (AD restore) to keep sync alive after recovery or we need to make addiotional steps to amke everything work?

We are performing image backup

[veremin]

Hi, Atta,

Your post has been merged into existing discussion regarding Exchange DAG backup and recovery. For more information, please, see answers provided above, or more specifically, this one - it seems to contain information you're after.

Thanks.

[jasonwhand]

We still have the Databases failover to the secondary DAG partner every night during the VMware snapshot consolidation process and we are using fast storage and direct SAN backup. Anyone hit this issue with all of the recommended changes like we are?

[Gostev]

Hi, Jason!

Here is the most comprehensive guide of making this work that I'm aware of, created by one of our Solution Architects > Exchange (DAG) VMware Backups: Updated list of tips and tricks for Veeam Backup & Replication

Hope this helps!
Thanks.

[hscox030]

I am curious how other people are doing their Exchange 2010/2013 Veeam backups. I have tried Veeam suggestions related to snapshots (except for the vmx file change) but still have issues with a database failover during the snapshot creation. I have decided to backup my passive nodes but my problem is if I do a snapshot, backup my databases then integrate the snapshots I can not get my snapshot files integrated if I keep the database copies running. The DB Copies are writing so much data to the snapshot files that I cannot integrate the snapshots fast enough. I end up with a snapshot integration that can never finish. Is anyone else running into a similar issue? Currently I am turning off the DB Copies during the backup process then turning them back on after the job completes. Does anyone having any thoughts/suggestions/etc.?

Thank you for your time.

[foggy]

Sean, please review the hint provided in the above-referred blog post, should give you some thoughts.

[mobdoc]

Hi,

I need some advise with regards to our new Exchange 2013

scenario
we have Exchange 1 (All Exchange Roles) and Exchange 2(All Exchange Roles), its a load balance and replication

now since the two are mirror, do I need to include this two exchange in my Veeam Backup Jobs or only one of them?

What is the best practice for this scenario.

Regards,


Oliver

[foggy]

Oliver, please review hints discussed in this thread and feel free to ask any additional questions, if required.

[repa]

Hi,

how do you backup DAG with Veeam? I red, that a completly passive DAG Member should be used to backup the Databases. Using this method, no impact on the production should appear.

But how do you drop the transaction logs on the active DAG members?

Thanks for some hints...

[foggy]

Pascal, the thread above and this post are likely to answer your question.

[Andreas Neufert]

Transaction Log Truncating is replicated through DAG. So if you truncate on the passive Database, it is replicated to the other nodes. This is a core functionality from Exchange DAG (not from Veeam).
Here you can find a lot of informations:
http://andyandthevms.com/exchange-dag-v ... plication/

Just an additional small comment for all the others.
Lately some of our customers contacted me and said that round about VMware SnapShot commit they face DAG cluster failovers.
When I analysed it, the reason was NFS Datastores are used with HotAdd backup mode. This is ugly in 2 ways:

1) NFS is not supported with Exchange (and there is a technical reason behind that) => see my blog above.
2) HotAdd with NFS can cause some VM stunns at DiskHotAdd release => See my Blog and VMware KB.

=> Change Datastore for Exchange to a block media or if you want to keep it really on NFS, use NBD-Network mode or the new v9 DirectNFS backup mode.

[btmaus]

My updated list of tips, if you run into Exchange 2010 backup challenges:
1) Increase the DAG heartbeat time (no reboot needed) to avoid cluster failover
cluster /prop SameSubnetDelay=2000:DWORD
cluster /prop CrossSubnetDelay=4000:DWORD
cluster /prop CrossSubnetThreshold=10:DWORD
cluster /prop SameSubnetThreshold=10:DWORD

I'm looking at moving our Exchange 2010 DAG backups over to Veeam now. Should I apply these cluster timeout settings on both the Active and Passive node?

[Andreas Neufert]

It is a cluster wide setting. So per DAG Group.
To be save test it with
cluster /prop
on all cluster members.

Btw...my blog post was updated two days ago:
http://andyandthevms.com/exchange-dag-v ... plication/

[conextra]

we are about to implement an Exchange 2013 DAG. our current Backup strategy is to backup the Domain Controller and the Exchange Server with Veeam.
as for DAG members, it is possible to backup both, the active and passive member with Veeam.
in the Event of a disaster i see a lot of Posts to recover a lost DAG member with Exchange Setup recover Switch. i am wondering, if the following case will result in any success:
- Recover both, active and passive node
- DR happens and active node gets lost
- Exchange will continue to operate on the passive node
- Restore lost node with Veeam Backup
- Boot restored node -> what will happen then.

anyone any experience with this Situation. it seems to me like a very common Scenario in a 2 node DAG.