Issue with VMware & Exchange 2010 DAG

[vmmatty]

I realize up front that this is a VMware issue and not a Veeam issue, but am posting here since I'm curious what others are seeing. When taking a snapshot in vSphere 4, the VM usually drops a single ping as part of the process. This happens whether using guest quiesce or not and goes on for longer if you snapshot memory. For most VMs this isn't really an issue.

My issue relates to trying to use Veeam to backup Exchange 2010 VMs that are in a DAG. I'm seeing situations where this ping drop will cause the Windows Failover Cluster to perceive a failure and cause the databases to fail over to the other node. I'm not seeing this in all cases but I am definitely seeing it happen. In one particular case I can cause it to happen every time, and others I have to dig back through historical cluster event logs to find that it did happen a few times over the last 3 months. Exchange is resilient enough such that most of the time you don't even realize that it's happened.

Again I know this isn't a Veeam problem - I can reproduce it by just taking a regular VMware snapshot. I'm wondering what other folks have seen with this configuration and if they're having issues with using a VMware backup product like Veeam. Anyone else seeing this issue?

[Gostev]

Have not heard anyone else mentioning similar issues here, but there definitely were customers saying they are backing up Exchange 2010 DAG successfully...

I would expect Exchange to provide some tolerance settings for failover (e.g. only failover if node does not respond for more than 10 seconds). Did you try to dig in this direction? May be they are changed from default values to minimal tolerance by someone else?

[Bunce]

We've had this occur once but haven't yet had a chance to look at the available config options. I know you can setup dedicated dag networks however doubt this would help as all pings would be dropping.

Also need to setup a notification to inform of the failover but there doesn't appear to be anything inbuilt so may have to use a 3rd party monitoring tool or scripting.

[joergr]

You are right, sometimes, DAG fails over during vmware snapshot commit, at my site let´s say about once a month. Does not hurt too much, but it hurts when you configured ip failover without ex2010sp1, then you have to re-activate the cluster network to work around it - anyhow, this is vmware snapshot related, when you commit a snapshot the ping drops about one time and sometimes some cluster services get exactly that one and do a failover. Nothing you can do about that

best regards,
Joerg

[vmmatty]

It sounds like multiple people are seeing this issue so I'm surprised I haven't heard more about it.

It's true that DAG failover can be seamless and we've had this happen and we didn't even know about it until several days later. That said I don't think we can just say that this is one of those things that happens and accept it.

Since I don't see this frequently except for one specific case I'm going to investigate further to see if I can figure out why. It's possible there is an issue with the file share witness that is causing the failover when the snapshot occurs.

I'm curious on Veeam's take on this issue actually. Technically speaking Microsoft does not support the use of snapshots on a virtualized Exchange server, and yet Veeam relies on VMware snapshots to perform backups. Not that this is the first time that vendors don't agree on what should be supported, but I'm more curious if Veeam has an official position on that. I'd really like to continue to use and recommend Veeam but based on this issue and the MS support statement I'm not sure I can continue to recommend it for Exchange backups, or at least as the only method of backing up Exchange.

Not bashing Veeam - I think it's a great product and I use it everywhere, but going forward I'm less likely to use it against virtualized Exchange servers unless someone can convince me otherwise.

[joergr]

Hi,

Exchange 2010 with VMware IS fully supported. But the rumors saying the opposite die very hard i guess

So let´s have proof.

Please read first this
http://msexchangeteam.com/archive/2010/ ... 56851.aspx

and then this
http://www.windowsservercatalog.com/svv ... e=svvp.htm

Proof.

The only thing MS wants is that you don´t combine VMware HA with Exchange HA, so just disable HA in the vmware cluster if you even have one.

best regards,
Joerg

[vmmatty]

It's not an issue of Exchange support on VMware. What I'm referring to are the features that are supported when running Exchange in a VM.

If you run Exchange 2010 through the SVVP Wizard (from the link you posted), you get the following features that are supported and unsupported:

[These features are supported in this configuration.] Supported features: Anti-Virus, Back-up Software, Virtual Machine Management Software, Database Availability Groups (DAGs), Virtual Processors

[These features are not supported in this configuration.] Unsupported features: Unified Messaging, Dynamically Expanding Virtual Disks, Virtual disks that use differencing or delta mechanisms, Virtual Machine Snap Shots

What I'm saying is that technically Microsoft doesn't support taking snapshots of your Exchange virtual machine, and snapshots are required for Veeam to take a backup. So there's a supportability disconnect there and I'm curious what Veeam's take is on this. In the past I've disregarded this and used Veeam (and other competing virtual machine backup products) for VM backup, but now that I'm seeing this issue with Exchange 2010s using DAGs I'm starting to think I should rethink that.

[joergr]

Well I think no software vendor in the world would fully support virtual snapshots, because let´s be honest, a snapshot will freeze a machine for about a second. But as only one system can read the same blocks on a storage device at the same time, there is absolutely NO solution on the market today (esx / esxi / xen / hyperv /parallels / virtuozo / sun) which could do otherwise. You simply have no choice if you want to hot backup a full vm. And it is not related to VEEAM, it is related to the vendor (VMware / Citrix / Microsoft / and so on). The only way to use a FULLY by microsoft supported and approved backup technique of an Exchange Server is to use some agent inside the guest (for example windows backup or microsoft dpm). But then again, these systems will do a "soft freeze" too, by triggering the shadow copy provider inside the guest and forcing a file system and database harmonization. Ofcourse this produces no outtage but i even saw problems with these.

Me, personally, i backup a huge DAG Cluster with veeam every night.

Best regards,
Joeg

PS i just ran the wizard with Exchange 2010 and Hyper-V and guess what is NOT supported: Yeah. Snaphots.

[Gostev]

Exchange 2010 DAG is essentially 1.0 feature, so I guess it is expected that it lacks absolutely essential settings around failover trigger tolerance...

No, scratch that - I still don't understand how you can release software which does automated failover without providing at least minimum settings for failover tolerance customization... or at least a registry key... and instead perform failover immediately all the time (read - every single network glitch will cause failover).

[vmmatty]

I don't want to get into a discussion about how Microsoft has unrealistic guidance or anything like that. Also not trying to get into a Veeam vs. competitors discussion either.

I get that Veeam has put in a lot of great effort into making the product Exchange aware and give the ability to restore items in Exchange, etc. I think all of that is great. I think what annoyed me to some extent was a tweet from the Veeam Twitter account. Had I not seen this tweet I probably wouldn't have brought this up today:

http://twitter.com/veeam/status/4272025476530176
You really need to be careful with Exchange Backup & Recovery or you become unsupported: http://bit.ly/c6yAgJ <- ask your vendor!

Following Veeam's advice, if I adhere to Microsoft's policies for Exchange then using Veeam Backup to backup my Exchange environment makes me unsupported. It is clearly stated in the SVVP that snapshots aren't supported and I don't see how that point can be argued, regardless of whether or not you agree with Microsoft's stance on this issue. Do we agree on that?

Regarding the comment that DAGs are 1.0 - while that is technically true, I don't fully agree that it's brand new technology but that's a debate for another day. And regardless the issue we're talking about here has to do with Failover Clustering and not really with Exchange specifically. I agree that having tolerance parameters would be nice, and maybe they exist, but I haven't seen them.

Again not trying to start a pissing contest here, I'm trying to solve the problem and/or figure out if I can continue to use Veeam to backup Exchange VMs or any VMs that use Failover Clustering. I have no plans to stop using Veeam to backup all other VMs. I'm a huge Veeam fan and have deployed it at dozens of clients.

[Gostev]

It is clearly stated in the SVVP that snapshots aren't supported and I don't see how that point can be argued, regardless of whether or not you agree with Microsoft's stance on this issue. Do we agree on that?

No, I don't agree, this does not make any sense, because Exchange 2003 and above is designed to be backed up using VSS snapshots (read the link you have referenced). How snapshots can be not supported then? Can you please provide direct link to the statement that snapshots are not supported, may be it goes into some more details.

I do agree that "naked" VMware snapshots are not supported (seen SQL and Exchange corruption reported many times with those). But this is definitely not what Veeam Backup is doing.

[vmmatty]

I don't want to turn this into an argument here guys. No reason for anyone to get defensive. I'm on these forums because I support Veeam and use the product but want to make sure I'm doing right by my clients.

No one is arguing the point about the support for VSS snapshots. What Microsoft is stating in the SVVP is that virtual machine snapshots are not supported. You can run through the SVVP wizard yourself and see the results:

http://www.windowsservercatalog.com/svv ... wizard.htm
Unsupported features: Unified Messaging, Dynamically Expanding Virtual Disks, Virtual disks that use differencing or delta mechanisms, Virtual Machine Snap Shots

Microsoft doesn't make the distinction between "naked' snapshots and snapshots that leverage application aware VSS. I'm not sure how else to interpret the SVVP other than to say that virtual machine snapshots of any kind are not supported. Veeam is telling me to follow the vendor's advice regarding backup and even if I use compliant VSS (as stated in the link in that tweet) it is still unsupported because a virtual machine snapshot is required.

Microsoft has some wacky support policies (as does VMware), I don't think anyone is arguing that. But I think the wording in the SVVP is pretty clear even if none of us agree with it.

[Gostev]

Microsoft doesn't make the distinction between "naked' snapshots and snapshots that leverage application aware VSS.

That's the thing, Microsoft absolutely DOES make this distinction, and clearly states that the issue comes exclusively from the fact of (naked) virtual machine snapshots not being application-aware!

In the link you have referenced, clicking "more information" will get you here:
http://go.microsoft.com/fwlink/?LinkId=124624

Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments

Some hypervisors include features for taking snapshots of virtual machines. Virtual machine snapshots capture the state of a virtual machine while it is running. This feature enables you to take multiple snapshots of a virtual machine and then revert the virtual machine to any of the previous states by applying a snapshot to the virtual machine. However, virtual machine snapshots are not application-aware, and using them can have unintended and unexpected consequences for a server application that maintains state data, such as Exchange Server. As a result, making virtual machine snapshots of an Exchange guest virtual machine is not supported.

This is the actual reason why we implemented application-aware processing - to ensure snapshots we are making for backup purposes are application-aware. And mind you, we did it not just because this is "unsupported" for no good reason... the issue with "naked" VMware snapshots is very real, which I've been tirelessly saying for 2 years now, in numerous topics on this very forum about "why use Veeam VSS" and "how is it better than VMware Tools quiescence".

[vmmatty]

I don't want to split hairs here, but I think you're making a bit of an assumption here regarding support. We agree that Microsoft does not support the use of naked snapshots, but you're making a leap by saying that Microsoft does support virtual machine snapshots as long as they're application aware. It's a logical assumption, but I don't see anything in Microsoft's support policies that states that the use of application aware virtual machine snapshots are in fact supported. All I see is them saying that virtual machine snapshots are not supported.

If you can point me to an official support document that explicitly states as such, I'd be a very happy person. But as of now I cannot find that document. Saying that they don't support a feature because it is not application aware does not automatically mean that they do support it if the feature becomes application aware.

There seems to be a lot of bickering around Microsoft/VMware support in the industry right now. Rather than add to that here, I think we should work together to try to get an official answer on this. I work for a Microsoft partner and can reach out to my reps. If you can do the same and share what you hear I think that would benefit everyone.

Going back to the original purpose of this thread - I'm having a problem taking a Veeam backup of an Exchange VM that is part of a DAG. I suspect that you consider this a VMware issue and not a Veeam issue, correct? So should I be seeking VMware support instead of Veeam support here?

[Gostev]

It's a logical assumption, but I don't see anything in Microsoft's support policies that states that the use of application aware virtual machine snapshots are in fact supported.

Well, that's because no virtualization vendor provides application-aware snapshots today.

Going back to the original purpose of this thread - I'm having a problem taking a Veeam backup of an Exchange VM that is part of a DAG. I suspect that you consider this a VMware issue and not a Veeam issue, correct? So should I be seeking VMware support instead of Veeam support here?

Neither VMware, nor Veeam. I consider this to be bad feature design on Microsoft part (when single ping drop causes failover), so you should seek Microsoft support. Really, this design issue has little to deal with virtualization, and I am guessing can easily be reproduced by pulling out network cable for a second, imitating network glitch.

[Bunce]

The DAG's rely on Windows Clustering, so not XGE specific per se.

I've managed to find the following which seems to indicate some FCS delay parameters that can be set. Haven't tested them myself yet:
http://technet.microsoft.com/en-us/libr ... S.10).aspx
http://www.extropy.com/forums/knowledge ... g-over-wan

[joergr]

Anton is right, the failover is simply triggered at rare cases when the network connectivity is gone for a very short period. Can be easily reproduces on physical machine, when you pull the cable for a sec.

best regards,
Joerg

[vmmatty]

I agree this is a cluster issue, and my issue in particular is strange because I can reproduce it every single time I take a snapshot. This is not the case with the many other virtualized clusters I've deployed (including others at the same customer).

Microsoft clusters are designed so that a brief loss in heartbeat causes a failover. I don't think that is necessarily a bad thing. Tweaking those values might work around the issue a bit but could hurt the time it takes for a cluster to detect a failure. If high levels of uptime are required then even a few extra seconds could be bad.

I'm also not sure that comparing this to a brief network cable pull is really fair since that isn't something that would typically happen in a physical server. Yet a brief network drop is common when taking a virtual machine snapshot so it becomes a much more obvious problem.

[vmmatty]

Quick update - modifying the values that Bunce linked to does appear to mostly work around the problem. It still does happen but much less frequently. Again though that is in an environment where I can make it happen every single time when using default values. I'm using default values in my lab and I can only make this happen occasionally, so I think the issue might be worse at this one location.

I'm really curious how many other folks are seeing this issue. No one from Veeam saw this when testing Backup 4.0 or 5.0? I'm curious if you tested with a DAG or with a single mailbox server. And if it was a DAG, did you have large databases with large VMDK files attached?

I'm thinking that this becomes more of an issue if the VM has large VMDK files attached as the metadata updates required during a snapshot might freeze the VM for a longer period of time than if it only has a few smaller VMDK files. Thoughts?

[PFSIT]

I've seen this with an Exchange 2007 CCR (continuous cluster replication), its actually what i'm trying to get resolved at the moment. I'm possibly in the minority though since i'm using 2007 and not 2010.
The next stage,as suggested by joergr on my on thread (http://www.veeam.com/forums/viewtopic.php?f=2&t=5644) is to try and adjust the threshold values..

[joergr]

Yes, if you adjust the values to the max possible as i linked in your thread it will for certain appear much less than before. Secondly you might want to consider using only ESXi 4.1 for snapshots because vmware has improved very much with this release - and doing your backups off work hours thus the snapshot won´t grow as much as it would during high load workday access.

Me personally i use ex2010 DAG thus i don´t care that much if failover occurs but just for research i have been monitoring the failover cases the last weeks and since i adjusted the timeout values and also moved the backup window to deep night - it appears max once in a month.

best regards
Joerg

[StanO]

I've begun trying to resolve this issue as well, though am taking a slightly different tack. I noticed from the event logs that my cluster fail-over occurrences were preceded by events from time-service or kernel-general indicating that a time change had occurred (and sometimes on the order of several seconds). Doing some digging turned up that default behavior for snapshots is that the VM guest OS time is synced via VMware tools at snapshot creation or removal (http://kb.vmware.com/kb/1318) regardless of whether VMware Tools Time Sync is checked or not. It could be that this is the reason for the cluster fail-over rather than a well timed missed ping.

I've adjusted the time-syncing on my Exchange (2007) cluster VM to rely solely on VMware tools (with my ESX host using NTP to an external time source) and have had some success so far in performing Veeam backups without incurring a failover (or removal from the cluster). Granted, I have a small sample size at this point (only 3 runs), but figured I would throw this out as a possible cause and perhaps others can try as well and we can collectively get a larger sample size.

Stan

[StanO]

I've got over 20 successful runs in now without a cluster eviction when time-service (Windows Time) is not allowed to interfere with time syncing performed by VMware Tools (Kernel-General sourced events in System Log). No changes were made to the heartbeat or DNS settings.

I'm confident enough the issue is resolved that I'm going to let the VM node run as the active CCR node for a week before bringing the other cluster member into the virtual environment (and retire that nasty Agent for Microsoft Exchange license without shedding a tear). If anybody can corroborate my findings, I'd like to do it sooner.

[Bunce]

Interesting - good find. I've got a few of those as well..

Jees - I didn't know about that KB article... This would apply to any VM's which are backed up since they are snapshotted, and we sync all VM's to the DC using Windows Time and not to the hosts.

The Hosts also sync to the DC via NTP but I've noticed its pretty erratic - might have to look into this a bit.

Thanks for the post.

[joergr]

This is extremely interesting. If that is the main causer you found out something really tricky. Please keep us updated on your research.

best regards,
Joerg

[StanO]

No such luck. The very next backup after accumulating a day worth of changes resulted in a lossy failover...

[vmmatty]

I reached out to some contacts at Microsoft regarding my question about whether or not Microsoft supports virtual machine snapshots that are application aware. The answer I got was basically that they do not support virtual machine snapshots of any kind, regardless of whether or not they are application aware. They stated that the vendor supplying the application aware snapshot would be responsible for providing support for Exchange should there be any issues caused by the snapshot.

I'm curious if someone at Veeam can reach out to your Microsoft contacts and see if you can get this confirmed. I wouldn't think Veeam would provide support for Exchange in the event of a issue caused by a snapshot. They implied that the backup vendor may have relationship with Microsoft where they can get support if that were to happen.

Anton/Joergr - Can you confirm what I've been told by speaking to your Microsoft reps? Any info you can get would be helpful to the entire community.

[Gostev]

I will try to find out who on Microsoft side we can talk too... but obviously, right now is the worst time to discuss anything around Exchange and VMware with Microsoft, because of the whole Microsoft/VMware battle around Exchange 2010 and VMware HA that is going (I am sure you are following the story too).

[vmmatty]

Yep, I'm following the nonsense (I was quoted in the article that was linked earlier) between Microsoft and VMware. It doesn't help anyone in the industry when they have these battles since we (the customers) are the ones who suffer. But my opinion is to always try to stick with what is supported when dealing with your tier 1 applications like Exchange. That's why I reached out to my Microsoft partner rep to try to get an answer on the support.

At the end of the day if Microsoft says that they won't support Exchange if you use virtual machine snapshots then I probably won't use Veeam to backup my Exchange VMs. I'll still use Veeam to back up pretty much everything else though since I've had great success with the product since the 3.0 days. I hope to hear a better answer though because I'd much rather dump agent level backup in favor of using Veeam exclusively, but I won't if Microsoft is firm about not supporting it.

Thanks as always Anton, looking forward to whatever you hear from your Microsoft reps.

[tsightler]

At the end of the day if Microsoft says that they won't support Exchange if you use virtual machine snapshots then I probably won't use Veeam to backup my Exchange VMs. I'll still use Veeam to back up pretty much everything else though since I've had great success with the product since the 3.0 days. I hope to hear a better answer though because I'd much rather dump agent level backup in favor of using Veeam exclusively, but I won't if Microsoft is firm about not supporting it.

Microsoft lays out very specific requirements for VSS Exchange backups (2007 requirements are documented at http://technet.microsoft.com/en-us/libr ... 80%29.aspx). I think Veeam would not meet the requirements mainly due to the lack of a full integrity check prior to purging logs (although you could script an integrity check with SureBackup).

Microsoft's support policy for any vendor that uses VSS (which is their recommended method) is quite clear, and pushes the responsibility to the third-party vendor. This is not unique to Veeam. I believe that most modern agent based Exchange backups also use VSS. Here's their official statement from http://support.microsoft.com/?kbid=841696:

Volume Shadow Copy Service (VSS) is an infrastructure where multiple third-party storage management programs, business programs, and hardware providers cooperate in the creation and management of shadow copies.

Because VSS is an API that is licensed and built upon by these third parties, Microsoft makes no warranty or recommendation on the performance or reliability of the third-party solution. Microsoft relies on partners to offer VSS-enabled solutions and to provide support for them. We recommend that customers use VSS-enabled backup products.

When designed and implemented correctly, VSS-enabled products help make sure that synchronization occurs between the server program, the backup program, and the Windows storage subsystem. VSS-enabled products help make sure that pending writes are held while the backup is created. VSS-enabled products help make sure that system state information is accurately captured. This is especially true when backing up Exchange, SQL Server, the Microsoft Cluster Service (MSCS), and the Active Directory directory service.

So effectively, unless you're using a backup product from Microsoft, you're pretty at the mercy of the 3rd party agent vendor anyway. At least with SureBackup you could script up a test to verify the recoverability of your Exchange server every night. Having an integrity tested and verified recoverable backup would give me far more "warm fuzzy's" than the "luck of the draw" that is calling Microsoft and hoping you get a competent person. But of course, each person has to make their own choice, and you may want a direction to point the finger at when things go wrong.