VSS timeout with Exchange 2010

[Gostev]

Just out of interest sake - if I select Microsoft VSS option in my backup job (put a check mark next to 'Enable application image-aware processing') is it safe to assume I can disable/uncheck the 'Enable VMWARE tools quiescene'?

VMware Tools quiescence is automatically disabled if application image-aware processing is enabled. Basically, the state of this checkbox is ignored in that case.

[larry]

I had this issue (exchange 2010/ Win 2008R2) but by starting the job when the esx and veeam is not backing up anything else, and no big jobs running on the esx host , it has been 10 days without the error. When it was failing was when I start a lot of replication,cloning and backups, right before this there were a few large SQL copies taking place.

[Gostev]

This make sense, since Exchange freeze is quite I/O consuming process... so if backend storage is quite busy, Exchange may not have enough time to complete the process within the designated VSS timeout.

[1-0-1]

That is does makes sense and I moved the Exchange server to a datastore with only two other virtual servers (two domain controllers) and made sure there are no other jobs running and it still times out. Could it be that 1.4TB is too much too handle at once? The disks being used are 15k SAS drives by the way so storage performance should not be a problem.

[Gostev]

That's right, but keep in mind that your issue may be different as well. You may want to open a support case with Microsoft and have them troubleshoot what makes VSS freeze last so long on your Exchange. Keep in mind that Exchange VSS Writer and whole VSS framework is all proprietary Microsoft code, so if something does not work well there, they are in better position than us to troubleshoot the issue.

[sampol]

We have the same problem. Windows 2008R2, Exchange 2010 SP1 (R3). No symantec product installed.

[Gostev]

Please read my previous post.

[newpylong]

Unfortunately, to your end users, the "we don't have access to source code, it's not our product, etc" does not hold any ground. If you are relying on a technology in your product, even if it isn't yours, you need to support it until you get to a point where you have exhausted all internal resources and need to leverage Microsoft. You do realize that is like calling Dell and having them tell you to call Microsoft for any Windows problems right?

Veeam should have adequate engineers who are trained on Windows Server, and thus VSS to be able to assist troubleshooting issues like this since a big marketing point to your product is based around replacing file level backups.

I know we didn't shell out $20,000 to be told to call Microsoft because our Veeam Exchange backups are failing...

[Gostev]

on a technology in your product, even if it isn't yours, you need to support it until you get to a point where you have exhausted all internal resources and need to leverage Microsoft

And that's exactly the point where we were at when I said that. Point where further troubleshooting required to have our developers looking at Exchange VSS Writer source code, that we do not have access to. Which is why it was suggested to open a case with Microsoft support directly. Thanks.

[Wil]

[merged]

Hello,
Has anyone had the error message below?

Freezing guest operating system
Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [NTDS]. Class ID: [{b2014c9e-8711-4c5c-a5a9-3cf384484757}]. Instance ID: [{670ad215-83ce-4c5f-8987-d727cfc4d466}]. Writer's state: [VSS_WS_FAILED_AT_POST_SNAPSHOT]. Error code: [0x800423f4].]

I've been trying to find a solution online but, no luck.
Thanks,

[Wil]

I was able to fix the issue by restarting the COM+ Event System service. A window will pop up letting you know that it will restart that service and others.
I retried the backup job and it was successful. Let's see how it goes tomorrow when it runs again.

[cliffm]

Similar to many posts here I have been unable to get VSS stable with Veeam backup on a SBS 2008 for over a month now. Exchange writers fail and various other writers get stuck. Support has not been able to fix this and also suggested this is a Microsoft problem. Eventually I worked out this has nothing to do with Microsoft. If I run Veeam backup against the same SBS VM on a network that does not have a VC, the jobs ALL succeed immediately. So I tried turning the VC on and off on my production network: VC ON and jobs fail, VC OFF (shut down) and the exact same jobs succeed with NO changes made to the SBS abd no SBS reboots. I have reliably produced this behaviour over and over. Since I have a small environment I can live without a VC. I suspect this has something to do with the SBS being on a 4.1 host as this did not occur when the same VM lived on a 4.0 host. I don't have the time or resources to test all the different variations of this but support tell me they have now replicated this behaviour in a lab. In the meantime until there is a fix, I can live without the VC.

[Vitaliy S.]

Hello Cliff, could you please tell me your ticket number?

[cliffm]

The ticket is 5135996. This has now been closed. The VC is turned off and the Veeam backup product has been working as advertised for a week now so at this stage there is no way I am turning the VC on again.

[J1mbo]

VC is?

[cliffm]

VC is VMware vCentre server. VC is some jargon that Veeam support started using but yes, my apologies, it probably did not make sense. It took me a long time to realise that this had something to do with the issue as I was fully focused for weeks on VSS being the problem. But as soon as I turned the VC off my Veeam jobs started working. As soon as I turn it on the Veeam jobs stop working. Veeam has been OK backing up everything else except a Microsoft SBS. For the SBS I need the VC turned off or to run the backups on a different network without a VC. The SBS is the only Microsoft server we have, everything else is Linux. So I can't comment on any other flavours or versions of MS Server. Veeam has however backed up various XP machines using VSS with no issue; for whatever reason it is just the SBS that has caused me grief so I can understand the focus on VSS. I don't know if the VC is the root of the issue but at this stage it is the only reliable reproducible thing I have found that gets me a working backup every time. Something somewhere in the complexity is not communicating in the Veeam/ESXi/VC/VMware tools mix, I don't have the resources to test all the combinations. But; at least for my case I have got something that works; which is all I need for now.

[tsightler]

When you are using VC to perform backups are you sure that the account that you used to add the VC to Veeam has full administrative permissions on the VM?

The reason I'm asking is that I've seen at least a case or two where the user tried to be more restrictive with the permissions that just giving "full access" to the Veeam account to vCenter. This mostly worked, except in cases where the Veeam server was firewalled or otherwise unable to access the Windows VM it is trying to back up. In this scenario Veeam cannot activate the VSS runtime service via the administrative share and must access the host via the VMware Tools interface, however, the account being used by the Veeam server doesn't have permissions to perform this.

When running the job via the ESX host directly it's typically running with an account that has full admin privileges on the ESX host and then it works fine.

[J1mbo]

Surely the problem here is that Veeam can't confirm what's happened with vCentre due to what is effectively a race between the VSS quiesce request and the vCentre need to update the database with what is going on. So it would be expected behaviour (for me anyway).

I'd suggest running vCentre in it's own VM anyway (if you update to vSphere 5 there will be the vCentre appliance option, to save on licensing another Microsoft instance, at the expense of the vRAM nonsence).

[jbrullemans]

We also had this problem in the beginning of May this year. Contacted MS contacted EMC contacted Veeam. No one new what the problem was and where the time out came from. On one i day i increased the memory of VC and all the problems dissapeard.

[tsightler]

Surely the problem here is that Veeam can't confirm what's happened with vCentre due to what is effectively a race between the VSS quiesce request and the vCentre need to update the database with what is going on. So it would be expected behaviour (for me anyway).

I'd suggest running vCentre in it's own VM anyway (if you update to vSphere 5 there will be the vCentre appliance option, to save on licensing another Microsoft instance, at the expense of the vRAM nonsence).

That's a good point, and certainly possible, but I didn't understand that from his posts. If vCenter itself is installed on, or uses the SQL database on the SBS server with which he's having a problem then you're right, that's probably the issue, and makes perfect sense.

[cliffm]

I had the VC installed on a VM that was on a brand new IBM server, using approx 1/3 of all the server resources. I also had Veeam server installed (64 bit) on the same VM as the VC. The SBS is on the same host but doers not have anything other that VMware Tools installed in it. It was not until I tried Veeam server (32 bit) on another network that I started to see any success with this issue. Eventually I tried turning the VC off completely and the Veeam jobs started working on my original production network. As for admin access, support also asked me about this. Occasionally Veeam jobs would succeed and all writers were stable after a successful backup. So I don't think it is a permissions thing because nothing has changed permissions wise from success to failure or vice versa.

I bought Veeam to backup this pesky MS SBS. It has turned out to be quite a path of discovery! However for me now the VC is turned off because a backup is more important to me than a VC. This I can live with until there is some clarity about what the actual issue is here, and of course a fix!

[gerdawg]

I have the same issue after setting up a DAG between my two exchange 2010 SP1 servers.

Does this link seem to be the culprit?
________________________________________________________________________________________________________________________________
Understanding Backup, Restore and Disaster Recovery

http://technet.microsoft.com/en-us/libr ... 76874.aspx

Be aware of these limitations when using VSS for backup and restore of Exchange data:

The VSS plug-in that ships with Exchange 2010 can be used to back up volumes containing active mailbox database copies or standalone (non-replicated) mailbox databases only. It can't be used to back up volumes containing passive mailbox database copies. To back up passive mailbox database copies, you need either Microsoft System Center Data Protection Manager or a third-party Exchange-aware VSS-based application.

Passive mailbox database copies are backed up using a separate VSS writer in the Microsoft Exchange Replication service. The Microsoft Exchange Replication service VSS Writer doesn't support restores. Although you can back up a passive mailbox database copy using Microsoft System Center Data Protection Manager or a third-party Exchange-aware VSS-based application, you can't perform a VSS restore directly to a passive mailbox database copy. However, you can perform a VSS restore to an alternate location, suspend replication to the passive copy, and then copy the database and log files from the alternate location to the location of the passive database copy in the file system.
_________________________________________________________________________________
I am not sure how the mailbox is being called by veaam and exactly what Exchange writers it uses but at least for me I've narrowed down having two active/active databases on one server and not having a problem. It seem to me that it's also plausible as right now I am in the middle of moving many of our users to different mailbox stores based on location. My one mailbox store is over a 10mb link and DAG between the two databases (and the move to that server's active database) takes place over that link. I was wondering if it's also possible that simply seeding the DAG database requires the VSS writer to be in-use and this causes the issue with Veaam/Microsoft's VSS writer when another application tries to gain access to perform the backup for Veeam. What concerns me most is the fact that the passive database from my other mail server resides on the same server that at times backs up and other times does not and I am wondering if the tech net article above is the reason for this.

Usually after a few tries or possibly letting the database sit idle and running the job again later that night fixes the issue. What I do notice is when that happens my traffic between sites are lower - which in turn means the replication of DAG between them is also lower.

For the record Veeam is setup to backup the server from the same site and uses 1 gbps link to do so for the exchange server that is on the same switch.

[gerdawg]

This may be a silly question but should i be backing up BOTH of my exchange servers in a DAG configuration? It still seems if one backup is going on one server that it is frozen on the other server.

Should I just be copying the active/passive database on the one server and NOT backing up my other server that has an active/passive on it (the inverse of Server 1).

If that is the case, can you easily restore a passive database copy? All in all a passive copy should hold the exact same information that an active copy has regardless of location. Is there a reason to backup my secondary exchange server other than for configuration purposes?

I read the Microsoft best practices guide for backing up DAG servers but I didn't see anything in regards to this that made sense to me, I have read elsewhere by Microsoft that it is acceptable to backup the passive database rather than an active but I wanted to see what everyone else thoughts were on this first. Other types of backup products (BE) give you the option to backup an "active" copy only, which I am guessing is fine but to take it one step further...let's say I have a connectivity issue one night and my primary exchange server swaps the database to passive for both stores and puts an active copy of both DB"s on server 2 which isn't backed up by Veeam. Would a passive copy of BOTH databases by Veeam for Server 1 still be recoverable using AIR?

I just want to make sure I'm doing the right thing, sometimes Microsoft says on thing but you guys have the experience when it comes to backing up a Virtual Exchange server running DAG so any help is appreciated.

Thanks!

[gerdawg]

Would anyone be able to take a stab at the question above? Is it necessary to backup an exchange DAG server from both servers or would backing up a passive database from one of the servers be sufficient?

[foggy]

Gerry, there are a few existing topics on this that might be helpful to you, for example this one (particularly, the last page).
Thanks.

[gerdawg]

Foggy,

Thank you for the reply. I've looked through the forums before posting the above question and cannot find an answer from someone @ veeam that will tell me if it's an OK process or not just to backup one member of the DAG cluster or not.

(http://www.veeam.com/forums/viewtopic.p ... kup#p31803 and http://www.veeam.com/forums/viewtopic.p ... kup#p31803
)

To be clear, my question is - Can I only backup one member of a DAG cluster regardless if it holds and active/passive - passive/passive - active/active database configuration using Veeam or do I have to backup both of my server in my DAG configuration. Microsoft recommends that you only backup the passive configuration per my reading, however I can't see how you can control that in the event of a failover.

What's Veeam's stance on backing up a DAG database? - Am I supported for a restore of any database on my DAG cluster regardless if it was backed up as Passive/Active if I'm only backing up one server in my DAG configuration. It seems like it would be a waste to have to backup both of my DAG servers if they are supposed to contain the same information.

[Gostev]

Veeam does not have official stance on backing up a DAG database, so it is up to customer what to backup. AFAIK, most customers backup both in a single job, and because of dedupe this does not really affects backup size, but you have both VMs backed up.

Kindly please do not further derail this topic with DAG-specific questions, as OP is talking about different issue, not connected with DAG. There are existing topics discussing DAG specific that you can continue posting into. I will clean this topic up a bit later to remove all offtopic. Thanks.

[gerdawg]

Gostev,

Thank you for the reply. That said, I'll disagree that the topic entirely a derail from the OP's issue and that mostly my fault as I did not specify that I am having a freezing issue as well when Veeam is trying to backup our exchange servers, so I apologize.

I am having the same issue as the OP with the guest freezing. However my issues are FURTHER complicated by using two separate exchange servers, which I would believe happen irrespective of DAG.

I have searched the forums looking for some information on this configuration and my issue of the DB writer freezing and this is something Veeam claims "is supported" however I have yet to find a post that provides insight on how to make that happen reliably. My main concern and the reason I posted in the thread is that I'm looking for information on how to stop the unreliable freezing issues of our VSS writer when trying to backup our Exchange server. Any insight you can provide there further would be appreciated - or if this could be moved into a more specific topic that would provide better understanding of the freezing issue along with DAG that might be more helpful.

Thanks!

[Bunce]

The freezing issue has been done to death and workarounds posted.Search my posts for the FCS modification commands.

Its not up to veeam to tell you what to backup.. Thats your job to work out based on your particular needs, environment, and XGE administration knowledge.

The link Anton provided gives examples of what customers (including myself) have chosen to backup in various DAG scenarios, so use them, or don't use them, as a guide.

[gerdawg]

Bunce,

I saw your FCS mod commands and unfortunately they don't help me at all in my situation. What has worked however at this point is not backing up my secondary server or only backing it up after the first Veeam server is finished. For reference I have figured out what my issue is and my work around works for me just fine to backup what I need.

That said, I'm not asking Veeam to tell me what to backup. I am asking them what they believe the best practice is for using the product that they have developed. This shouldn't turn into a pissing match as you can go to any other product vendor and ask this very question and they already have pre-canned information to guide you in a direction of what they recommend for their product. Perhaps asking in these forums is the issue, and i should call support for my issue and guidance. Based on other posts I have read it seems like whenever the question is asked of what's "preferred" to backup with Veeam for certain technologies the answer is always, whatever it's your choice. Yes, clearly, it's my choice and my responsibility of what I want to backup. However you should keep in mind, the Veeam product, regardless of the fact the Microsoft's VSS writers are being use, ONLY backs up disks which complicates what "best practice" can and will be for a customer that can no longer choose to backup EXACTLY what they WANT - such as backing up active vs passive databases residing on a single server. Perhaps I'm off base by asking in this post which was resurrected fresh by another individual and concurred with an issue as I was having, and since there isn't an official company answer, fine, but this shouldn't be a difficult question that a customer is asking nor should it be a problem that I am asking it.

That said, Veeam forums have been very informative and I'm appreciative of the advice I've been given so far from all of the board members, I was just looking to find out what the official stance was based on a problem I was having with my freezing of the VSS, and I'm clear based on the previous post that Veeam does not have an official stance on what they deem should be backed up for Exchange (DAG). I think my specific issue likely stems from the fact that I have 2 separate Veeam Servers (so two different jobs) backing up DAG servers at two different sites and I have yet to find a post that deals with my specific issue.

As Gostev has already suggested - if this post is off topic for this particular thread and should likely be moved to another area with close to the same issue, but is a DAG thread, please feel free to merge and/or move it there.