VSS timeout with Exchange 2010

[Jamie Pert]

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7
--------------------

I won't deny it I have been asked to maintain Veeam and investigate failures despite not having much experience in terms of virtualization. Can you tell me what vmware tools quiescence is... I don't even know what the word "quiescence" means.

I have done some research and it seems as if the vmware tools quiescence should definitely not be enabled for Exchange servers. Me and a colleague thought that by ticking "Enable application-aware image processing" it would over-ride the "Enable VMware Tools quiescence" option.

[Cokovic]

VMware tools quiescence option enables freezing of the file-system for proper snapshotcreation. But it's not application aware so your backups of an Exchange database may be inconsistent.

Just disable it in your job configuration. Right click your job and click edit. Then click two times next. You should be in the window where you can select the backup proxy and the repository. Click on Advanced. Under the tab "vSphere" you can uncheck "Enable VMWare Tools quiescence". Thats it. Leave the application aware image processing by Veeam on.

[Vitaliy S.]

Me and a colleague thought that by ticking "Enable application-aware image processing" it would over-ride the "Enable VMware Tools quiescence" option.

If "application-aware image processing" is enabled, it overrides previously selected "VMware Tools quiescence" option, you are correct in your assumption.

[Jamie Pert]

Thanks guys, I will let you know the results of my backups jobs which should run tonight.

If the backups succeed tonight it will confuse me as surely "VMware Tools quiescence" must have been doing something despite me having the "application-aware image processing" over-riding this setting...

[Jamie Pert]

Despite changing that setting you guys were right, the same damn error still occurs on pretty much every exchange server I backup

[Cokovic]

Recently i had a support call with Microsoft regarding VSS timeouts with Exchange. Finally there were different conditions that occured letting my VSS backups fail (like I/O on the SAN and also load on the Veeam backupserver itself).

Maybe a good idea would be to check if VSS will work itself without Veeam. There is a script you can use for that to ensure VSS is working as expected. If you want i can PM you these scripts and instructions by Microsoft.

[Jamie Pert]

That would be great Cokovic

[Moebius]

Cokovic, I'd like to give the script you mentioned a try as well. I have an Exchange 2007 that regularly fails the first three or four backup attempts due to VSS timeout, and usually (but not always) succeeds on the last attempts. Could you PM me too with the scripts and instructions? Thanks!

[rino451]

I messed with this error for weeks, kept reading the same stuff on the forums about how it's MS's issue, or to call tech support, but never saw a slam dunk fix to the issue. I went out on a limb and installed Quest's vRanger trial on my Veeam server and from day 1, never had the issue ever again. I witched to vRanger and haven't looked back. I've kept Veeam around b/c I liked some of it's options, but getting backups trumped convenience, so I no longer use Veeam in our home office. Unfortunately, I bought 3 years of Veeam.

To be clear, we have about 1tb of Exchange data in three mail stores. I was always able to take a vmware snapshot and the writers all were fine BEFORE attempting a Veeam backup. After the attempted backup, one of the exchange writers was goofed up requiring me to perform the steps in KB1646.

The handling if this issue is completely unsat.

[Gostev]

Hi Randall, I am sorry but I just don't see how your "success story" relates to what is being discussed in this topic, when vRanger simply does not implement application-aware processing. Did you notice that it never asks you for guest credentials, unlike Veeam? To draw the analogy - if you swap a car to a bicycle, sure thing you will never get your engine brake up on you again - because bicycle does not have one...

Also, while obviously bicycle is still much more useful than a car with broken engine, please do keep in mind that you can easily switch Veeam to this "bicycle mode" too. Basically, you can have the exact same behavior with Veeam by simply disabling application-aware processing, as discussed above. Obviously, in this case VM will be backed up and restored without application-specific logic - meaning, without following Microsoft requirements on Exchange backup and restore. If you are fine about backing up and restoring your Exchange in an unsupported manner, you can sure do that with Veeam as well. And you will find that Exchange VSS writer timeouts will indeed stop messing your backups, and they will be successful each and every time - simply because VSS will never be used in this case

The issue discussed in this topic only appears with backup solutions which implement Microsoft Exchange backup and restore according to Microsoft requirements. This requires run-time coordination process present in guest for the duration of backup, and so guest credentials. Since most competing vendors (including vRanger) simply do not implement those Microsoft requirements (or implement them only partially), you will not see the same issue there. Likewise, Veeam will operate in the exact same mode if you do not enable application-aware processing. Just disable the corresponding option in Veeam to get the same results.

I understand your frustration, but this issue has to deal with Microsoft VSS framework code that we do not control and thus cannot debug/fix/enhance (this is something Microsoft itself implements). Other customers have already shared above what have helped them to resolve this issue, and just as Harris said a few posts above, it always comes down to some infrastructure problems. Hopefully Microsoft will enhance Exchange VSS writer soon to work more reliably under heavy load (if this is even possible, as VSS quiescence is supposed to be I/O heavy process anyway).

Internally at Veeam, when designing our Exchange infrastructure we just made sure our Exchange 2010 has plenty of I/O capacity (dedicated spindles for database and transaction logs), and we have never experienced this issue in the past 2 years. And these days, we are talking about Exchange server with almost 900 of very active mailboxes, and 1 TB in size.

Thanks.

[Jamie Pert]

Hello Gostev - if I update vanilla Exchange 2010 will there be any change in the Exchange VSS Writer? Or does it not differ.

(by the way I love your bicycle analogy!)

[foggy]

Anyway, it's always better to keep the software at its latest patch level.

[mysidia]

Internally at Veeam, when designing our Exchange infrastructure we just made sure our Exchange 2010 has plenty of I/O capacity (dedicated spindles for database and transaction logs), and we have never experienced this issue in the past 2 years.

Is there anything anyone found to reduce the probability of a timeout, when storage performance is already great?
e.g. Are there any configuration options in Veeam to disable wasting time trying to connect to the guest IP over the network?

So I have a question... in designing your Exchange infrastructure, did you isolate the backup system, so that there is no network connectivity between the Veeam VM and the Exchange VM? My understanding is Veeam 6 application image processing is supposed to still work in a scenario where Veeam will not have network connectivity to the same LAN as the VM being backed up, correct? (E.g. VM in the DMZ, Veeam in isolated network)
I am having problems in that app image processing seems to fail _consistently_ with the Exchange 2010 servers; in my environment. It works fine for Exchange 2007, heavily loaded SQL servers, etc. Are there any suggestions about steps to take in that case?

What I see are consistent application image processing timeouts Releasing guest via VIX issue as well, for almost all Exchange 2010 mailstore VMs I have tried to backup with Veeam, there are some that work 20% of the time (servers with few very small databases). I have had few/no issues with Exchange 2010 VMs that live in IP address space that the Veeam backup server has access to, ever, only the VMs that exist on another private LAN that the Veeam server cannot contact due to firewall or lack of routability, are the ones I am seeing this on. No issues with CAS/Hub Transport VMs.

I am sure that: (1) vCenter is functioning with reasonable performance, most operations such as "Search for VM", "power on VM", etc, complete within 5 seconds, vC is managing ~250 VMs.
(2) Storage performance is not currently a bottleneck in the environment; as measured based on capacity of the Netapp FAS array, low latency, and Veeam is able to push >600mbits during backup. In case of a storage performance is an issue... Exchange users would tend to notice that, most of these Exchange servers have over 1500 users, and the storage is sized with enough 10k SAS spindles and controller cache for peak load + headroom.

Exchange 2010 is also specifically designed to work on slower SATA storage, hence features such as 24x7 ESE scanning, and no, disabling, and scheduling maintenance to not overlap with backup, did nothing to help with the application image processing.

(3) There are no third-party VSS modules or other software installed, only the default Windows and Exchange VSS writers.

(4) In testing... making the Veeam server have connectivity to the Exchange VM guest IP address on a temporary basis fixes the issue, but unfortunately, this does not work as a permanent repeatable solution, because there is an inherent security issue with it, at least for some of the servers.
That is, specifically: The VM being backed up lives on another entity's VLAN in a multi-tenant environment, and no routability between the VM infrastructure network and their Exchange LAN is wanted. The temporary "fix" was to NAT the unroutable private IP to a public one-to-one IP address of the mailstore server, and arrange for the Veeam server to be temporarily allowed Network-based access to the Exchange server, for testing purposes.

Once this was done, Veeam application image processing started working.
As soon as the firewall pass rule was removed, the VIX Timeout errors returned.

Windows 2008 R2 server backup utility which uses VSS was tested on a couple of these servers after Veeam, as were some alternate otherwise less-attractive third party backup products, and had no issues with their VSS processing.

[PedroVDP]

I've been suffering under this same issue for quite some time now and the problem as far as I can determine is performance.

The Exchange VSS has an extremely short time-frame in which it allows you to take a snapshot. In this moronically small window you need to signal to veeam the vss freeze is ready, have veeam create a vmware snapshot and signal a thaw to the vm running exchange. The only real solution in my mind would be a way to increase the Exchange VSS time-out window but as thusfar I haven't found any way to do this. When you are connecting directly to Exchange from Veeam instead of using VIX (where I still have issues myself even) the communication simply is faster... wasting 5 seconds less in a 60 second time-frame simply is the narrow margin you have making it work.

[Cokovic]

The VSS timewindow for Exchange is 20 seconds. And there is no way to change this as it is harcoded (thats the reason why you haven't found any way to do this as it's simply not possible).

[Jamie Pert]

mysidia and PedroVDP are having the exact same problem as me - It's annoying to think that if Microsoft increased the 20 second VSS Timewindow it "could" fix the problem - All I get is "job finished with warnings" everyday on all of my Exchange servers, so what happens if heaven forbid a server dies and I have to restore a server from my Veeam jobs - how significant are the "warnings"?

I understand Veeam support have no control over this writer - but seeing as how this forum post is 11 pages long already I really do feel that a knowledge-base article should be written to address our issues.

Does it matter whether you are running Exchange 2010 (Vanilla), Exchange 2010 SP1 or SP2 - obviously we want all our clients to upgrade to the latest "stable" build, but we would be embarrassed to say to our client "pay us to upgrade from Vanilla to SP2 and your backups will work" - only to find that reports still show warnings on every single backup of their exchange server.

I could upgrade all our clients to Exchange 2010 SP2 - this "could" fix the Veeam issues, but what if it fails, I have no success jobs to fall back on.

Getting such a headache from all this, appreciated the help from you guys, but when you have been assigned the task of logging veeam jobs and each day you can predict the results it gets quite frustrating...

---------------------------------------------

I'd be more understanding if clients hadn't spent a huge amount on storage for their veeam backups

[mysidia]

Does it matter whether you are running Exchange 2010 (Vanilla), Exchange 2010 SP1 or SP2 - obviously we want all our clients to upgrade to the latest "stable" build, but we would be embarrassed to say to our client "pay us to upgrade from Vanilla to SP2 and your backups will work" - only to find that reports still show warnings on every single backup of their exchange server.

Don't get me wrong, there are very good reasons to _not_ be running anything older than SP2 Rollup 2 at this point; your server could incur downtime due to one of the critical issues that was addressed after SP2.
It does not seem to matter what Service Pack release of Exchange 2010 is running, I get VSS timeout failures on SP1.

I get VSS failures on SP1 Rollup 6, and SP2 Rollup 3.
When Veeam has no connectivity to the guest OS, that is. Having a way to open one port for Veeam to signal a Veeam agent running on the guest, and to be able to configure Veeam to use a specific IP address, instead of an IP "detected" using VM Tools, would be OK.

Opening Windows RPC and SMB is a big problem, as there is no way to differentiate between a "signal" to a Veeam agent and arbitrary Windows control traffic; the number of TCP ports that have to be opened for Windows RPC to work is fairly "all-inclusive", such that opening these ports on the Firewall is pretty much equivalent to having no firewall at all.

[tsightler]

@mysidia Have you at least proven that opening the port does indeed work? If so, that should be something that can be addressed. I'm curious if the bulk of users on this list have their Exchange server firewalled from their Veeam server because that is a different issue from VSS simply timing out in general. If others could chime in as to whether or not they're Exchange servers are firewalled or not.

One thing that is extremely important if you are attempting to use "connectionless mode" for VSS (i.e. if there is a firewall and thus we rely on the VIX API to communicate) is that you must meet at least ONE of the following conditions:

1. The account being used for Application Aware Processing MUST be either the "built-in" local administrator, or the "built-in" domain administrator (i.e. it must have a "well-know" SID ending in 500), other local or domain administrator accounts will not work.

--OR--

2. UAC must be disabled on the guest VM.

The most common issue with "connectionless" mode is UAC blocking it. The reason that using the "built-in" administrator accounts work is that the default security policy for these "well-known" accounts are exempt from UAC even if it is enabled, while all other administrator accounts are forced to use UAC for privilege escalation.

[mysidia]

@mysidia Have you at least proven that opening the port does indeed work? If so, that should be something that can be addressed.

Yes. It works with traffic allowed from Veeam to the VM on all TCP ports; 135, 445, 1024-65535, there is no VSS timeout error, 90% of the time on most VMs; 1 out of 7 VMs still occasionally has a Timeout, the one VM that is failing is a database server running and old release of Exchange 2010 SP1 (no rollup updates) and only has 1 Exchange DB of size ~500gb. As soon as the firewall "allow all" rule from Veeam to the Exchange VM is removed, the VSS Timeout returns for all the Exchange 2010 mailbox DB VMs.

1. The account being used for Application Aware Processing MUST be either the "built-in" local administrator, or the "built-in" domain administrator (i.e. it must have a "well-know" SID ending in 500), other local or domain administrator accounts will not work.

The server's BUILTIN\Administrator account credentials are used; and trying DOMAIN\Administrator has no effect, the same VSS timeout is still logged.

[Jamie Pert]

Really frustrated with the support I've been given. For 2 months our Exchange servers have constantly finished with "Warning" states - all saying:

Unable to release guest. Details: Unfreeze error (over VIX): [Backup job failed.] followed by Processing finished with warnings

I've probably spent near on 50 hours trying to fix this as I was under the impression that this meant my Exchange server backups weren't properly finishing (you'd think that by message above) - however today Veeam support have said that this is not actually something you need to worry about!?

If you are getting warning the backup is OK. VM is frozen properly; since this moment Veeam is working with original disks. All acitivity of VM - new emails, unfreeze peocedures and so on are kept on the snapshot so that doesn't correspond to backup. You even may start removing some files once snapshot is created. Those files will be in backup. The only problem is that VSS writer is failing due to timeout so it takes a bit time to bring it back. As you do not make new backup of this server during the day it is enough time for writer to get to the normal state for next session.

So recreating the jobs, creating new repositories and proxies, rebooting the servers, troubleshooting VSS writers, installing updates on the server OS, installing service packs for Exchange, moving servers to another host, re-running jobs at different times, uploading logs, reading through huge logs myself, re-arranging the times at which the backups run, changing the time windows used by Exchange to carry out maintenance was all an absolute waste of time?

Do I know trust Veeam's answer? I honestly don't know, if the VM dies and I find out that my Exchange backups are not infact "OK" I'm f****d

[Cokovic]

Hi Jamie,

why not try to use Surebackup to verify your Exchange Backup? Or maybe get the Veeam Explorer for Exchange utility and try to open your DB files from the backup and if you can extract single items. If so then you are good to go with your backups

[POCveam6]

Recently i had a support call with Microsoft regarding VSS timeouts with Exchange. Finally there were different conditions that occured letting my VSS backups fail (like I/O on the SAN and also load on the Veeam backupserver itself).

Maybe a good idea would be to check if VSS will work itself without Veeam. There is a script you can use for that to ensure VSS is working as expected. If you want i can PM you these scripts and instructions by Microsoft.

can you pm me, that would be grate to have that script

[Gostev]

Just wondering if anyone with 6.1 Patch 1 installed seeing any improvements with the above issue?

[Cokovic]

@POCveam6
I'm currently on holiday. Will provide you the script in about two weeks.

@Gostev
No improvements here. Still seeing VSS timeouts like before.

[Jamie Pert]

Hey Gostev - I still have multiple Exchange servers with same issue despite having patch 1 installed - however I now treat "warning" results as "successful" as advised by Veeam support

Mmm... it was a problem you would get "failed" status instead of "warning". We are not allowed pushing "success" if some steps didn't complete successfully even if they do not cause problem with backups

[tholyoak]

I'm currently evaluating version 6.1 (with latest patch, installed yesterday) for backing up a Win2008R2 server with Exchange 2010. I found this thread because I've been unable to get it to replicate. I finally found that replication has to be done in the middle of the night in order to get past the VSS timeout problem. (Although the Windows Backup seems to not ever have this problem, even if run in the middle of the day - it just takes a really long time for it to check the Exchange database consistency. But we can't rely on Windows Backup because it takes too long to restore.)

The reason we are looking at Veeam is because we want to be able to have a replicated server ready to turn on at any time if something happens to our Exchange server. We were hoping to have continuous replication going, but with this problem, it doesn't look possible.

[Gostev]

Continuous or near-continues replication must be performed crash-consistent no matter what replication product or technology you are using, even including SAN-based replication. The problem is that VSS quiescence causes too much impact on the application, and realistically you just cannot do this every few minutes throughout the day - even with the most powerful hardware available.

[tholyoak]

So if we wanted continuous replication and had to simply turn off quiescence, the results would basically be like unplugging the server while it's running and then hoping for the best when it comes back up, right? That actually doesn't seem so bad, unless I'm overlooking something. (I must be, or MS wouldn't be so picky about quiescence...)

And if I may ask a related question, for applications like Exchange, is there any point in having more than 1 restore point available? It doesn't seem like I'd want to go back any further than the latest because then I'd lose messages.

[Gostev]

That is correct. You still want to have consistent restore points at least daily, which is why you also need to perform proper backups of near-CDP replicated VMs.

As far as multiple restore points, this does not really belong to this topic, so it would be best if you create new one.

In short, multiple restore points requirement does not matter of the application you are running in a VM, but it becomes significantly important for when you are replicating very often. Let's say, you are replicating every 15 minutes. Imagine the situation when your source VM gets infected by the 0-day virus, or some database file gets corrupted, or your application just stops working altogether for whatever reason. You now have less than 15 minutes to find out about this actually happened, perform basic troubleshooting, and make the decision to perform failover (otherwise, the corrupted state will be replicated over to replica, and then you are done - game over).

Even if you are replicating as rarely as every few hours, imagine the issue happening during the night, when you are not even monitoring your systems closely. You are simply not around to do anything, so when you come back next morning, there is no good restore point to perform failover to already.

Yes, failover to earlier restore point in some cases means extended data loss, but this is still tons better then going back up to 24 hours ago to your daily backup... and so much better than complete data loss in case you don't do daily backups for VMs you are replicating. Besides, in cases such as "application broke", there is no new data generated that you can lose when failing over to latest good state. So, failing over further back is not always a bad thing.

As you can see, multiple restore points are absolutely essential for you to be able to guarantee being able to meet RTO for protected VMs in every disaster scenario.

[tsightler]

So if we wanted continuous replication and had to simply turn off quiescence, the results would basically be like unplugging the server while it's running and then hoping for the best when it comes back up, right?

It's actually slightly better than this. When you "pull the plug" there's a lot of things that can go wrong, e.g. partial writes to disks can cause corruption (battery backed caches on RAID controllers have made this less of an issue, but problems can still arise). A VMware snapshot is atomic across all disk, there's no possibility of a partial write or in general anything catastrophic. A modern, transitionally logged database running on a journaling filesystem should pretty much always be able to recover from this situation.