VSS timeout with Exchange 2010

[Cokovic]

Hi all,

actually i do have a project where we are going to deploy a new Exchange 2010 DAG cluster for about 3000 mailboxes. Before we started to build our environment we had tested everything regarding Exchange and Veeam and recoverabilty. When we made the tests Exchange had SP1 and the newest Updaterollup installed. Everything went fine.

Now we are installing our new production Exchange cluster and had major problems regarding this VSS timeout. SP2 with Updaterollup 2 was installed and 6 or 7 backups out of 10 went into this timeout issue. After 2 or 3 retrys backup went fine. And i could observe this on the cluster where no mailboxes were migrated up to now so there is no load on theses servers. My concern now was what will happen when we migrate all the mailboxes. Will it fail all the time? Or will it succeed at any time? So we decided to open a support case at Veeam and Microsoft. Cases are still going on.

BUT last wednesday we've installed the newest Updaterollup 3 for Exchange 2010 SP2 (Release date 20.05.2012) and what i can tell you so far is it seems that the VSS freeze errors magically went away. Now maybe 1 out of 10 backups fail with a VSS freeze error and the first retry is working. This is a behaviour i can live with.

Just wanted to ask if anyone else experienced something similar. I know there is nothing to find in the release notes of Updaterollup 3 about VSS enhancement. But this is what i've observed so far. I didnt' finished my testings yet but so far it looks good.

[habibalby]

Hello,
I do have Exchange 2007 with 1700 users almost 1 in every 10 backups I got VSS freeze after couple of retryes the backup goes fine. some time it doesn't go away unless I reboot the exchange server, which most of the time I can't reboot it My alternative is the Symantec Backup Exec to make sure I'm secured with a backup.

Don't believe that if someone tells you Backup Exec VSS interrupts Microsoft VSS when it comes to Veeam Backup, as in my environment I do have both agents and backup works fine. last time since Exchange Backup failed it's around 3 or 4 months ago.

Thanks,

[Cokovic]

Hello,
I do have Exchange 2007 with 1700 users almost 1 in every 10 backups I got VSS freeze after couple of retryes the backup goes fine. some time it doesn't go away unless I reboot the exchange server, which most of the time I can't reboot it My alternative is the Symantec Backup Exec to make sure I'm secured with a backup.

This is something i would not like to do. How do you handle transaction log truncating? With Veeam AND Symantec? Not a good idea as it would spread the transaction logs over two solutions. So this is a no go for us here. I will stay with one solution. And this is Veeam

Don't believe that if someone tells you Backup Exec VSS interrupts Microsoft VSS when it comes to Veeam Backup, as in my environment I do have both agents and backup works fine. last time since Exchange Backup failed it's around 3 or 4 months ago.

Thanks,

Sorry but i have to disagree. I had it myself that an installed Symantec VSS writer screwed up my Veeam backup. I had to uninstall the Symantec VSS writer and remove a registry key to get the backup with Veeam up and running. I'm not saying that it will break up Veeam backup in any situation. But i faced it myself

[Andreas Neufert]

Hi, just my 2 cents to your entry Cokovic.

If you Backup an SQL Server with 2 solutions you need to decide who is responsible for log file handling.
You can edit our VSS - advanced settings to disable our log file processing.

I had also problems with older BackupExec Agent Versions. The newer once looking better. But in case you are running into problems I think its better to have no other backupsoftware that uses or can use VSS on the client.

AND never schedule 2 backup solutions at the same timeframe.

CU Andy

[habibalby]

Hello,

I can prove that for you if you like, I can run Symantec backup as well as Veeam Backup.

I agree with you logs will be responsible by two product, but if I require the logs date that been backed up by Symantec, I can reach it or by veeam I can reach it.

Thanks,

[Cokovic]

Hi, just my 2 cents to your entry Cokovic.

If you Backup an SQL Server with 2 solutions you need to decide who is responsible for log file handling.
You can edit our VSS - advanced settings to disable our log file processing.

I had also problems with older BackupExec Agent Versions. The newer once looking better. But in case you are running into problems I think its better to have no other backupsoftware that uses or can use VSS on the client.

AND never schedule 2 backup solutions at the same timeframe.

CU Andy

Hallo Herr Neufert,

Ich wusste das ich Ihren Namen schonmal irgendwo gelesen habe

Now back to english. I'm aware that i'm able to change the behaviour of log processing within Veeam as i am in Backup Exec too. But as far as i understood habibalby right he is using BackupExec instead of Veeam in case he got VSS errors until he is able to reboot the Exchange server before able to use Veeam again (so he is not using it at the same time). And to prevent logs growing bigger and bigger and maybe filling up the logdrive i understood it that way he is truncating logs with BackupExec too in case of a VSS timeout with Veeam (correct me if i'm wrong habibalby. This is why i asked if he is handling log truncation with both solutions). And that would spread the logfiles in some cases over two different backup solutions.

[habibalby]

Hello,
That's right, I'm turnicating the logs on both solution. In case I got VSS error, "specilly with veeam" ir doesn solve unless I reboot the exchange server. If I'm unable toi reboot it in any free maintenance window, I continue with Backup Exec.

That's how I protect my servers as long as I'm supported by both.

[JoshF]

Has anyone with this issue upgraded to v6.1 and seen any noticable differences with this issue?

[habibalby]

Has anyone with this issue upgraded to v6.1 and seen any noticable differences with this issue?

Hello Josh,

I'm planning to upgrade this week, but I'm already don't experience this issue since long time ago almost 3 to 4 weeks

Thanks,

[Cokovic]

We will upgrade here too. But it will takes us at least 2 or 3 weeks to test it here before placing it into production.

[JoshF]

Thanks guys.. most interested to hear how your progress goes

[habibalby]

Hello Josh,

I'm planning to upgrade this week, but I'm already don't experience this issue since long time ago almost 3 to 4 weeks

Thanks,

On Thursday upgraded my production from Veeam 6.0 to 6.1, Thursday was an Incremental and Friday was Full Backup.. All went jobs success expect one error due to Repository Disk Space..

Exchange Enterprise with vRDM of 1.0 TB and 1700 users

[Kristiaanj]

Hey you guys,

I experienced this problem as wel. at first it was enough to restart the exchange server so the vss writer would work again. But recently i could not get it to work no matter what.
I installed exhcnage 2010 SP2 and updated the Veeam B&R to version 6.1, this allso did not help.
I went ahead en monitor the exchange server while making the backup and i noticed that the on-access scanner from our virusscanner was using up a lot of resources.
After disabling the on-access scanner it al started working.
We are using Microsoft Forefront Endpoint Protection 2010.
I'm going to try to exclude some vss related processes from virus scanning.

Greetz,
Kris.

[Cokovic]

No changes here. Still failing very often at the first try on v6.1. It works but needs always a second or third retry. No virusscanner installed yet.

[Cytek]

I was having the same issue with several Exchange 2010 servers and SBS 2011 servers no matter what I tried from this post I was consistently getting freeze error. I followed one suggestion and opened up specific rules in the firewall to allow communication between the guest VM OS and the Veam backup server. After doing this the backups were 100% successful. This is a serious problem as I don't want to have huge punched holes in my firewall for Veeam to successfully backup. Many of these servers are SBS so please don't tell me to use a local account for Veeam - there is no local account, there are only domain accounts.

[Kristiaanj]

I have excluded the following processes from our virusscanner. after that it al started working like a charm:

C:\windows\system32\VSSVC.exe
C:\windows\VeeamVssSupport\VeeamGuestAgent.exe
C:\windows\VeeamVssSupport\VeeamOibIndexer.exe

allso excluding the store and log drive might not be a bad idea.

@Cokovic : Try using sysinternals process explorer to look at the server what is happening at the moment of the backup.

[Cokovic]

We do have already a case open with Veeam and Microsoft. So far they cant find anything in the logs
No virusscanner installed and local firewalls are disabled on our servers per policy.

[massimiliano.rizzi]

Hi there,

which rules did you open to allow comunication between SBS 2011 and the Veeam backup server ?

Thanks,

Massimiliano

[Cokovic]

Since one week my backups are running fine without any VSS timeout errors anymore.

What i can tell so far is that it seems that there are alot of different situations that can occur preventing a successfull VSS backup. People here for example reported firewall problems or virusscanner related problems that did the trick for them.

In our case it seems that different things came together. First i had scheduled our Exchange backups between 8pm and 10pm. This was a time window where alot of backup jobs ran and our storage was under heavy load. After i moved my jobs to run between 1am and 3am t he situation became better. VSS timeouts still occured but not that often like before. What seems to have made the trick was to move the Exchange database maintenance window. Per default Exchange performs database maintenance between 1am and 5am which was exactly within my backup time window. After moving the Exchange database maintenance window to between 9pm and 1am everything runs fine. Still watching this but as i wrote before since one week no VSS timeouts anymore

[habibalby]

Hi there,

which rules did you open to allow comunication between SBS 2011 and the Veeam backup server ?

Thanks,

Massimiliano

Hello,
SBS is an all-in-one box from Microsoft which has got TMG, Exchange,AD, etc etc... You need to open the ports listed in this KB and blog.
http://www.veeam.com/KB1518 Ports needed for use with Veeam backup and replication 6
http://www.veeam.com/blog/tcp-port-map- ... on-v5.html TCP port map for Veeam Backup & Replication v5

Or what you can do is, make an access rule in TMG/ISA Server and create an objects computer of both servers and put them in the From and To All Ports for All Users.

Run the backup and enable Logging in the TMG to monitor which ports are being used in that communication.

Thanks,

[habibalby]

@Cokovic,

thanks for sharing this great info. I had the similar this issue before, when Veeam running a job and dumping the backup to the same Repository where Exchange ran after a few minutes after and dump it's backup to the same Repository, a VSS TimeOut accrued.

Lesson learned, that I don't keep the exchange to overlap with any job nor the time or the Repository being utilized by other job. Even one proxy always dedicated for Exchange only..

Thanks,

[Jamie Pert]

[merged]

Case number = 5200139

Our company's Veeam installation is pretty successful overall, however I have 2 common errors which seem to affect multiple backups and result in them finishing with warnings rather than "Success"

One problem reads "Unable to release guest. Details: Unfreeze error (over VIX): [Backup job failed.]"

The other reads "Failed to prepare guest for hot backup. Details: VSSControl: -2147212301 Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS Critical writer has failed. Writer name: [Microsoft Exchange Replica Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{95941a68-5876-4d39-82a8-c380e8eea8ac}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_BACKUP]. Error code: [0x800423f3]".

Any ideas of the cause / resolutions for these errors?

[Cokovic]

One problem reads "Unable to release guest. Details: Unfreeze error (over VIX): [Backup job failed.]"

Veeam has two approaches how they inject their own agent into your VM. First is always to go over the administrative share which is the faster one. If administrative share is not available or if the account under the backup job runs has not the right to access the share Veeam will inject the agent over VIX libraries (VMWare tools got to be installed).

So maybe you should give it a try and enable administrative shares (if disabled and viable in your environment) or use an account with proper permissions to access the administrative share on your VM.

The other reads "Failed to prepare guest for hot backup. Details: VSSControl: -2147212301 Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS Critical writer has failed. Writer name: [Microsoft Exchange Replica Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{95941a68-5876-4d39-82a8-c380e8eea8ac}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_BACKUP]. Error code: [0x800423f3]".

Do you have any other VSS writer installed in your VM? Could you please open a command prompt and post the result of "vssadmin list providers"?

Thanks,
Haris

[Jamie Pert]

[merged]

5200139

Hello guys,

Veeam support have been a real help so far, but I'm looking for some suggestions from the community in regards to a problem which is affecting multiple backups jobs at the company I work for.

We backup quite a few servers using application aware processing - however multiple Exchange servers report that they are unable to release guest / unfreeze error over (VIX).

If I run vssadmin list writers in command on any of the exchange servers which report this issue I can see that the Microsoft Exchange Writer has failed it also shows (9) and timed out.

Has anyone else experienced this? Could Veeam's application aware processing be somehow knocking this writer over? Is there a hotfix? Is there a way to get this writer running again without disrupting the client's network(s)?

[Jamie Pert]

Hey Harris - thanks for your reply.

I have noticed that on 3 of the problematic Exchange server backups the Microsoft Exchange Writer has failed. I guess what I want to know is why this is failing and whether I can get it functional again without rebooting the server / causing downtime.

[Cokovic]

Hi Jamie,

hard to tell why it's failing in your case. Only possibility would be to open a case with Microsoft for further investigations if none of the discussed solutions in this thread can help you as Veeam completly relies on Microsoft VSS.

If you want to get Microsoft VSS Writer for Exchange back into a clean state only solution without rebooting your server would be to restart Microsoft Information Store. This worked for me so far.

And did you try out my suggestions from my posting above with the administrative share and vssadmin list providers?

[Jamie Pert]

Thanks, I wonder whether scheduling that service to restart prior to the exchange backup will help - I will let you guys know

[Jamie Pert]

Restarting the servers this weekend didn't work, Veeam support are pushing the fact that this is a VSS writer issue - I understand that this can be brushed off as a Windows issue - but could it be Veeam which is causing the vss writer to fall over?

There are endless possibilities and I thought I'd collate all the relevant information for one particular virtual server which just fails to backup successfully no-matter what.

The server runs Windows Server 2008 R2 Standard, all it does serves is Exchange.

The version of Exchange 2010 running is build 14.00.0722.000 (I’m aware that this is quite old, but we don’t currently want to install updates unless we really have to). If someone categorically says installing a particular update will fix this we will!

We carry out two backups of this server every Monday, Tuesday, Wednesday, Thursday and Friday.

One of the backups is a reverse incremental job which runs at 10.30pm and goes to repository A, it never takes any longer than 1 hour and 15 minutes. The next kicks off 00.20 and this is a standard incremental job which goes to repository B, this takes no-longer than 20 minutes. The only common factor is that they use the same proxy.

Looking further into the jobs, they both use inline data duplication, one is set to ‘BEST’ level optimized for WAN Target, whilst the other is to ‘Optimal’ level optimized for LAN Target (based on the repository locations this should be fine. Both jobs also use Guest quiescence, changed block tracking and have the box ticked for enabling block tracking on all VMs. Finally they both exclude the swap file blocks from processing and they both allow auto backup integrity checks.

In terms of guest processing they both are set to use application-aware image processing, but they don’t have guest file indexing enabled. They also both ignore application processing failures and truncate logs immediately. In terms of the indexing tab there are some exceptions, but these aren’t really relevant.

As requested here’s the result of vssadmin list writers:

Code: Select all

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {3c195135-356a-46d2-9aeb-300b6ae33cb3}
   State: [1] Stable
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {2008529d-1263-41e8-ab46-2a9401df1edc}
   State: [1] Stable
   Last error: No error

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {e2eee938-bf8b-4f7d-a5a9-5677dbeaf977}
   State: [1] Stable
   Last error: No error

Writer name: 'FRS Writer'
   Writer Id: {d76f5a28-3092-4589-ba48-2958fb88ce29}
   Writer Instance Id: {1efd257f-8e8b-41c1-baf4-9e8da338ad97}
   State: [1] Stable
   Last error: No error

Writer name: 'Microsoft Exchange Writer'
   Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Instance Id: {10a08fd1-ae91-4eb9-9ced-782f771132e9}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {e22cda15-668a-497e-8887-89558df266cd}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {5fae1b11-4eb3-43fb-a703-4c3873417763}
   State: [1] Stable
   Last error: No error

Writer name: 'NTDS'
   Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Writer Instance Id: {e14cd577-8ebc-4e0a-b419-fe0c31b68336}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {6c652e0b-434d-4bfa-8b7c-b6bec62ecb77}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {f38700e0-6310-4e50-964e-f192ec0ee5b1}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {cfdabd2b-2b63-4c3e-9fe2-d0557c08d437}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {fd14d1e0-227c-4f30-9c8a-e282b1764b20}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {9b6e96d3-92db-48b5-991b-9232682027fa}
   State: [1] Stable
   Last error: No error

Sorry to flood you with info, but I just wanted to list everything just in case 1 setting is causing all these headache.

[Jamie Pert]

weirdly the Microsoft Exchange writer is currently stable, this is not usually the case?! It seems I get one of these errors during both backups

On the server I have seen multiple VSS log in events, these are listed as "information" not errors or critical etc. Again I don't know whether this bares any relevance.

Code: Select all

Log Name:      Application
Source:        VSS
Date:          10/07/2012 23:10:14
Event ID:      8220
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server01.exchange.local
Description:
Ran out of time while deleting files. 

Operation:
   OnPostSnapshot event
   PostSnapshot Event

Context:
   Execution Context: Shadow Copy Optimization Writer
   Execution Context: Writer
   Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Name: Shadow Copy Optimization Writer
   Writer Instance ID: {c2df4052-3fef-4a71-b617-bb2fc1e9221f}
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="VSS" />
    <EventID Qualifiers="0">8220</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-07-10T22:10:14.000000000Z" />
    <EventRecordID>246821</EventRecordID>
    <Channel>Application</Channel>
    <Computer>server01.exchange.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>

Operation:
   OnPostSnapshot event
   PostSnapshot Event

Context:
   Execution Context: Shadow Copy Optimization Writer
   Execution Context: Writer
   Writer Class Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Name: Shadow Copy Optimization Writer
   Writer Instance ID: {c2df4052-3fef-4a71-b617-bb2fc1e9221f}</Data>
    <Binary>2D20436F64653A2057525444454C455430303030303331332D2043616C6C3A2057525444454C455430303030303234382D205049443A202030303030363832382D205449443A202030303031333130342D20434D443A2020433A5C57696E646F77735C73797374656D33325C76737376632E6578652020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820</Binary>
  </EventData>
</Event>

[Cokovic]

Wait...you are using vmware tools quiescence? I would disable it as it isn't really application aware and instead only use application aware image processing from Veeam. Untick this box of VMWare tools quiescence please.

Next i would suggest you to only truncate logs after a successfull backup as you could really run into trouble when trunkating logs immediatly before backup has processed its operations successfully.

Could you tell me the output of "vssadmin list providers"? The output should look like this:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7

Or do you see anything else besides the Microsoft Software Shadow Copy provider? vssadmin list writers looks like it should be.