VSS timeout with Exchange 2010

[rowdy]

Veeam support is unable to assist me with this, perhaps someone at the forum knows...

We recently upgraded to Exchange 2010 (single server, 18 users nothing fancy).
At first, Veeam with VSS was backing up just fine. Then, I migrated all 18(!) users from 2007 to 2010.
From that moment on, the first backup always failed on VSS, but the first retry always went okay. So, not really a big problem.

However, after several days backups started failing on VSS completely.
So, I rebooted the machine and the backup went okay once.
So, several days later I rebooted once more and the backup kept failing. I retried a gazillion times and the millionth time the backup went okay again.
Now, two days in a row the backup went okay the first time (without retry)... And now it's failing completely (on VSS) again.
Rebooting the machine or restarting Exchange sevices sometimes helps, but mostly it doesn't help.
So, in 1% of the cases the backup goes okay and 99% it fails...
This is really annoying me. I also tried increasing the memory assigned to Exchange, I read that somewhere on the internet, but that didn't help either.
The host machine is really very powerful and only hosts a handful of VM's.

The error reported by Veeam is:

"Freezing guest operating system
Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{0db23250-4d1e-42c1-8d14-2be32f448184}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] "

The Windows eventlogs have the following to say:
(each item is a new entry in the eventlog, with only 0 to 90 seconds between them.

• Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6) has prepared for backup successfully.
• Information Store (3460) Shadow copy instance 14 starting. This will be a Full shadow copy.
  For more information, click http://www.microsoft.com/contentredirect.asp.
• Exchange VSS Writer (instance 14) has successfully prepared the database engine for a full or copy backup of database 'Public Folders 2010'.
• Exchange VSS Writer (instance 14) has successfully prepared the database engine for a full or copy backup of database 'Mailbox Database 2058872270'.
• Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6:14) has prepared for Snapshot successfully.
• Information Store (3460) Shadow copy instance 14 freeze started.
• Information Store (3460) Mailbox Database 2058872270: Shadow copy instance 14 freeze started.
• Information Store (3460) Public Folders 2010: Shadow copy instance 14 freeze started.
• Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6:14) has frozen the database(s) successfully.
• Information Store (3460) Shadow copy instance 14 freeze ended.
• Information Store (3460) Shadow copy instance 14 aborted. (EventID 2007)

VSSAdmin list writers:
(note: above eventlog entries are not the most recent, below VSSAdmin is the most recent - that's why instance ID's are different)

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {0db23250-4d1e-42c1-8d14-2be32f448184}
State: [9] Failed
Last error: Timed out



So, it only takes a few seconds to go from 'working' to 'timed out'. It's not trying for that long, so I wonder if a timeout is the real reason for the error.

[itcaptain]

We have the same issue. We were able to do full and one incremental backup and now it's failing with the same error.

[rowdy]

Veeam Support said it was an issue with Exchange, which is possible of course but the Windows eventlog seems ok to me... No real errors or anything.

I have found several VSS fixes for Windows 2003 and/or Exchange 2007, but we're running Exchange 2010 SP1 on Windows 2008 R2.

[jwaynejones]

i've been seeing the same thing on our windows 2008/exchange 2007 environment. i have a support ticket open, but over two weeks later i still have barely heard from their support, and the only "fixes" presented are from MS, and only apply to windows 2003. if you find a solution, pass it along, please!

*edit: as a note, we also run a Backup Exec job (it runs at a different time from the Veeam job), and it also uses the Exchange VSS writer, and it works flawlessly, so i have a hard time believing it's related to Exchange.

[Gostev]

VSSAdmin list writers:
(note: above eventlog entries are not the most recent, below VSSAdmin is the most recent - that's why instance ID's are different)

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {0db23250-4d1e-42c1-8d14-2be32f448184}
State: [9] Failed
Last error: Timed out

This indicates an issue with Microsoft VSS component. It would be best to open support case with Microsoft on this.

[Gostev]

i have a support ticket open, but over two weeks later i still have barely heard from their support, and the only "fixes" presented are from MS

This is because VSS is Microsoft functionality... our code does nothing but notifying VSS that we want to perform a backup. Everything else is handled solely by VSS (which also talks to application specific writers as a part of system freeze). In the end, VSS is either able to freeze the system (and then it tells us "OK, you're good to go with backup"), or it is not successful, and then it tells us the reason why it failed which we are showing in the session log. So, there is absolutely nothing we could "fix" from our side if VSS framework itself, or specific VSS writer fails to do its job.

*edit: as a note, we also run a Backup Exec job (it runs at a different time from the Veeam job), and it also uses the Exchange VSS writer, and it works flawlessly, so i have a hard time believing it's related to Exchange.

There were quite a few reports on this forum stating that BackupExec agent prevents other VSS applications from working correctly (because it installs its own VSS components). Uninstalling it and rebooting VM used to help. I cannot be sure if this is the same issue, but worths trying. Thanks!

[Gostev]

We have the same issue. We were able to do full and one incremental backup and now it's failing with the same error.

What is the output of the following commands for you:

Code: Select all

vssadmin list writers
vssadmin list providers

[jwaynejones]

other VSS-integrated products work fine. we only have the issue with Veeam.

[Gostev]

What other VSS-integrated backup products are you talking about? Again, BackupExec bring its own VSS components (its own VSS provider), while Veeam uses native Microsoft VSS provider.

[jwaynejones]

there were a few other VSS-based products that we tried over the past few year or so. i wasn't involved in their testing so i don't know what they were off hand (sorry). i did, however, find out that the VSS components were functioning properly. i'll see if i can try removing the Backup Exec agent and see if that helps. thanks for the tip

[Bunce]

They wouldn't have been with 2010, and certainly not SP1 which has only been out a short time..

[Gostev]

Actually, I just recalled we are mostly finishing our migration to Exchange 2010... I can check with admins if they have similar issue (since they are using Veeam Backup to protect it as well). Probably not since I am sure I would hear of any issues very fast.

[jpjetlinx]

I cam brand new user and still getting the software set up, but I can report that while backing up my Exchange 2010 server I am experiencing the same issue as the original post with the exception that I recieve the error on every attempt. Until the investment in Veeam, I was sucessfully using the built-in Windows Server Backup utility to backup the system. If I am not mistaken, I think the built-in utility uses the native Microsoft Exchange VSS writer to acomplish the backups so I am not sure how it could be related to the native VSS Writer. I really need a solution as this is one of the most critical servers to back up.

[itcaptain]

Well I did uninstall the Symantec Backup Exec remote Agent from our Exchange 2010 server and restarted the server. Ran 2 incremental jobs and they were successful without any issues. I will keep everyone posted if there is any other issue related to this.

Thanks Gostev

[Gostev]

Glad it worked, thanks for the update!

Also, I have just checked with our admins as promised, it appears that we have migrated 180 users to Exchange 2010 as of today, and have absolutely no issues backing it up with Veeam (VSS enabled, of course).

[Bunce]

We've been backup up 2010 RTM for a few months now and no issues (apart from the DAG failover issue reported due to FCS timeouts), so keenly awaiting the XGE Item restore component..

[rowdy]

Well, today the backups decided it was a beautiful day and they started going okay again.
Absolutely nothing has changed, besides that we have only more VM's running on the same host now...

[ZackAngelo]

I'm a brand new Veeam customer and I've just run into the same issue. I'm running Exchange 2010 SP1 on Windows Server 2K8 R2. I've just opened a case with support; from the looks of it, it doesn't look like there is much they can do.

[c.bocquet]

Same issue for me: 2008 R2 with Exchange 2010
Resolved by the uninstall of Backup Exec remote Agent and a reboot.

Thanks guys.

[Bunce]

What does Backup Exec have to do with this issue?

[Gostev]

BackupExec requires deploying an agent on each guest. This agent also brings custom VSS provider that "brakes" normal VSS operations unless I assume you do some special custom calls that their agent does. Veeam leverages standard VSS API calls, and those are affected. This is known issue, there were a few other topics about this over the course of last couple of years. Uninstall + reboot always helps.

I figured this is Symantec's last revenge to us before they get thrown away

[Bunce]

gotchya. Actually, I do recall seeing some options in the BE GUI where you can choose to use BE's provider or native.

[adean]

I'm having the same issue even though this server has never had any Symantec products installed. No other resolutions? I'm having the same experience as rowdy, same error message, same trouble with the MS Exchange VSS Writer... 2008 R2, Exchange 2010 SP1. My tried-and-true Veeam job was working great until one day, 'poof'. Thanks.

[Gostev]

Hi Anne, do these commands show any issues with VSS writers or providers when you run them on your Exchange 2010 box?

Code: Select all

vssadmin list writers
vssadmin list providers

[adean]

Yes, same as Rowdy:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {8d784b70-e66f-430a-b876-afa4d7b31c0f}
State: [9] Failed
Last error: Timed out

Veeam Support was not able to offer assistance other than suggesting that I contact Microsoft. Thank you in advance for your help!

[Gostev]

Yes, in this case you do indeed need to contact Microsoft directly for assistance. We do not possess source code of this VSS writer, nor knowledge of Microsoft Exchange VSS functionality troubleshooting via debug logs, so we are unable to assist here. This is not a finger pointing by any means, please understand that we simply cannot support the code we do not own. Thanks.

[adean]

We opened a case with Microsoft and they crawled all over the server insisting there was nothing wrong with VSS.

Several theories and support groups later, it turned out that our VSS issues were caused by low resources on the ESXi being used. Although the host didn’t appear to be fully taxed, apparently it was enough to slow down the response from the Exchange Writer. We migrated the server to a different host and the Veeam backups started working immediately.

[Gostev]

Hi Anne, thank you very much for taking time to update this post with the resolution.

[1-0-1]

I have the same issue and last avenue is moving the VM onto a unpopulated dedicated host. Just out of interest sake - if I select Microsoft VSS option in my backup job (put a check mark next to 'Enable application image-aware processing') is it safe to assume I can disable/uncheck the 'Enable VMWARE tools quiescene'?

[J1mbo]

Since it appears to be a time-out, I wonder if the underlying storage system is simply unable to complete the freeze in whatever time is allocated? Perhaps it would be helpful to understand the underlying storage in systems that see failures.