VSS timeout with Exchange 2010

[Andreas Neufert]

Thanks for this feedback. If I run into this situation with a customer again. I will check your solution out and try to find the root cause, because I can not imagine why disabling the vcenter solved your VSS timeout problems. But again, thanks for sharing.

[Vitaliy S.]

The only thing I have ever found that works is to turn vCentre server off. If there is a vCentre server running anywhere on the networks the VSS times out. This is strange because no Veeam backups or replicas I have configured use vCentre server VIX connections, they are all configured directly through standalone hosts.

That's really strange, do you have both job logs to compare the procedures triggered with vCenter Server powered On and Off?

[cliffm]

Hi Vitaly,

I went through this for weeks with support some 3 years ago. In the end we could not get VSS to work at all and I stumbled on the vCentre idea by accident. Support reproduced my environment and could not get it to work either. Last month I upgraded to ESXi5 and decided to try out vCentre 5 again, thinking maybe things are better now. They are certainly faster with bigger hardware. Bang, almost immediately I started getting VSS errors in SBS backups that have been reliable for years. The jobs are not configured through vCentre. I am again frustrated and at the point of turning it off, again. I really don't want to spend more time on this so in short no, I don't have the logs. Also have just moved B&R today to a new server, migrated the database and settings etc. and turned the old one off. I am having VSS failures so if they don't stop the vCentre is in mothballs again for me.

[cliffm]

And just to put the icing on the cake, I just turned off vCentre and my SBS is happily producing snapshots again.

[Vitaliy S.]

Well...if a pure standalone connection to ESXi hosts is used every time, then vCenter Server power state should not affect anything. I will ask our QC team if they've seen anything similar before, thanks for the feedback.

[nmace]

I have not read through all of this giant thread, but for 3 days my Exchange backups have failed with a "Error: VSSControl: Failed to freeze guest, wait timeout".

Here are the things I've already tried courtesy of http://www.veeam.com/kb1680.

I've rebooted Exchange (no effect), I've connected to the Exchange VM via the host and not vCenter (no effect), and I've done an Exchange backup using the built in Windows Backup Tool. That succeeded fine. Running "vssadmin list writers" reports that the Microsoft Exchange Writer has timed out. Per the logs in vCenter, all of my Exchange snapshot creation times are about 22-25 seconds, which is outside of the 20 second window. This started happening as we moved more and more people onto the Exchange VM from a physical Exchange server. Am I really looking at the prospect of needing faster storage? Would I be able to utilize Storage IO Control on the VM instead to make sure it's getting enough disk performance instead? We have Enterprise+ for this cluster's vSphere hosts.

Is there anything else I should be trying?

[Vitaliy S.]

Moving this VM to a faster storage should definitely help. On a side note, seems like the latest recommendation from Andy can also help > http://forums.veeam.com/vmware-vsphere- ... ml#p101378

[kinger]

Hello, im new in this forum however I have problem with costumer - he faced the VIX unfreeze timeout error. How Can I fix it?

[veremin]

Hi, Wojtek,

If I were you, I would analyse thoroughly the existing KB article regarding this matter, and see whether the tips mentioned there helps to solve VSS timeout issue. If nothing helped, I would probably start thinking about new faster storage.

Thanks

[anubis]

Hello,

regarding the article http://www.veeam.com/kb1680. Could you please confirm, if I understood the process right? :

1- Veeam requests (via admion share or VIX) the VSS service to prepare for a backup (freeze the disks).
2- The VSS service freezes the disks
3- Veeam/VMware creates a snapshot (using Vcenter/ESXi)
4- Veeam/Vmware notifies the Microsoft VSS service that it can unfreeze the disks.
5- Veeam copies the data from the Vmware Snapshot

Between the steps 2 and 4, we should have less than 20 seconds, or the Exchange Writer will unfreeze the disks aborting the process.

From the article : "So, between step 2.1(when countdown begins) and step 2.3 we have only 20 seconds. If Veeam doesn't freeze the VM in timely manner, Windows VSS subsystem does it by itself. Hence you get unfreeze error." . Shouldnt this read "Windows VSS subsystem unfreezes the disks by itself"?

This line is also listed in the solutions : "VMware snapshot takes longer than 20 seconds (hardcoded Exchange VSS Writer timeout)" .. This should be part of the cause, not the solution IMHO.

[Andreas Neufert]

Hi Anubis,

thanks for the feedback.
You are right with the timeline...
1- => We will start an VSS requestor and this triggers the VSS writers.

I think the message came from Exchange VSS framwork itself and is bypassed to the Veeam UI.

I have created a good blog post with tips an tricks round about this:
http://neufert-at-veeam.blogspot.de/201 ... dated.html

[sambooka]

I have been banging my head with this issue for 3 weeks now Case # 00572707. I consider the issue resolved in my case and am about to move on. This will not be a solution for everyone but maybe it applies to someone and will help. No one at veeam was able to confirm my hypotheses so I just did it. It worked.

Background:
Exchange 2010 SP2 RU5.
2 node DAG
All Active DBs on Node1
All passive DBs on Node 2
PUB folder on Node 1 and 2

VSS Freeze timeouts on back. No log tuncation.

Still failing. We see the failures are specifically with the PUB folder DB.
Tried a cagillion things. Reregistering the VSS dlls .. restarting services. Doubled the CPU on Node 2 (passive node). Pretty much everything in this thread. Spoke to Nutanix support. Nada.. Spoke with Veeam support. VMWare support says nothing wrong on their end.. talk to Veeam. Veeam Tier2 says it is a VSS issue.. try to contact MS. I have not contacted MS but have read that MS doesnt not support snapshoting technologies and they dont support NFS storage (Nutanix). So I removed the Pub Folder DB from the server. Full backup complete. Logs Truncated. Exchange Explorer for Veeam sees the stores. Perfect. I have not tried a full restore of the server but dont see why it wouldnt work.

TLDR: VSSTimeouts on passive node of Exchange Dag. Fix: Passive node had Pub folder DB for HA. Removed the pub folder. Resolved.

[Andreas Neufert]

Thank you very much for figuring this out. Great work. I will let support know and update my recommendation list soon.
Also I will test it in my environment. Can you please share the configuration of the public folder dbs with me (screenshots or PS output) so that I have a higher chance to reproduce that. andreas.neufert(at)veeam.com. Thanks in advance...Andy

[bairda]

I have had the VSS timeout errors and the ONLY fix that worked was enabling a network connection between the Veeam B&R server and the problem exchange 2010 servers.

With this connection you must either disable the firewall or put in rules to pass the traffic. Far from a perfect solution, but it is the only way that the freezes would come in under 20 seconds and proceed with the application aware exchange backups via Veeam. This works via Vcenter VM add or via ESXi host add.

I love the Veeam product but I wish Veeam would fix this particular issue or just state that you need to put a network connection to the vB&R and the exchange server VM's.

Yes, MS needs to lengthen the Timeout on the freeze. But those are the cards that Veeam is currently dealt and the conditions it has to work under. The connection-less, VIX method of exchange snapshots for Veeam backups don't seem to work consistently.

BTW...this doesn't seem to impact any other backups except Exchange. Those can go without a network connection between the veeam B&R and the VM's.

[Andreas Neufert]

Hi, did you tried to reverse Network/VIX prozessing as descriebed in my guide?
If not, B&R wait for network timeout first and then process VIX based processing.

If you only perform VIX processing, this is a very good idea in any way.

I saw some cool improvements in v8beta1 as well, if you have the chance to test it with your Veeam SE let us know here.
As you process VIX only, the registry setting for reversing processing order needs to be set as well.

[Andreas Neufert]

Ouh... and don't forgett to restart veeam backup service after you create/change the registry value.

[richardb092]

I think it has been touched on, but other VSS providers can cause this. In my case it was not exchange, but on a server I had the Dell MD software to manage my MD3600i. As soon as I removed this software and its custom VSS provider, all was well.
I think that this will be the error here. I think some backup agents will cause this (Symantec yes, but Tivoli is ok.), but not all...

[Wad4iPod]

I have encountered this with Exchange 2010 on Win2k8R2.

Performing this (we're not replicating only BU but it works for the VSS timeout issue) - has resolved the VSS Timeout issue until the next VM reboot (so it seems).
Originally found here:
http://forums.veeam.com/vmware-vsphere- ... 0-180.html

----------------------------------------------------------------------------------------------------------------------
We were experiencing VSS timeouts when replicating Exchange 2010 servers, and we found out that if we
restart the "COM+ Event System" service prior to replicating, it seems to work everytime.

[Andreas Neufert]

Be carefull with this, restarting this service will empty the vss writer list for some time, which lead into successfull but crash consistent backups. The correct script, restart service and wait till exchange writter is listed at vssadmin list writers back again.

But then you will run into the same timeout most likely again.

This way is helpful when you backup DAG members with only inactive databases. Then you can restart com+ and exchange information store and then wait till writers are back to perform backup.

Restarting information store will lead into significant faster vss prozessing.

My update list of tips and tricks can be found now at my new blog:
http://andyandthevms.com/exchange-dag-v ... plication/

[eholz1]

Hello All,
I have a fix/workaround for this. Works every time. I create a task in the task scheduler (Windows SBS 2011, etc)
This task stops and starts the service 5 minutes before the backup runs. Hence the VSS writer is running
when the backup goes.
create a cmd or bat file with the following: net stop MSExchangeIS && net start MSExchangeIS

use this bat file in the task you create. NOTE create a BASIC task, the run program, check "run if user not logged in"

Thanks Veeam,
eholz1

[Andreas Neufert]

Hello All,
I have a fix/workaround for this. Works every time. I create a task in the task scheduler (Windows SBS 2011, etc)
This task stops and starts the service 5 minutes before the backup runs. Hence the VSS writer is running
when the backup goes.
create a cmd or bat file with the following: net stop MSExchangeIS && net start MSExchangeIS

use this bat file in the task you create. NOTE create a BASIC task, the run program, check "run if user not logged in"

Thanks Veeam,
eholz1

Thanks for sharing this. Yes, a restart of information store will help in that case, while not so much data is open and processed, Exchange consistency can be created much faster.
This is a good example for a Microsoft own workaround and show that not in any case the backup vendor can help with a solution...
For those who want to use this, keep in mind that this workaround came with short Exchange service downtime.

If you do this at nearly same time at all DAG members, you will see extra load at indexing service. Keep an eye on it.

Again ... v8 will bring us additional interresting things. In the meantime check my tips above or use this workaround.

CU Andy

[AdrianHinton]

I really hope V8 fixes this issue. One of my mail DB's logs folder is at 60GB now. And I'm copy-syncing this offsite! Also, IMHO, this is Veeam's issue. Veeam have written an application that is not interacting with the available interfaces in the way the interface designers have specified and we are now stuck with the problem.

[Andreas Neufert]

v8 is out since 6. Nov 2014. Also v8 Patch1 is out now as well (for those who do not implement vanilla releases).

v8 Exchange Enhancements:

Microsoft Exchange VSS improvements:
If standard VSS processing of a Microsoft Exchange Server times out, the job will retry processing using persistent in-guest VSS snapshot, which should prevent
VSS processing timeouts commonly observed with Microsoft Exchange 2010. Persistent snapshots are removed from the production VMs once backups are finished. All VM, file-level and item-level restore options have been enhanced to detect and properly handle restores from Exchange backups created with the help of persistent VSS snapshots.

Veeam Explorer for Microsoft Exchange
Veeam Explorer for Microsoft Exchange brings additional functionality, making granular Exchange restores easier and more efficient than ever. New features include:
• Web-based restore portal for delegated help desk operators enables secure mailbox items recoveries without restore operators having access to the content of restored items
• Ability to recover hard-deleted items that you thought might have been lost forever
• Support for searching of and recovering to Online Archive mailboxes
• Restore of Public Folder permissions (in addition to Public Folder contents)
• Support for item-level recovery from secondary (passive) DAG node backup

v8 Patch1 bring in addition:

Veeam Explorer for Exchange
• Process of exporting email messages cannot be stopped.
• Copying or saving attachments from multiple different emails always copies or saves attachment from the first email for which this operation was performed.
• Opening public folder’s mailbox fails with the "Unable to find wastebasket" error.

Overall the majority of our customers that use B&R with Exchange are very happy and do not experience this VSS timeout. Sometimes it is really hard to analyse the root cause of such a problem. I worked with some of the above mentioned customers to find the root causes and documented everything in the above mentioned blog post. Anyway Veeam added in v8 a way to bypass the complicated root cause analyse by using a different VSS mode when the problem occure. I personally suggest to analyse your Exchange System if B&R use this new method as there is somewhere a infrastructure bottleneck or misconfiguration that lead to it.

I´m looking forward to your feedback after v8 upgrade.
Thanks Andy

[zmedgyesi]

For us, v8 solved the problem.

[AdrianHinton]

V8 has also solved the problem for me.

[Andreas Neufert]

Thanks for the feedback.

Someone still has problems with Exchange 2010 after v8 Patch 1?

[sibinsam]

V8 solved the issue for us too!!!!

[kyoon@casusa.com]

Veeam CASE # 00733881.

Hello, I started testing VEEAM to backup a exchange 2010 VM with following setup.

disk1: OS(VMDK)
disk2: database 1 db(VMDK)
disk3: database 1 log(VMDK)
disk4: database 2 db(in-guest iscsi - Veeam didn't see this)
disk5: database 2 log(in-guest iscsi - Veeam didn't see this)
disk6: database 3 db(in-guest iscsi - Veeam didn't see this)
disk7: database 3 log(in-guest iscsi - Veeam didn't see this)
disk8: database 4 db(in-guest iscsi - Veeam didn't see this)
disk9: database 4 log(in-guest iscsi - Veeam didn't see this)
disk10: database 5 db(in-guest iscsi - Veeam didn't see this)
disk11: database 5 log(in-guest iscsi - Veeam didn't see this)
disk12: database 6 db(in-guest iscsi - Veeam didn't see this)
disk13: database 6 log(in-guest iscsi - Veeam didn't see this)

Initially, only disk 1~3 were presented at Vsphere level and disk 4~ 13 were connected using guest iscsi to SAN directly. Veeam backup was running fine at the time. With the good result i had, i moved all disks to hyper-visor level and backup job started to fail with VSS timed out error. I've been researching and reading the thread in the forum. And it seems to be related to 20 second limit per veeam KB1680. There are many suggestions to mitigate the problem on this forum. I tried to register ESXi host instead of vSPHERE. Now, the job is running fine in network mode instead of hotadd mode being used. Does this imply that there's resource problem at vCenter VM or exchange server VM? If backup is ran at VM under ESXi server, hotadd is not supported?

[Vitaliy S.]

There are many suggestions to mitigate the problem on this forum. I tried to register ESXi host instead of vSPHERE. Now, the job is running fine in network mode instead of hotadd mode being used. Does this imply that there's resource problem at vCenter VM or exchange server VM? If backup is ran at VM under ESXi server, hotadd is not supported?

Hot=add backup mode does not depend on the vCenter Server presence. Does your proxy server has access to the datastores of the backed up VM? Do you see the same VSS issue when running backup through standalone host? Also do you have the same errors if you use network mode and vCenter Server connection?

[Andreas Neufert]

Backup mode has nothing to do with VSS Timouts, as the Backup modes are only used after vss release.
If direct ESX access is faster than over vcenter (which is the case as no VSS timout came up), you should analyse your vcenter. The simplest way is to analyse the Task pannel of vsphere client. If the snapshot command recived time and command start time is not in the same second, most likely the vcenter is a bit slower than expected.

Install Backup & Replication v8 Patch1, will help to address this issue in general with a new VSS Snapshot mode on demand.(See above in the thread).

In the Veeam KB there are several good hints for the hotadd not possible at direct ESX connection.