VSS timeout with Exchange 2010

VMware specific discussions

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Sat Aug 09, 2014 9:49 pm

Ouh... and don't forgett to restart veeam backup service after you create/change the registry value.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby richardb092 » Sun Aug 10, 2014 10:15 pm

I think it has been touched on, but other VSS providers can cause this. In my case it was not exchange, but on a server I had the Dell MD software to manage my MD3600i. As soon as I removed this software and its custom VSS provider, all was well.
I think that this will be the error here. I think some backup agents will cause this (Symantec yes, but Tivoli is ok.), but not all...
richardb092
Novice
 
Posts: 5
Liked: never
Joined: Tue Oct 29, 2013 10:10 am
Full Name: Richard Bance

Re: VSS timeout with Exchange 2010

Veeam Logoby Wad4iPod » Wed Aug 13, 2014 5:27 am

I have encountered this with Exchange 2010 on Win2k8R2.

Performing this (we're not replicating only BU but it works for the VSS timeout issue) - has resolved the VSS Timeout issue until the next VM reboot (so it seems).
Originally found here:
vmware-vsphere-f24/vss-timeout-with-exchange-2010-t5760-180.html

----------------------------------------------------------------------------------------------------------------------
We were experiencing VSS timeouts when replicating Exchange 2010 servers, and we found out that if we
restart the "COM+ Event System" service prior to replicating, it seems to work everytime.
Wad4iPod
Enthusiast
 
Posts: 69
Liked: never
Joined: Wed Aug 04, 2010 12:34 am

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Wed Aug 13, 2014 6:55 am

Be carefull with this, restarting this service will empty the vss writer list for some time, which lead into successfull but crash consistent backups. The correct script, restart service and wait till exchange writter is listed at vssadmin list writers back again.

But then you will run into the same timeout most likely again.

This way is helpful when you backup DAG members with only inactive databases. Then you can restart com+ and exchange information store and then wait till writers are back to perform backup.

Restarting information store will lead into significant faster vss prozessing.

My update list of tips and tricks can be found now at my new blog:
http://andyandthevms.com/exchange-dag-v ... plication/
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010 - have fix

Veeam Logoby eholz1 » Wed Aug 20, 2014 5:46 pm

Hello All,
I have a fix/workaround for this. Works every time. I create a task in the task scheduler (Windows SBS 2011, etc)
This task stops and starts the service 5 minutes before the backup runs. Hence the VSS writer is running
when the backup goes.
create a cmd or bat file with the following: net stop MSExchangeIS && net start MSExchangeIS

use this bat file in the task you create. NOTE create a BASIC task, the run program, check "run if user not logged in"

Thanks Veeam,
eholz1
eholz1
Lurker
 
Posts: 1
Liked: never
Joined: Wed Aug 20, 2014 5:39 pm

Re: VSS timeout with Exchange 2010 - have fix

Veeam Logoby Andreas Neufert » Wed Aug 20, 2014 9:51 pm

eholz1 wrote:Hello All,
I have a fix/workaround for this. Works every time. I create a task in the task scheduler (Windows SBS 2011, etc)
This task stops and starts the service 5 minutes before the backup runs. Hence the VSS writer is running
when the backup goes.
create a cmd or bat file with the following: net stop MSExchangeIS && net start MSExchangeIS

use this bat file in the task you create. NOTE create a BASIC task, the run program, check "run if user not logged in"

Thanks Veeam,
eholz1



Thanks for sharing this. Yes, a restart of information store will help in that case, while not so much data is open and processed, Exchange consistency can be created much faster.
This is a good example for a Microsoft own workaround and show that not in any case the backup vendor can help with a solution...
For those who want to use this, keep in mind that this workaround came with short Exchange service downtime.

If you do this at nearly same time at all DAG members, you will see extra load at indexing service. Keep an eye on it.

Again ... v8 will bring us additional interresting things. In the meantime check my tips above or use this workaround.

CU Andy
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby AdrianHinton » Fri Jan 02, 2015 2:37 am

I really hope V8 fixes this issue. One of my mail DB's logs folder is at 60GB now. And I'm copy-syncing this offsite! Also, IMHO, this is Veeam's issue. Veeam have written an application that is not interacting with the available interfaces in the way the interface designers have specified and we are now stuck with the problem.
AdrianHinton
Enthusiast
 
Posts: 71
Liked: 5 times
Joined: Mon Apr 07, 2014 10:00 am
Full Name: Adrian Hinton

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Fri Jan 02, 2015 9:45 am 1 person likes this post

v8 is out since 6. Nov 2014. Also v8 Patch1 is out now as well (for those who do not implement vanilla releases).

v8 Exchange Enhancements:

Microsoft Exchange VSS improvements:
If standard VSS processing of a Microsoft Exchange Server times out, the job will retry processing using persistent in-guest VSS snapshot, which should prevent
VSS processing timeouts commonly observed with Microsoft Exchange 2010. Persistent snapshots are removed from the production VMs once backups are finished. All VM, file-level and item-level restore options have been enhanced to detect and properly handle restores from Exchange backups created with the help of persistent VSS snapshots.

Veeam Explorer for Microsoft Exchange
Veeam Explorer for Microsoft Exchange brings additional functionality, making granular Exchange restores easier and more efficient than ever. New features include:
• Web-based restore portal for delegated help desk operators enables secure mailbox items recoveries without restore operators having access to the content of restored items
• Ability to recover hard-deleted items that you thought might have been lost forever
• Support for searching of and recovering to Online Archive mailboxes
• Restore of Public Folder permissions (in addition to Public Folder contents)
• Support for item-level recovery from secondary (passive) DAG node backup

v8 Patch1 bring in addition:

Veeam Explorer for Exchange
• Process of exporting email messages cannot be stopped.
• Copying or saving attachments from multiple different emails always copies or saves attachment from the first email for which this operation was performed.
• Opening public folder’s mailbox fails with the "Unable to find wastebasket" error.

Overall the majority of our customers that use B&R with Exchange are very happy and do not experience this VSS timeout. Sometimes it is really hard to analyse the root cause of such a problem. I worked with some of the above mentioned customers to find the root causes and documented everything in the above mentioned blog post. Anyway Veeam added in v8 a way to bypass the complicated root cause analyse by using a different VSS mode when the problem occure. I personally suggest to analyse your Exchange System if B&R use this new method as there is somewhere a infrastructure bottleneck or misconfiguration that lead to it.

I´m looking forward to your feedback after v8 upgrade.
Thanks Andy
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby zmedgyesi » Mon Jan 05, 2015 10:04 am 1 person likes this post

For us, v8 solved the problem.
zmedgyesi
Novice
 
Posts: 4
Liked: 2 times
Joined: Sat Dec 06, 2014 10:04 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby AdrianHinton » Tue Jan 06, 2015 11:18 pm 1 person likes this post

V8 has also solved the problem for me.
AdrianHinton
Enthusiast
 
Posts: 71
Liked: 5 times
Joined: Mon Apr 07, 2014 10:00 am
Full Name: Adrian Hinton

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Wed Jan 07, 2015 9:52 am

Thanks for the feedback.

Someone still has problems with Exchange 2010 after v8 Patch 1?
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby sibinsam » Thu Jan 08, 2015 4:35 am 1 person likes this post

V8 solved the issue for us too!!!!
sibinsam
Veeam ProPartner
 
Posts: 64
Liked: 6 times
Joined: Mon Nov 03, 2014 5:31 am
Full Name: Sibin Sam

Re: VSS timeout with Exchange 2010

Veeam Logoby kyoon@casusa.com » Thu Jan 22, 2015 4:52 pm

Veeam CASE # 00733881.

Hello, I started testing VEEAM to backup a exchange 2010 VM with following setup.

disk1: OS(VMDK)
disk2: database 1 db(VMDK)
disk3: database 1 log(VMDK)
disk4: database 2 db(in-guest iscsi - Veeam didn't see this)
disk5: database 2 log(in-guest iscsi - Veeam didn't see this)
disk6: database 3 db(in-guest iscsi - Veeam didn't see this)
disk7: database 3 log(in-guest iscsi - Veeam didn't see this)
disk8: database 4 db(in-guest iscsi - Veeam didn't see this)
disk9: database 4 log(in-guest iscsi - Veeam didn't see this)
disk10: database 5 db(in-guest iscsi - Veeam didn't see this)
disk11: database 5 log(in-guest iscsi - Veeam didn't see this)
disk12: database 6 db(in-guest iscsi - Veeam didn't see this)
disk13: database 6 log(in-guest iscsi - Veeam didn't see this)

Initially, only disk 1~3 were presented at Vsphere level and disk 4~ 13 were connected using guest iscsi to SAN directly. Veeam backup was running fine at the time. With the good result i had, i moved all disks to hyper-visor level and backup job started to fail with VSS timed out error. I've been researching and reading the thread in the forum. And it seems to be related to 20 second limit per veeam KB1680. There are many suggestions to mitigate the problem on this forum. I tried to register ESXi host instead of vSPHERE. Now, the job is running fine in network mode instead of hotadd mode being used. Does this imply that there's resource problem at vCenter VM or exchange server VM? If backup is ran at VM under ESXi server, hotadd is not supported?
kyoon@casusa.com
Novice
 
Posts: 9
Liked: never
Joined: Mon Dec 29, 2014 7:01 pm
Full Name: Ken Yoon

Re: VSS timeout with Exchange 2010

Veeam Logoby Vitaliy S. » Thu Jan 22, 2015 5:11 pm

kyoon@casusa.com wrote:There are many suggestions to mitigate the problem on this forum. I tried to register ESXi host instead of vSPHERE. Now, the job is running fine in network mode instead of hotadd mode being used. Does this imply that there's resource problem at vCenter VM or exchange server VM? If backup is ran at VM under ESXi server, hotadd is not supported?

Hot=add backup mode does not depend on the vCenter Server presence. Does your proxy server has access to the datastores of the backed up VM? Do you see the same VSS issue when running backup through standalone host? Also do you have the same errors if you use network mode and vCenter Server connection?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Thu Jan 22, 2015 6:11 pm

Backup mode has nothing to do with VSS Timouts, as the Backup modes are only used after vss release.
If direct ESX access is faster than over vcenter (which is the case as no VSS timout came up), you should analyse your vcenter. The simplest way is to analyse the Task pannel of vsphere client. If the snapshot command recived time and command start time is not in the same second, most likely the vcenter is a bit slower than expected.

Install Backup & Replication v8 Patch1, will help to address this issue in general with a new VSS Snapshot mode on demand.(See above in the thread).

In the Veeam KB there are several good hints for the hotadd not possible at direct ESX connection.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 12 guests