Host-based backup of VMware vSphere VMs.
Post Reply
rowdy
Enthusiast
Posts: 54
Liked: never
Joined: Jan 19, 2010 12:59 pm
Contact:

VSS timeout with Exchange 2010

Post by rowdy »

Veeam support is unable to assist me with this, perhaps someone at the forum knows...

We recently upgraded to Exchange 2010 (single server, 18 users nothing fancy).
At first, Veeam with VSS was backing up just fine. Then, I migrated all 18(!) users from 2007 to 2010.
From that moment on, the first backup always failed on VSS, but the first retry always went okay. So, not really a big problem.

However, after several days backups started failing on VSS completely.
So, I rebooted the machine and the backup went okay once.
So, several days later I rebooted once more and the backup kept failing. I retried a gazillion times and the millionth time the backup went okay again.
Now, two days in a row the backup went okay the first time (without retry)... And now it's failing completely (on VSS) again.
Rebooting the machine or restarting Exchange sevices sometimes helps, but mostly it doesn't help.
So, in 1% of the cases the backup goes okay and 99% it fails...
This is really annoying me. I also tried increasing the memory assigned to Exchange, I read that somewhere on the internet, but that didn't help either.
The host machine is really very powerful and only hosts a handful of VM's.

The error reported by Veeam is:

"Freezing guest operating system
Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{0db23250-4d1e-42c1-8d14-2be32f448184}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] "

The Windows eventlogs have the following to say:
(each item is a new entry in the eventlog, with only 0 to 90 seconds between them.
  • Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6) has prepared for backup successfully.
  • Information Store (3460) Shadow copy instance 14 starting. This will be a Full shadow copy.
    For more information, click http://www.microsoft.com/contentredirect.asp.
  • Exchange VSS Writer (instance 14) has successfully prepared the database engine for a full or copy backup of database 'Public Folders 2010'.
  • Exchange VSS Writer (instance 14) has successfully prepared the database engine for a full or copy backup of database 'Mailbox Database 2058872270'.
  • Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6:14) has prepared for Snapshot successfully.
  • Information Store (3460) Shadow copy instance 14 freeze started.
  • Information Store (3460) Mailbox Database 2058872270: Shadow copy instance 14 freeze started.
  • Information Store (3460) Public Folders 2010: Shadow copy instance 14 freeze started.
  • Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6:14) has frozen the database(s) successfully.
  • Information Store (3460) Shadow copy instance 14 freeze ended.
  • Information Store (3460) Shadow copy instance 14 aborted. (EventID 2007)
VSSAdmin list writers:
(note: above eventlog entries are not the most recent, below VSSAdmin is the most recent - that's why instance ID's are different)

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {0db23250-4d1e-42c1-8d14-2be32f448184}
State: [9] Failed
Last error: Timed out



So, it only takes a few seconds to go from 'working' to 'timed out'. It's not trying for that long, so I wonder if a timeout is the real reason for the error.
itcaptain
Novice
Posts: 3
Liked: never
Joined: Nov 30, 2010 3:22 pm
Full Name: it captain

Re: VSS timeout with Exchange 2010

Post by itcaptain »

We have the same issue. We were able to do full and one incremental backup and now it's failing with the same error.
rowdy
Enthusiast
Posts: 54
Liked: never
Joined: Jan 19, 2010 12:59 pm
Contact:

Re: VSS timeout with Exchange 2010

Post by rowdy »

Veeam Support said it was an issue with Exchange, which is possible of course but the Windows eventlog seems ok to me... No real errors or anything.

I have found several VSS fixes for Windows 2003 and/or Exchange 2007, but we're running Exchange 2010 SP1 on Windows 2008 R2.
jwaynejones
Novice
Posts: 8
Liked: 1 time
Joined: Oct 07, 2010 1:55 pm
Full Name: Jeremy Jones
Contact:

Re: VSS timeout with Exchange 2010

Post by jwaynejones »

i've been seeing the same thing on our windows 2008/exchange 2007 environment. i have a support ticket open, but over two weeks later i still have barely heard from their support, and the only "fixes" presented are from MS, and only apply to windows 2003. if you find a solution, pass it along, please! :)

*edit: as a note, we also run a Backup Exec job (it runs at a different time from the Veeam job), and it also uses the Exchange VSS writer, and it works flawlessly, so i have a hard time believing it's related to Exchange.
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

rowdy wrote:VSSAdmin list writers:
(note: above eventlog entries are not the most recent, below VSSAdmin is the most recent - that's why instance ID's are different)

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {0db23250-4d1e-42c1-8d14-2be32f448184}
State: [9] Failed
Last error: Timed out
This indicates an issue with Microsoft VSS component. It would be best to open support case with Microsoft on this.
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

jwaynejones wrote:i have a support ticket open, but over two weeks later i still have barely heard from their support, and the only "fixes" presented are from MS
This is because VSS is Microsoft functionality... our code does nothing but notifying VSS that we want to perform a backup. Everything else is handled solely by VSS (which also talks to application specific writers as a part of system freeze). In the end, VSS is either able to freeze the system (and then it tells us "OK, you're good to go with backup"), or it is not successful, and then it tells us the reason why it failed which we are showing in the session log. So, there is absolutely nothing we could "fix" from our side if VSS framework itself, or specific VSS writer fails to do its job.
jwaynejones wrote:*edit: as a note, we also run a Backup Exec job (it runs at a different time from the Veeam job), and it also uses the Exchange VSS writer, and it works flawlessly, so i have a hard time believing it's related to Exchange.
There were quite a few reports on this forum stating that BackupExec agent prevents other VSS applications from working correctly (because it installs its own VSS components). Uninstalling it and rebooting VM used to help. I cannot be sure if this is the same issue, but worths trying. Thanks!
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

itcaptain wrote:We have the same issue. We were able to do full and one incremental backup and now it's failing with the same error.
What is the output of the following commands for you:

Code: Select all

vssadmin list writers
vssadmin list providers
jwaynejones
Novice
Posts: 8
Liked: 1 time
Joined: Oct 07, 2010 1:55 pm
Full Name: Jeremy Jones
Contact:

Re: VSS timeout with Exchange 2010

Post by jwaynejones »

other VSS-integrated products work fine. we only have the issue with Veeam.
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

What other VSS-integrated backup products are you talking about? Again, BackupExec bring its own VSS components (its own VSS provider), while Veeam uses native Microsoft VSS provider.
jwaynejones
Novice
Posts: 8
Liked: 1 time
Joined: Oct 07, 2010 1:55 pm
Full Name: Jeremy Jones
Contact:

Re: VSS timeout with Exchange 2010

Post by jwaynejones »

there were a few other VSS-based products that we tried over the past few year or so. i wasn't involved in their testing so i don't know what they were off hand (sorry). i did, however, find out that the VSS components were functioning properly. i'll see if i can try removing the Backup Exec agent and see if that helps. thanks for the tip :)
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: VSS timeout with Exchange 2010

Post by Bunce »

They wouldn't have been with 2010, and certainly not SP1 which has only been out a short time..
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

Actually, I just recalled we are mostly finishing our migration to Exchange 2010... I can check with admins if they have similar issue (since they are using Veeam Backup to protect it as well). Probably not since I am sure I would hear of any issues very fast.
jpjetlinx
Lurker
Posts: 2
Liked: never
Joined: Dec 01, 2010 9:33 pm
Full Name: Jeremy Parks
Contact:

Re: VSS timeout with Exchange 2010

Post by jpjetlinx »

I cam brand new user and still getting the software set up, but I can report that while backing up my Exchange 2010 server I am experiencing the same issue as the original post with the exception that I recieve the error on every attempt. Until the investment in Veeam, I was sucessfully using the built-in Windows Server Backup utility to backup the system. If I am not mistaken, I think the built-in utility uses the native Microsoft Exchange VSS writer to acomplish the backups so I am not sure how it could be related to the native VSS Writer. I really need a solution as this is one of the most critical servers to back up.
itcaptain
Novice
Posts: 3
Liked: never
Joined: Nov 30, 2010 3:22 pm
Full Name: it captain

Re: VSS timeout with Exchange 2010

Post by itcaptain »

Well I did uninstall the Symantec Backup Exec remote Agent from our Exchange 2010 server and restarted the server. Ran 2 incremental jobs and they were successful without any issues. I will keep everyone posted if there is any other issue related to this.

Thanks Gostev :)
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

Glad it worked, thanks for the update!

Also, I have just checked with our admins as promised, it appears that we have migrated 180 users to Exchange 2010 as of today, and have absolutely no issues backing it up with Veeam (VSS enabled, of course).
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: VSS timeout with Exchange 2010

Post by Bunce »

We've been backup up 2010 RTM for a few months now and no issues (apart from the DAG failover issue reported due to FCS timeouts), so keenly awaiting the XGE Item restore component.. :)
rowdy
Enthusiast
Posts: 54
Liked: never
Joined: Jan 19, 2010 12:59 pm
Contact:

Re: VSS timeout with Exchange 2010

Post by rowdy »

Well, today the backups decided it was a beautiful day and they started going okay again.
Absolutely nothing has changed, besides that we have only more VM's running on the same host now...
ZackAngelo
Novice
Posts: 6
Liked: never
Joined: Jan 10, 2011 2:47 am
Full Name: Zack Angelo
Contact:

Re: VSS timeout with Exchange 2010

Post by ZackAngelo »

I'm a brand new Veeam customer and I've just run into the same issue. I'm running Exchange 2010 SP1 on Windows Server 2K8 R2. I've just opened a case with support; from the looks of it, it doesn't look like there is much they can do.
c.bocquet
Lurker
Posts: 1
Liked: never
Joined: Feb 22, 2011 11:59 am
Contact:

Re: VSS timeout with Exchange 2010

Post by c.bocquet »

Same issue for me: 2008 R2 with Exchange 2010
Resolved by the uninstall of Backup Exec remote Agent and a reboot.

Thanks guys. :D
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: VSS timeout with Exchange 2010

Post by Bunce »

What does Backup Exec have to do with this issue?
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

BackupExec requires deploying an agent on each guest. This agent also brings custom VSS provider that "brakes" normal VSS operations unless I assume you do some special custom calls that their agent does. Veeam leverages standard VSS API calls, and those are affected. This is known issue, there were a few other topics about this over the course of last couple of years. Uninstall + reboot always helps.

I figured this is Symantec's last revenge to us before they get thrown away ;)
Bunce
Veteran
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: VSS timeout with Exchange 2010

Post by Bunce »

gotchya. Actually, I do recall seeing some options in the BE GUI where you can choose to use BE's provider or native.
adean
Novice
Posts: 3
Liked: never
Joined: Apr 04, 2011 9:40 pm
Full Name: Anne Dean
Contact:

Re: VSS timeout with Exchange 2010

Post by adean »

I'm having the same issue even though this server has never had any Symantec products installed. No other resolutions? I'm having the same experience as rowdy, same error message, same trouble with the MS Exchange VSS Writer... 2008 R2, Exchange 2010 SP1. My tried-and-true Veeam job was working great until one day, 'poof'. Thanks.
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

Hi Anne, do these commands show any issues with VSS writers or providers when you run them on your Exchange 2010 box?

Code: Select all

vssadmin list writers
vssadmin list providers
adean
Novice
Posts: 3
Liked: never
Joined: Apr 04, 2011 9:40 pm
Full Name: Anne Dean
Contact:

Re: VSS timeout with Exchange 2010

Post by adean »

Yes, same as Rowdy:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {8d784b70-e66f-430a-b876-afa4d7b31c0f}
State: [9] Failed
Last error: Timed out

Veeam Support was not able to offer assistance other than suggesting that I contact Microsoft. Thank you in advance for your help!
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

Yes, in this case you do indeed need to contact Microsoft directly for assistance. We do not possess source code of this VSS writer, nor knowledge of Microsoft Exchange VSS functionality troubleshooting via debug logs, so we are unable to assist here. This is not a finger pointing by any means, please understand that we simply cannot support the code we do not own. Thanks.
adean
Novice
Posts: 3
Liked: never
Joined: Apr 04, 2011 9:40 pm
Full Name: Anne Dean
Contact:

Re: VSS timeout with Exchange 2010

Post by adean »

We opened a case with Microsoft and they crawled all over the server insisting there was nothing wrong with VSS.

Several theories and support groups later, it turned out that our VSS issues were caused by low resources on the ESXi being used. Although the host didn’t appear to be fully taxed, apparently it was enough to slow down the response from the Exchange Writer. We migrated the server to a different host and the Veeam backups started working immediately.
Gostev
Chief Product Officer
Posts: 31780
Liked: 7280 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VSS timeout with Exchange 2010

Post by Gostev »

Hi Anne, thank you very much for taking time to update this post with the resolution.
1-0-1
Enthusiast
Posts: 58
Liked: 2 times
Joined: Nov 30, 2010 1:38 pm
Full Name: Bernd
Contact:

Re: VSS timeout with Exchange 2010

Post by 1-0-1 »

I have the same issue and last avenue is moving the VM onto a unpopulated dedicated host. Just out of interest sake - if I select Microsoft VSS option in my backup job (put a check mark next to 'Enable application image-aware processing') is it safe to assume I can disable/uncheck the 'Enable VMWARE tools quiescene'?
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: VSS timeout with Exchange 2010

Post by J1mbo »

Since it appears to be a time-out, I wonder if the underlying storage system is simply unable to complete the freeze in whatever time is allocated? Perhaps it would be helpful to understand the underlying storage in systems that see failures.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 23 guests