VSS timeout with Exchange 2010

VMware specific discussions

VSS timeout with Exchange 2010

Veeam Logoby rowdy » Wed Dec 01, 2010 2:23 pm

Veeam support is unable to assist me with this, perhaps someone at the forum knows...

We recently upgraded to Exchange 2010 (single server, 18 users nothing fancy).
At first, Veeam with VSS was backing up just fine. Then, I migrated all 18(!) users from 2007 to 2010.
From that moment on, the first backup always failed on VSS, but the first retry always went okay. So, not really a big problem.

However, after several days backups started failing on VSS completely.
So, I rebooted the machine and the backup went okay once.
So, several days later I rebooted once more and the backup kept failing. I retried a gazillion times and the millionth time the backup went okay again.
Now, two days in a row the backup went okay the first time (without retry)... And now it's failing completely (on VSS) again.
Rebooting the machine or restarting Exchange sevices sometimes helps, but mostly it doesn't help.
So, in 1% of the cases the backup goes okay and 99% it fails...
This is really annoying me. I also tried increasing the memory assigned to Exchange, I read that somewhere on the internet, but that didn't help either.
The host machine is really very powerful and only hosts a handful of VM's.

The error reported by Veeam is:

"Freezing guest operating system
Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{0db23250-4d1e-42c1-8d14-2be32f448184}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] "

The Windows eventlogs have the following to say:
(each item is a new entry in the eventlog, with only 0 to 90 seconds between them.

  • Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6) has prepared for backup successfully.
  • Information Store (3460) Shadow copy instance 14 starting. This will be a Full shadow copy.
    For more information, click http://www.microsoft.com/contentredirect.asp.
  • Exchange VSS Writer (instance 14) has successfully prepared the database engine for a full or copy backup of database 'Public Folders 2010'.
  • Exchange VSS Writer (instance 14) has successfully prepared the database engine for a full or copy backup of database 'Mailbox Database 2058872270'.
  • Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6:14) has prepared for Snapshot successfully.
  • Information Store (3460) Shadow copy instance 14 freeze started.
  • Information Store (3460) Mailbox Database 2058872270: Shadow copy instance 14 freeze started.
  • Information Store (3460) Public Folders 2010: Shadow copy instance 14 freeze started.
  • Exchange VSS Writer (instance dd3a9020-cd8a-410b-b91c-605388a497f6:14) has frozen the database(s) successfully.
  • Information Store (3460) Shadow copy instance 14 freeze ended.
  • Information Store (3460) Shadow copy instance 14 aborted. (EventID 2007)

VSSAdmin list writers:
(note: above eventlog entries are not the most recent, below VSSAdmin is the most recent - that's why instance ID's are different)

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {0db23250-4d1e-42c1-8d14-2be32f448184}
State: [9] Failed
Last error: Timed out



So, it only takes a few seconds to go from 'working' to 'timed out'. It's not trying for that long, so I wonder if a timeout is the real reason for the error.
rowdy
Enthusiast
 
Posts: 54
Liked: never
Joined: Tue Jan 19, 2010 12:59 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby itcaptain » Wed Dec 01, 2010 2:52 pm

We have the same issue. We were able to do full and one incremental backup and now it's failing with the same error.
itcaptain
Novice
 
Posts: 3
Liked: never
Joined: Tue Nov 30, 2010 3:22 pm
Full Name: it captain

Re: VSS timeout with Exchange 2010

Veeam Logoby rowdy » Wed Dec 01, 2010 3:36 pm

Veeam Support said it was an issue with Exchange, which is possible of course but the Windows eventlog seems ok to me... No real errors or anything.

I have found several VSS fixes for Windows 2003 and/or Exchange 2007, but we're running Exchange 2010 SP1 on Windows 2008 R2.
rowdy
Enthusiast
 
Posts: 54
Liked: never
Joined: Tue Jan 19, 2010 12:59 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby jwaynejones » Wed Dec 01, 2010 4:23 pm

i've been seeing the same thing on our windows 2008/exchange 2007 environment. i have a support ticket open, but over two weeks later i still have barely heard from their support, and the only "fixes" presented are from MS, and only apply to windows 2003. if you find a solution, pass it along, please! :)

*edit: as a note, we also run a Backup Exec job (it runs at a different time from the Veeam job), and it also uses the Exchange VSS writer, and it works flawlessly, so i have a hard time believing it's related to Exchange.
jwaynejones
Novice
 
Posts: 8
Liked: 1 time
Joined: Thu Oct 07, 2010 1:55 pm
Full Name: Jeremy Jones

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Wed Dec 01, 2010 7:35 pm

rowdy wrote:VSSAdmin list writers:
(note: above eventlog entries are not the most recent, below VSSAdmin is the most recent - that's why instance ID's are different)

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {0db23250-4d1e-42c1-8d14-2be32f448184}
State: [9] Failed
Last error: Timed out

This indicates an issue with Microsoft VSS component. It would be best to open support case with Microsoft on this.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Wed Dec 01, 2010 7:40 pm

jwaynejones wrote:i have a support ticket open, but over two weeks later i still have barely heard from their support, and the only "fixes" presented are from MS

This is because VSS is Microsoft functionality... our code does nothing but notifying VSS that we want to perform a backup. Everything else is handled solely by VSS (which also talks to application specific writers as a part of system freeze). In the end, VSS is either able to freeze the system (and then it tells us "OK, you're good to go with backup"), or it is not successful, and then it tells us the reason why it failed which we are showing in the session log. So, there is absolutely nothing we could "fix" from our side if VSS framework itself, or specific VSS writer fails to do its job.

jwaynejones wrote:*edit: as a note, we also run a Backup Exec job (it runs at a different time from the Veeam job), and it also uses the Exchange VSS writer, and it works flawlessly, so i have a hard time believing it's related to Exchange.

There were quite a few reports on this forum stating that BackupExec agent prevents other VSS applications from working correctly (because it installs its own VSS components). Uninstalling it and rebooting VM used to help. I cannot be sure if this is the same issue, but worths trying. Thanks!
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Wed Dec 01, 2010 7:41 pm

itcaptain wrote:We have the same issue. We were able to do full and one incremental backup and now it's failing with the same error.

What is the output of the following commands for you:
Code: Select all
vssadmin list writers
vssadmin list providers
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VSS timeout with Exchange 2010

Veeam Logoby jwaynejones » Wed Dec 01, 2010 7:49 pm

other VSS-integrated products work fine. we only have the issue with Veeam.
jwaynejones
Novice
 
Posts: 8
Liked: 1 time
Joined: Thu Oct 07, 2010 1:55 pm
Full Name: Jeremy Jones

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Wed Dec 01, 2010 8:09 pm

What other VSS-integrated backup products are you talking about? Again, BackupExec bring its own VSS components (its own VSS provider), while Veeam uses native Microsoft VSS provider.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VSS timeout with Exchange 2010

Veeam Logoby jwaynejones » Wed Dec 01, 2010 8:27 pm

there were a few other VSS-based products that we tried over the past few year or so. i wasn't involved in their testing so i don't know what they were off hand (sorry). i did, however, find out that the VSS components were functioning properly. i'll see if i can try removing the Backup Exec agent and see if that helps. thanks for the tip :)
jwaynejones
Novice
 
Posts: 8
Liked: 1 time
Joined: Thu Oct 07, 2010 1:55 pm
Full Name: Jeremy Jones

Re: VSS timeout with Exchange 2010

Veeam Logoby Bunce » Wed Dec 01, 2010 8:30 pm

They wouldn't have been with 2010, and certainly not SP1 which has only been out a short time..
Bunce
Expert
 
Posts: 254
Liked: 8 times
Joined: Fri Sep 18, 2009 9:56 am
Location: Adelaide, Australia
Full Name: Andrew

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Wed Dec 01, 2010 9:02 pm

Actually, I just recalled we are mostly finishing our migration to Exchange 2010... I can check with admins if they have similar issue (since they are using Veeam Backup to protect it as well). Probably not since I am sure I would hear of any issues very fast.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VSS timeout with Exchange 2010

Veeam Logoby jpjetlinx » Wed Dec 01, 2010 10:32 pm

I cam brand new user and still getting the software set up, but I can report that while backing up my Exchange 2010 server I am experiencing the same issue as the original post with the exception that I recieve the error on every attempt. Until the investment in Veeam, I was sucessfully using the built-in Windows Server Backup utility to backup the system. If I am not mistaken, I think the built-in utility uses the native Microsoft Exchange VSS writer to acomplish the backups so I am not sure how it could be related to the native VSS Writer. I really need a solution as this is one of the most critical servers to back up.
jpjetlinx
Lurker
 
Posts: 2
Liked: never
Joined: Wed Dec 01, 2010 9:33 pm
Full Name: Jeremy Parks

Re: VSS timeout with Exchange 2010

Veeam Logoby itcaptain » Thu Dec 02, 2010 1:43 pm

Well I did uninstall the Symantec Backup Exec remote Agent from our Exchange 2010 server and restarted the server. Ran 2 incremental jobs and they were successful without any issues. I will keep everyone posted if there is any other issue related to this.

Thanks Gostev :)
itcaptain
Novice
 
Posts: 3
Liked: never
Joined: Tue Nov 30, 2010 3:22 pm
Full Name: it captain

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Thu Dec 02, 2010 1:54 pm

Glad it worked, thanks for the update!

Also, I have just checked with our admins as promised, it appears that we have migrated 180 users to Exchange 2010 as of today, and have absolutely no issues backing it up with Veeam (VSS enabled, of course).
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot], cthorpe and 10 guests