Comprehensive data protection for all workloads
Post Reply
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Replication Exchange Server - VSS writer error event logs

Post by james411 »

Scenario: ESXi 5.5 U2 host with single Windows 2012 R2 VM running Exchange 2013 CU9. I've just started running replication jobs of this VM and see the following errors in the event log:

Code: Select all

Log Name:      Application
Source:        MSExchangeRepl
Date:          10/26/2015 4:48:49 AM
Event ID:      2038
Task Category: Exchange VSS Writer
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      name.domain.local
Description:
Microsoft Exchange VSS Writer backup failed. No log files were truncated. Instance c9e49f8f-2754-4364-bb4a-c0545eac904e. Database a3623468-fb24-466f-99fb-cef7a273ead4.
followed by:

Code: Select all

Log Name:      Application
Source:        MSExchangeRepl
Date:          10/26/2015 4:48:49 AM
Event ID:      2034
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      name.domain.local
Description:
The Microsoft Exchange Replication service VSS Writer (Instance c9e49f8f-2754-4364-bb4a-c0545eac904e) failed with error FFFFFFFC when processing the backup completion event.
Interestingly enough, I do not see these errors after a Veeam Backup Job. The Veeam Replication job reports Success and no errors. I'm wondering if maybe this is because I currently have circular logging enabled? Is this expected or is there something wrong with my replication job?
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by foggy »

Do you have application-aware image processing enabled in the job settings?
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by james411 »

foggy wrote:Do you have application-aware image processing enabled in the job settings?
Oops, sorry left out that detail. Yes, AAIP is enabled on this replication job.

Edit: also forgot I am on Veeam V8 U3
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by foggy »

What application-aware image processing settings do you have for this job? If you're using circular logging, there's no need for backup software to truncate the logs.
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by james411 »

foggy wrote:What application-aware image processing settings do you have for this job? If you're using circular logging, there's no need for backup software to truncate the logs.
It is the default settings for replication job: VSS: Require Success, Transaction logs: Copy only
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by foggy »

Have you checked Exchange writer state (vssadmin list writers)?
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by james411 »

foggy wrote:Have you checked Exchange writer state (vssadmin list writers)?
Microsoft Exchange Writer currently shows no error. Now, I have not run a replication job since I posted but have continued to run my nightly backup jobs (which as I mentioned don't seem to trigger these log events).
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by foggy »

Most likely this comes down to the Microsoft Exchange Writer being unstable during replication job, so I'd contact MS to verify the reason. Is this behavior consistent (I mean do you see it every time replication job runs?)?
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by james411 » 1 person likes this post

foggy wrote:Most likely this comes down to the Microsoft Exchange Writer being unstable during replication job, so I'd contact MS to verify the reason. Is this behavior consistent (I mean do you see it every time replication job runs?)?
Follow-up for those who may be encountering similar situation: I opened a case with Veeam support (Case # 01093428) and the engineer came to the following conclusion: during a normal backup that truncates the logs, Veeam communicates to the VSS writer that the VSS backup succeeded which then allows for log truncation. In the case of Copy Only, to avoid log truncation, Veeam sends a VSS backup failed message to the VSS writer so it will not attempt to truncate the logs. Essentially, it's a false error and is just a way for Veeam to get around truncating the logs.

I've currently switched my replication job to Process the Logs and since I have circular logging enabled, nothing happens anyway (there is an informational message in the Event Log that no logs can be truncated). This avoids my event log being filled with these errors. Only problem is, at some point I intended to disable circular logging (had it enabled while we migrated from Exchange 2007). In that case, if I switch the job back to Copy Only, I'll get those errors again and since I have the job running so often, it really clutters the event log. Veeam engineer recommended leaving process the logs on for the replication job and Copy Only for the backup job to avoid have the event log filled with these errors.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by foggy »

Confirmed, those messages in Event Log are expected with the current implementation of Copy Only. However you should observe them during backup jobs as well, since the same VSS mechanism is used in both jobs.
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by james411 »

foggy wrote:Confirmed, those messages in Event Log are expected with the current implementation of Copy Only. However you should observe them during backup jobs as well, since the same VSS mechanism is used in both jobs.
Correct, the engineer just suggested this since the backup job runs once a day so I'll only see the message once as opposed to every 15 minutes :)
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by foggy »

Ah, ok. I was confused by that since initially you've stated the opposite.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

[MERGED] : Need for Perform Copy Only to Cause a VSS Error

Post by mkaec »

I set Guest Processing > Applications > [VM] Edit > Transaction logs to "Perform copy only" in some jobs and started getting emails from our server monitor because the Hyper-V host started logging errors during each backup. The backups complete fine in Veeam, but the host is unhappy. The error is:

Log: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 10172
Description: VSS writers inside virtual machine '<VM>' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: Catastrophic failure (0x8000FFFF).

I opened a support case about this (01239526) and was eventually told that this is deliberate behavior on the part of Veeam when "Perform copy only" is selected. It was explained that Veeam completes the backup, then tells VSS there was a failure in order to avoid truncating Exchange logs.

It was explained that responding with success would cause the Exchange logs to be truncated. So Veeam pretends there was a failure in order to preserve them. It's a bit puzzling, however, because we found in the Veeam log that it was passing the "copy only" flag to VSS. For this to be required, the Exchange VSS writer would have to not properly support the copy only flag. But I know that when backing up Exchange via an agent-based backup application, the copy only option works well.

I was wondering if you could point me in the direction of some Microsoft documentation, a blog post, a forum thread, etc. that documents that this defect exists in the Exchange VSS writer. I need to explain to my colleagues why a "catastrophic failure" is actually quite normal and could use some backing documentation.
veremin
Product Manager
Posts: 20413
Liked: 2301 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by veremin »

I'm not sure whether such documentation exists, so you can take either explanation given within your support ticket or the one provided to James above. Thanks.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Replication Exchange Server - VSS writer error event log

Post by mkaec »

I posted here because I found the explanation lacking.

Support Rep: It does that on purpose to prevent Exchange logs from being truncated.
Me: But it passes the VS_BS_COPY flag. Other backup applications can do a copy only backup of Exchange without generating these errors. Why is this needed?
Support Rep: I don't know. That's just the way it works. Try the forums.

Having thought about it for a while, maybe the behavior is like that in order allow a backup job to prevent truncation of logs involving unknown applications that may not support the VS_BS_COPY flag. If that were the case, I'd really like to see an "Exchange" tab added to the dialog so that log truncation can be turned off for Exchange specifically like can be done for SQL.

As with James, I just desire to eliminate superfluous "catastrophic failure" events from my event logs. These show up in the "Administrative Events" custom view in the Event Viewer and can be picked up by server monitor tools.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Semrush [Bot] and 72 guests