Replication Exchange Server - VSS writer error event logs

[james411]

Scenario: ESXi 5.5 U2 host with single Windows 2012 R2 VM running Exchange 2013 CU9. I've just started running replication jobs of this VM and see the following errors in the event log:

Code: Select all

Log Name:      Application
Source:        MSExchangeRepl
Date:          10/26/2015 4:48:49 AM
Event ID:      2038
Task Category: Exchange VSS Writer
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      name.domain.local
Description:
Microsoft Exchange VSS Writer backup failed. No log files were truncated. Instance c9e49f8f-2754-4364-bb4a-c0545eac904e. Database a3623468-fb24-466f-99fb-cef7a273ead4.

followed by:

Code: Select all

Log Name:      Application
Source:        MSExchangeRepl
Date:          10/26/2015 4:48:49 AM
Event ID:      2034
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      name.domain.local
Description:
The Microsoft Exchange Replication service VSS Writer (Instance c9e49f8f-2754-4364-bb4a-c0545eac904e) failed with error FFFFFFFC when processing the backup completion event.

Interestingly enough, I do not see these errors after a Veeam Backup Job. The Veeam Replication job reports Success and no errors. I'm wondering if maybe this is because I currently have circular logging enabled? Is this expected or is there something wrong with my replication job?

[foggy]

Do you have application-aware image processing enabled in the job settings?

[james411]

Do you have application-aware image processing enabled in the job settings?

Oops, sorry left out that detail. Yes, AAIP is enabled on this replication job.

Edit: also forgot I am on Veeam V8 U3

[foggy]

What application-aware image processing settings do you have for this job? If you're using circular logging, there's no need for backup software to truncate the logs.

[james411]

What application-aware image processing settings do you have for this job? If you're using circular logging, there's no need for backup software to truncate the logs.

It is the default settings for replication job: VSS: Require Success, Transaction logs: Copy only

[foggy]

Have you checked Exchange writer state (vssadmin list writers)?

[james411]

Have you checked Exchange writer state (vssadmin list writers)?

Microsoft Exchange Writer currently shows no error. Now, I have not run a replication job since I posted but have continued to run my nightly backup jobs (which as I mentioned don't seem to trigger these log events).

[foggy]

Most likely this comes down to the Microsoft Exchange Writer being unstable during replication job, so I'd contact MS to verify the reason. Is this behavior consistent (I mean do you see it every time replication job runs?)?

[james411]

Most likely this comes down to the Microsoft Exchange Writer being unstable during replication job, so I'd contact MS to verify the reason. Is this behavior consistent (I mean do you see it every time replication job runs?)?

Follow-up for those who may be encountering similar situation: I opened a case with Veeam support (Case # 01093428) and the engineer came to the following conclusion: during a normal backup that truncates the logs, Veeam communicates to the VSS writer that the VSS backup succeeded which then allows for log truncation. In the case of Copy Only, to avoid log truncation, Veeam sends a VSS backup failed message to the VSS writer so it will not attempt to truncate the logs. Essentially, it's a false error and is just a way for Veeam to get around truncating the logs.

I've currently switched my replication job to Process the Logs and since I have circular logging enabled, nothing happens anyway (there is an informational message in the Event Log that no logs can be truncated). This avoids my event log being filled with these errors. Only problem is, at some point I intended to disable circular logging (had it enabled while we migrated from Exchange 2007). In that case, if I switch the job back to Copy Only, I'll get those errors again and since I have the job running so often, it really clutters the event log. Veeam engineer recommended leaving process the logs on for the replication job and Copy Only for the backup job to avoid have the event log filled with these errors.

[foggy]

Confirmed, those messages in Event Log are expected with the current implementation of Copy Only. However you should observe them during backup jobs as well, since the same VSS mechanism is used in both jobs.

[james411]

Confirmed, those messages in Event Log are expected with the current implementation of Copy Only. However you should observe them during backup jobs as well, since the same VSS mechanism is used in both jobs.

Correct, the engineer just suggested this since the backup job runs once a day so I'll only see the message once as opposed to every 15 minutes

[foggy]

Ah, ok. I was confused by that since initially you've stated the opposite.

[mkaec]

I set Guest Processing > Applications > [VM] Edit > Transaction logs to "Perform copy only" in some jobs and started getting emails from our server monitor because the Hyper-V host started logging errors during each backup. The backups complete fine in Veeam, but the host is unhappy. The error is:

Log: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Event ID: 10172
Description: VSS writers inside virtual machine '<VM>' failed to perform BackupComplete to its shadow copy (VSS snapshot) set: Catastrophic failure (0x8000FFFF).

I opened a support case about this (01239526) and was eventually told that this is deliberate behavior on the part of Veeam when "Perform copy only" is selected. It was explained that Veeam completes the backup, then tells VSS there was a failure in order to avoid truncating Exchange logs.

It was explained that responding with success would cause the Exchange logs to be truncated. So Veeam pretends there was a failure in order to preserve them. It's a bit puzzling, however, because we found in the Veeam log that it was passing the "copy only" flag to VSS. For this to be required, the Exchange VSS writer would have to not properly support the copy only flag. But I know that when backing up Exchange via an agent-based backup application, the copy only option works well.

I was wondering if you could point me in the direction of some Microsoft documentation, a blog post, a forum thread, etc. that documents that this defect exists in the Exchange VSS writer. I need to explain to my colleagues why a "catastrophic failure" is actually quite normal and could use some backing documentation.

[veremin]

I'm not sure whether such documentation exists, so you can take either explanation given within your support ticket or the one provided to James above. Thanks.

[mkaec]

I posted here because I found the explanation lacking.

Support Rep: It does that on purpose to prevent Exchange logs from being truncated.
Me: But it passes the VS_BS_COPY flag. Other backup applications can do a copy only backup of Exchange without generating these errors. Why is this needed?
Support Rep: I don't know. That's just the way it works. Try the forums.

Having thought about it for a while, maybe the behavior is like that in order allow a backup job to prevent truncation of logs involving unknown applications that may not support the VS_BS_COPY flag. If that were the case, I'd really like to see an "Exchange" tab added to the dialog so that log truncation can be turned off for Exchange specifically like can be done for SQL.

As with James, I just desire to eliminate superfluous "catastrophic failure" events from my event logs. These show up in the "Administrative Events" custom view in the Event Viewer and can be picked up by server monitor tools.