Exchange VSS Writers Failing

[igibason]

Hello Everyone,

We are very new to Veeam and just started using it. So far it has not been a good experience as far as Exchange backups go (along with a few other things). Every time our Exchange backup runs the Exchange VSS Writer goes into a Failed state with a Retryable error. I have been restarting services and rebooting mailbox servers for the past week.

We have a support case opned however it is recommend to open another case but with MS to troubleshoot the VSS Writer.

If I run a manual Full Mode Diskshadow from this article (https://www.veeam.com/kb1980) the VSS writer is fine. Only when the Veeam backup runs does the VSS writer error out.

My question is, what have other customers done to solve this problem?

To troubleshoot we have:
Uninstall old backup agents (wishing I hadn't at this point)
Removed a few very old shadow copies
Updated vmware tools
Added more memory to mailbox servers
Enabled logging with diskshadow cmds it appears manually running diskshadow does NOT cause the VSS write to go into an error state. Only when Veeam backups run.

Any insight on what others have done to solve this would be great. In the meantime I do have a case open with MS to assist.

Thanks!
Ian...

[soncscy]

Hi Ian,

I'm curious what the specific error you're getting is from the logging. Exchange backups should have two phases:

1. Prepare a disk shadow to include all volumes (and you're certain you included all the exchange volumes?)
2. End the backup, which triggers exchange to truncate the logs

When you watch what a backup actually does, it's legitimately no different than just a disk shadow aside from the snapshot that occurs at the hypervisor level, so I suppose that it might have something to do with the IO pressure from the snapshot __after__ the initial disk shadow. Snapshot IO impact is intense, and without this element, a simple manual disk shadow might not replicate what you're seeing.

But, I'd be curious on the exact error that happens and what process during the backup fails due to the writer. My experience with clients is that it's datastore performance related, but that's just a guess without knowing the specifics.

[igibason]

Hey Soncscy,

Thanks for the reply. The errors I see in exchange are:

Code: Select all

Event ID:      2140
Task Category: Exchange VSS Writer
Description:
The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::PrepareSnapshot. HResult -2147467259.

Event ID:      2026
Task Category: Exchange VSS Writer
Description:
The Microsoft Exchange Replication service VSS Writer (Instance 7b03ed21-ad8a-4b35-b2e9-f505a4ecbb04) failed with error 80004005 when preparing for snapshot.


Event ID:      2028
Task Category: Exchange VSS Writer
escription:
Microsoft Exchange Replication service VSS Writer instance 7b03ed21-ad8a-4b35-b2e9-f505a4ecbb04 failed when freezing the databases.

In Veeam report from the job I see this error.

VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{6dbbf2b4-33db-4fed-bd25-2c1806b49190}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3].

The backup process does seem pretty straight forward. Thanks for the explanation.

As for the datastore performance I would certainly hope thats not he issue as we are running an all flash Pure array. However I can get our SAN admin to look at performance on these datastores.

Hopefully these errors help. I have a couple calls scheduled tomorrow to troubleshoot this. Most cases it ends up with MS pointing to a Veeam issue and Veeam pointing to a MS issue.

Thanks again,
Ian...

[Dima P.]

Hello Ian,

Can you please share the case ID? Did I get it right that Exchange Server is a virtual machine and your protecting it with Windows agent? Thanks!

[soncscy]

>Event ID: 2140
Task Category: Exchange VSS Writer
Description:
The Microsoft Exchange Replication service VSS Writer encountered an exception in function Microsoft::Exchange::Cluster::ReplicaVssWriter::CReplicaVssWriterInterop::PrepareSnapshot. HResult -2147467259.

Event ID: 2026
Task Category: Exchange VSS Writer
Description:
The Microsoft Exchange Replication service VSS Writer (Instance 7b03ed21-ad8a-4b35-b2e9-f505a4ecbb04) failed with error 80004005 when preparing for snapshot.


Event ID: 2028
Task Category: Exchange VSS Writer
escription:
Microsoft Exchange Replication service VSS Writer instance 7b03ed21-ad8a-4b35-b2e9-f505a4ecbb04 failed when freezing the databases.

Those are very interesting for me.

I'm sorry, I have to ask again, but during manual testing of the disk shadow, you included all volumes and you manually included the Exchange writer?

https://www.veeam.com/kb1980?ad=in-text-link#full

I actually would expect you'd get the same errors even without the snapshot being a component.

But those errors should be simple enough to simply ask Microsoft "Why are we getting these?"

[igibason]

Hi Dima,

Thanks for joining the conversion. The case ID is 04657426. The techs have been very helpful. I am just looking around to see if other customers experienced this and what they did the fix it.

Correct, the server we are backing up is a VM using VMware. The job type is a VMware Backup so I don't think its using the Windows Agent? Kind of new to Veeam

@soncscy no problem, during the manual test of diskshadow I didnt see any errors in the Output log. We have a C, D, and E, drive for OS, Mailbox Data, and Logs. The cmds I used were:


diskshadow /l c:\admin\vssoutput.txt
set verbose on
set context volatile

add volume C:
add volume D:
add volume E:


begin backup
create

end backup

list shadows


I was also expecting them to show errors. I have a call with MS today so hopefully they will be able to point us in the right direction.

Thanks again for the replies!
Ian...

[Regnor]

Hi Ian,
it's probably best to work with Veeam/MS support on this case but here are a few ideas:

*Do you have a DAG? If so have you separated them in your backup jobs?
*How long does the snapshot creation and deletion process take?
*During backup & snapshot processing is/are the exchange server(s) reachable via network?
*Is VSS support enabled in VMware Tools? I remember some backup agents/software uninstall VMware Tools VSS

[igibason]

I agree...still waiting on the call from MS.

case id 04657426

*Do you have a DAG? If so have you separated them in your backup jobs?
Yes we do have a DAG. Four mailbox server in a single DAG and three client access servers. All 7 servers are in the same exchange backup job.

*How long does the snapshot creation and deletion process take?
From quickly looking the Veeam backup job started around 9:09PM and does not have an end time. But the server Detail list in the report shows end time of 10:19.

Looking in vSphere on just one mbx server I have issues with I see a Create VM Snapshot entry at 9:17 then a Remove VM Snapshot at 10:15.

*During backup & snapshot processing is/are the exchange server(s) reachable via network?
Good question, I am assuming they are but I have not done a stream ping to them while the jobs was running. I also don't see any network related errors. But I can dig a little deeper.

*Is VSS support enabled in VMware Tools? I remember some backup agents/software uninstall VMware Tools VSS
I am assuming it is enabled on VMware Tools as I do not see any setting for it. I will need to look more into this as well.

Thanks again for your posts. I will post back here what I end up doing to fix this.

Regards,
Ian...

[Regnor]

I would try to split up the exchange job. In a DAG it can cause problems (and VSS errors like yours) if you backup/snapshot multiple servers at the same time; at least the database/mailbox servers.
You could also try to backup only a single server for the moment and check if the errors persist.

Regarding the snapshots; it's more important how long the creation and removal steps take in total. If it takes much time and you're servers are under heavy load you could experience stuns during snapshot removal.

There's a very extensive blog post on exchange backups from Andreas Neufert from Veeam; you can check it for additional tips/info:
https://andyandthevms.com/exchange-dag- ... plication/

[igibason]

Hi Regnor,

Thanks for the input. I think you are on to something. We do have our 4 MBX servers in a single DAG. In the backup job they are all listed in the Virtual Machines section as tagged. I just removed them as tagged and manually added them in the Virtual Machines section by the VM name. When the job ran I expecting it the backup each VM one at a time in the order they were listed. But it seemed to just run all at the same time and failed. Only one succeeded with no VSS errors.

We actually just had a Health Check with an Veeam Engineer who was really good. So I think we are pointed in the right direction!

While we wait for results I am going to run the job with one server at a time.

Thanks again!
Ian...

[Regnor]

But it seemed to just run all at the same time and failed.

Veeam will run as many tasks(disks) in parallel as you define in the proxy and repository settings. So as long as you have all exchange servers in one job chances are high that they will be run in parallel. Try to create a separate job per exchange server and chain those; that way they won't be processed in parallel.

[igibason]

For now we ended up creating four backup jobs for each of our mailbox servers. The jobs are set to not run until the previous one finishes using the 'After this job' Scheduling setting.

So far backups have been working great. Would like to run them from a single job but this will have to do.

Waiting for results from our Veeam Health Check to see if the engineer can suggest anything else.

[Regnor]

I'm glad you were able to solve your problem.
If you want to create a single job then add your exchange servers to an existing job with other VMs. Then setup the VM processing order in that way that no Exchange servers are processed in parallel.

[bhagen]

I have occasionally experiencee this error from random backup jobs over the last several years. The "vss" mitigations that are published on the veeam and microsoft sites (and other places online) never seem to fix the issue; the only thing I've found that fixes it is to reboot the VM that I'm trying to backup.

I'm not sure if this is worth a support ticket or not; maybe somebody here has seen this and can tell me what is actually going on and how to avoid it?

3/8/2022 3:23:53 AM :: VSS: Backup job failed.
Cannot notify writers about the 'BACKUP FINISH' event.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{28536bd6-7845-4f80-b246-97c8cbd2f969}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3].

[PetrM]

Hello,

I moved your question into the existing topic, please review the suggestions above. Also, I recommend to check Application event log on the Exchange server and open a support case as it is required for any technical issue. Please don't forget to share a support case ID with us. You may also attach vsstrace to the support case, the instructions are outlined in this KB. One more thing to check is to try to create a shadow copy manually, please review this KB for more details.

Thanks!

[bhagen]

Thanks @petrm, I'll look thru this post. Of note is that this issue doesn't only affect my exchange servers, but sql servers and simple file servers on occasion. But hopefully the concepts will apply to all.

[jcwrks]

I am surprised nobody mentioned that you should create a Veeam protected group plus the appropriate backup job for backing up an Exchange DAG.

https://www.veeam.com/kb2463

Starting with Veeam Backup & Replication 9.5 Update 4, Microsoft Exchange database availability group (DAG) nodes are now automatically processed sequentially. Hence, it is possible to use Veeam Agent for Microsoft Windows to process Microsoft Exchange Database Availability Groups (DAGs).

The procedure of adding a DAG to a Veeam Agent backup job differs depending on the type of the DAG that you want to process:

For a regular DAG, the backup job configuration procedure is the same as for any failover cluster mentioned above, so all the steps above are relevant
For an IP Less DAG (a DAG without an Administrative Access Point), the backup job configuration procedure is similar to the same method for standalone servers. To process an IP Less DAG, you must create a Veeam Agent backup job for servers in the Veeam backup console and add all nodes of the IP Less DAG (the steps below show it in detail) to this job.

[PetrM]

Hello and Welcome to Veeam R&D Forums!

Yes, it's a valid note for backup based on Veeam Agent but you can also process DAG Exchange in vSphere, you may refer to this KB.

Also, I don't think that the errors mentioned in this topic depend on the selected backup method whether it is Veeam Agent or vSphere VM processing, the errors come from Exchange VSS writer which works inside the guest operating system.

Thanks!