Standalone backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
KnB567
Influencer
Posts: 24
Liked: 3 times
Joined: Nov 24, 2020 4:57 am
Full Name: Ray Cockshell
Contact:

Exchange BSOD

Post by KnB567 »

Has anyone come across issues where Exchange server VMs BSOD just after Veeam backup starts?
First come across this with an Exchange 2016/Windows 2012R2 running on Windows 2016 Hyper-V. Ran for years OK then a couple of months ago would reboot every day around backup time. Did not originally associate this with VBR and eventually retired/removed this Exchange server (it was a stand-alone server in a remote site and not really needed any more).
About two weeks ago another Exchange server started similar behaviour. This one was Exchange 2016/Windows 2012R2 running on Nutanix AHV. This one is a critical DAG member, backed up by Veeam Agent for Windows (as there is no native Application Aware processing with Veeam Agent for Nutanix). Now we start to correlate BSOD times to VAW backups - they are consistent. Disabling the VAW backup stops the BSOD.
We were just starting a Hybrid environment but have not migrated any mailboxes yet, which will take months. We had already built a new Exchange 2019 server as our Management server going forward, so we decided to upgrade the (now two) mailbox servers from 2016 to Exchange 2019.
Built a new Exchange 2019 server (same as management server (Exchange 2019/Windows 2019 on Nutanix AHV) ready for new DAG and now the first 2019 server is BSOD as well. Times match Veeam Agent backup starting.
All servers are fully patched and latest CUs.
We now have only one DAG member doing all the work without failing. This is last of original Exchange 2016/Windows 2012R2 servers, also on Nutanix. This one was completely rebuilt after the Hafnium exploits earlier this year, at the same time as we built the first Exchange 2019 server. It is also running VAW to truncate Transaction Logs.
I'm getting a bit nervous. There is definitely some correlation between Veeam and the BSODs, but cannot find it yet. Will lodge case soon. Still on v10 (politics).
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Exchange BSOD

Post by Gostev »

No reason to be guessing here, BSOD memory dump will show the offending process... I can only say that in 90% of cases, it ends up being an antivirus.
KnB567
Influencer
Posts: 24
Liked: 3 times
Joined: Nov 24, 2020 4:57 am
Full Name: Ray Cockshell
Contact:

Re: Exchange BSOD

Post by KnB567 » 1 person likes this post

Thanks Gustav.
Had already examined Memory dumps (gotta love BlueScreenViewer). They all pointed to NTOSKERNEL.EXE.
Reason for asking the Veeam community was that the timing of Veeam backup jobs and BSOD's were very closely related in all cases.
I am now confident that Veeam is not a player here. Exchange HealthMonitoring was triggering reboots.
I was mislead by reported timings of Veeam backup jobs and BSODs: still cannot explain why they overlapped. It actually showed Veeam backups running during some BSODs. One of them must have been reported in GMT, although don't know which one. I know VAN only reports in GMT, but VAW is usually accurate.

Root cause turns out to be a weird combination of:
  • New and heightened Health Monitoring in latest Exchange CUs (including Emergency Mitigation Service),
  • Corrupt Monitoring Mailbox in one server (Exchange complained about this as a User mailbox without a database),
  • Some residual Monitoring mailboxes from retired Exchange servers,
  • Bindings on Nutanix vNICs on Database servers and File Share Witness servers (had to replace NICs on each server and reconfigure IP),
  • All these contributing to failing Exchange server unable to communicate with FSW on non-Exchange VM or to partner Exchange server on original NICs,
  • IPv6 also a possible factor.
Kept peeling the onion, and still don't have an absolute, but its all working again.

Ray
Post Reply

Who is online

Users browsing this forum: ybarrap2003 and 34 guests