Host-based backup of VMware vSphere VMs.
Post Reply
ChrisL
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 21, 2011 12:04 pm
Full Name: Chris Leader
Contact:

BSOD on client host during FLR

Post by ChrisL »

I have a case open with support on this already (Case # 02079640) but thought I'd mention it here in case anyone has come across this before.

For a particular backup chain, performing an FLR on a remote client causes the remote computer running the client to BSOD during the mounting process. Other backup chains can mount and FLR OK, so it's not a general issue. The same affected backup chain can be mounted as an Instant Recovery job, and boots OK (renamed and disconnected from the network) and I've been able to recover what I need from it, so the backup itself seems to be OK. Running the Veeam.Backup.Validator agains the backup job also succeeds (eventually! it's about 1.5TB to validate) so I'm guessing it's something about the mounted volume in the C:\VeeamFLR folder that the host doesn't like that's causing the BSOD.

I've seen two different BSOD's, having tried it a couple of times. Initially it was NONPAGED_AREA STOP Error 0x05, but the more recent ones have been BAD_POOL_HEADER STOP Error 0x19. At the time of the BSOD, the remote computer running the client reports in the Event Log an 'ntfs error Event ID 55: The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume.' though I'm currently reading this as a red herring, and not taking it to literally mean that it's actually corrupted. It's obviously not happy about something in the mounted volume though.

The VM in question has recently had one of its smaller disks increased in size (300GB -> 350GB) which all went to plan and something we've done many times before, but this feels like it may be more than coincidence. To test this however, I have excluded this resized disk from the backup and re-run the job but it's still doing the same thing.

We have also recently upgraded from vSphere v5.5 to v6.0, which may also be related, but I can't see how this would affect only one backup job whilst others seem to mount and do an FLR job just fine..
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: BSOD on client host during FLR

Post by foggy »

Hi Chri, we're aware of at least one occurrence of remote client crash during FLR (and that issue will be addressed in the future Update 2), however, the fact that you're seeing it on a particular backup chain only looks strange. Please continue looking into this with your engineer.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BSOD on client host during FLR

Post by Gostev »

Among all BSODs during FLR that we've seen in support, 99% were caused by antiviruses crashing the OS, so first thing to try is disable one and see if it helps. May be not your case, but I figured I'd put it here for the future readers. These sort of issues are actually quite easy to troubleshoot, because the crash dump always includes the offending process name. Thanks!
ChrisL
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 21, 2011 12:04 pm
Full Name: Chris Leader
Contact:

Re: BSOD on client host during FLR

Post by ChrisL »

Thanks Gostev (and actually, thanks too to Igor in support who also suggested the same thing) looks like this is indeed down to antivirus. Just tried disabling it on the remote client host and it's mounted the FLR session as usual. Tried again immediately after with the AV back on and sure enough, BSOD.

Presumably I should simply pop in an exception in the AV for C:\VeeamFLR for a more permanent solution..?
ChrisL
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 21, 2011 12:04 pm
Full Name: Chris Leader
Contact:

Re: BSOD on client host during FLR

Post by ChrisL » 1 person likes this post

So I've tried adding the mount path as an exception but still getting a BSOD. Disabling the AV entirely works though. Any suggestions what else needs to be excluded from the AV scanning?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: BSOD on client host during FLR

Post by Mike Resseler »

Chris,

This is our general KB for antivirus exclusions: https://www.veeam.com/kb1999
Did you exclude C:\VeeamFLR\* (because I have seen AV where excluding VeeamFLR only is not enough because it will still look at the other files

But lately I also saw that you need to exclude the Veeam process / service as that gets caught by AV also.

Thanks
Mike
ChrisL
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 21, 2011 12:04 pm
Full Name: Chris Leader
Contact:

Re: BSOD on client host during FLR

Post by ChrisL »

That's great, thanks Mike I'll give that a go. I wasn't including the main Veeam process, so that may be it.

Curious why it would only affect a certain backup chain or chains though. Most of ours a certainly unaffected, even though the AV (Sophos Anti-Virus) is up and running. Must be something about this particular backup chain/mount that's upsetting the AV enough to throw a BSOD.
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: BSOD on client host during FLR

Post by Mildur »

Sorry for reposting in this old thread.
Have a similar problem now with Veeam 9.5 U4b.

Veeam FLR for "VM's with backuped reFS Volumes" getting everytime an BSOD.
As soon as the vbk/vib is getting mountet to the windows 10 client computer.

BSOD Message: PNP_DETECTED_FATAL_ERROR

Case Number: 03699821
Product Management Analyst @ Veeam Software
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BSOD on client host during FLR

Post by Gostev »

Windows 10 does not support ReFS (except one special edition that you're unlikely to have), so you cannot mount backups there.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 76 guests