BSOD on client host during FLR

[ChrisL]

I have a case open with support on this already (Case # 02079640) but thought I'd mention it here in case anyone has come across this before.

For a particular backup chain, performing an FLR on a remote client causes the remote computer running the client to BSOD during the mounting process. Other backup chains can mount and FLR OK, so it's not a general issue. The same affected backup chain can be mounted as an Instant Recovery job, and boots OK (renamed and disconnected from the network) and I've been able to recover what I need from it, so the backup itself seems to be OK. Running the Veeam.Backup.Validator agains the backup job also succeeds (eventually! it's about 1.5TB to validate) so I'm guessing it's something about the mounted volume in the C:\VeeamFLR folder that the host doesn't like that's causing the BSOD.

I've seen two different BSOD's, having tried it a couple of times. Initially it was NONPAGED_AREA STOP Error 0x05, but the more recent ones have been BAD_POOL_HEADER STOP Error 0x19. At the time of the BSOD, the remote computer running the client reports in the Event Log an 'ntfs error Event ID 55: The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume.' though I'm currently reading this as a red herring, and not taking it to literally mean that it's actually corrupted. It's obviously not happy about something in the mounted volume though.

The VM in question has recently had one of its smaller disks increased in size (300GB -> 350GB) which all went to plan and something we've done many times before, but this feels like it may be more than coincidence. To test this however, I have excluded this resized disk from the backup and re-run the job but it's still doing the same thing.

We have also recently upgraded from vSphere v5.5 to v6.0, which may also be related, but I can't see how this would affect only one backup job whilst others seem to mount and do an FLR job just fine..

[foggy]

Hi Chri, we're aware of at least one occurrence of remote client crash during FLR (and that issue will be addressed in the future Update 2), however, the fact that you're seeing it on a particular backup chain only looks strange. Please continue looking into this with your engineer.

[Gostev]

Among all BSODs during FLR that we've seen in support, 99% were caused by antiviruses crashing the OS, so first thing to try is disable one and see if it helps. May be not your case, but I figured I'd put it here for the future readers. These sort of issues are actually quite easy to troubleshoot, because the crash dump always includes the offending process name. Thanks!

[ChrisL]

Thanks Gostev (and actually, thanks too to Igor in support who also suggested the same thing) looks like this is indeed down to antivirus. Just tried disabling it on the remote client host and it's mounted the FLR session as usual. Tried again immediately after with the AV back on and sure enough, BSOD.

Presumably I should simply pop in an exception in the AV for C:\VeeamFLR for a more permanent solution..?

[ChrisL]

So I've tried adding the mount path as an exception but still getting a BSOD. Disabling the AV entirely works though. Any suggestions what else needs to be excluded from the AV scanning?

[Mike Resseler]

Chris,

This is our general KB for antivirus exclusions: https://www.veeam.com/kb1999
Did you exclude C:\VeeamFLR\* (because I have seen AV where excluding VeeamFLR only is not enough because it will still look at the other files

But lately I also saw that you need to exclude the Veeam process / service as that gets caught by AV also.

Thanks
Mike

[ChrisL]

That's great, thanks Mike I'll give that a go. I wasn't including the main Veeam process, so that may be it.

Curious why it would only affect a certain backup chain or chains though. Most of ours a certainly unaffected, even though the AV (Sophos Anti-Virus) is up and running. Must be something about this particular backup chain/mount that's upsetting the AV enough to throw a BSOD.

[Mildur]

Sorry for reposting in this old thread.
Have a similar problem now with Veeam 9.5 U4b.

Veeam FLR for "VM's with backuped reFS Volumes" getting everytime an BSOD.
As soon as the vbk/vib is getting mountet to the windows 10 client computer.

BSOD Message: PNP_DETECTED_FATAL_ERROR

Case Number: 03699821

[Gostev]

Windows 10 does not support ReFS (except one special edition that you're unlikely to have), so you cannot mount backups there.