BSOD on client host during FLR

VMware specific discussions

BSOD on client host during FLR

Veeam Logoby ChrisL » Mon Feb 27, 2017 1:27 am

I have a case open with support on this already (Case # 02079640) but thought I'd mention it here in case anyone has come across this before.

For a particular backup chain, performing an FLR on a remote client causes the remote computer running the client to BSOD during the mounting process. Other backup chains can mount and FLR OK, so it's not a general issue. The same affected backup chain can be mounted as an Instant Recovery job, and boots OK (renamed and disconnected from the network) and I've been able to recover what I need from it, so the backup itself seems to be OK. Running the Veeam.Backup.Validator agains the backup job also succeeds (eventually! it's about 1.5TB to validate) so I'm guessing it's something about the mounted volume in the C:\VeeamFLR folder that the host doesn't like that's causing the BSOD.

I've seen two different BSOD's, having tried it a couple of times. Initially it was NONPAGED_AREA STOP Error 0x05, but the more recent ones have been BAD_POOL_HEADER STOP Error 0x19. At the time of the BSOD, the remote computer running the client reports in the Event Log an 'ntfs error Event ID 55: The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume.' though I'm currently reading this as a red herring, and not taking it to literally mean that it's actually corrupted. It's obviously not happy about something in the mounted volume though.

The VM in question has recently had one of its smaller disks increased in size (300GB -> 350GB) which all went to plan and something we've done many times before, but this feels like it may be more than coincidence. To test this however, I have excluded this resized disk from the backup and re-run the job but it's still doing the same thing.

We have also recently upgraded from vSphere v5.5 to v6.0, which may also be related, but I can't see how this would affect only one backup job whilst others seem to mount and do an FLR job just fine..
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

Re: BSOD on client host during FLR

Veeam Logoby foggy » Mon Feb 27, 2017 12:33 pm

Hi Chri, we're aware of at least one occurrence of remote client crash during FLR (and that issue will be addressed in the future Update 2), however, the fact that you're seeing it on a particular backup chain only looks strange. Please continue looking into this with your engineer.
foggy
Veeam Software
 
Posts: 15089
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: BSOD on client host during FLR

Veeam Logoby Gostev » Mon Feb 27, 2017 7:16 pm

Among all BSODs during FLR that we've seen in support, 99% were caused by antiviruses crashing the OS, so first thing to try is disable one and see if it helps. May be not your case, but I figured I'd put it here for the future readers. These sort of issues are actually quite easy to troubleshoot, because the crash dump always includes the offending process name. Thanks!
Gostev
Veeam Software
 
Posts: 21512
Liked: 2381 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: BSOD on client host during FLR

Veeam Logoby ChrisL » Mon Feb 27, 2017 7:49 pm

Thanks Gostev (and actually, thanks too to Igor in support who also suggested the same thing) looks like this is indeed down to antivirus. Just tried disabling it on the remote client host and it's mounted the FLR session as usual. Tried again immediately after with the AV back on and sure enough, BSOD.

Presumably I should simply pop in an exception in the AV for C:\VeeamFLR for a more permanent solution..?
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

Re: BSOD on client host during FLR

Veeam Logoby ChrisL » Mon Feb 27, 2017 8:51 pm 1 person likes this post

So I've tried adding the mount path as an exception but still getting a BSOD. Disabling the AV entirely works though. Any suggestions what else needs to be excluded from the AV scanning?
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

Re: BSOD on client host during FLR

Veeam Logoby Mike Resseler » Tue Feb 28, 2017 7:51 am

Chris,

This is our general KB for antivirus exclusions: https://www.veeam.com/kb1999
Did you exclude C:\VeeamFLR\* (because I have seen AV where excluding VeeamFLR only is not enough because it will still look at the other files

But lately I also saw that you need to exclude the Veeam process / service as that gets caught by AV also.

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3346
Liked: 380 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: BSOD on client host during FLR

Veeam Logoby ChrisL » Tue Feb 28, 2017 2:06 pm

That's great, thanks Mike I'll give that a go. I wasn't including the main Veeam process, so that may be it.

Curious why it would only affect a certain backup chain or chains though. Most of ours a certainly unaffected, even though the AV (Sophos Anti-Virus) is up and running. Must be something about this particular backup chain/mount that's upsetting the AV enough to throw a BSOD.
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 59 guests