BSOD PNP_DETECTED_FATAL_ERROR

[Leitner]

Hi,

I'm using a Server with several disks (some are NTFS formated, some are ReFS formated). OS is Server 2019.
I back up the whole machine with Veeam Agent 3.0.2.1170 on an external NTFS drive.

Every time when I want to restore files from this backup with FLR on a Win 10 1903 Enterprise (which also support ReFS 3.4 like server 2019) I got a BSOD "PNP_DETECTED_FATAL_ERROR".

Is there any way, how can I fix this?

I've also opened a support case (#03983208). But I didn't get any response since days.

Best Regards
Patrick

[asdffdsa6131]

hello,
at what point do you get that BSOD?
are you running the exact same versions of vagent on both the server and w10?

about the refs and ntfs, i do not think that matters.
the backup drive is ntfs and both ws2019 and w10 can read ntfs.

as a test, on the ws2019, can you recover files?

[Dima P.]

Hello Patrick,

Any chance you have any antivirus software installed? AV software may block file level restore functionality, so if you have it installed - try to run file level recovery with antivirus being disabled (for testing purposes only). Thank you in advance!

[Mildur]

I have the same problem.
All windows 10 devices can‘t do a FLR from a restore Point which contains reFS Volumes.

I‘m looking forward to veeam V10. With B&R v10, the mounting process should be done on the backup server conponents and not on the client, if I‘m correct.

#edit:
I had a case for that problem last August:
#03699821

Solution was to install a specific windows update

[asdffdsa6131]

someone wrote "All windows 10 devices can‘t do a FLR from a restore Point which contains reFS Volumes."

i respectfully disagree,
on my w10.pro, not w10.workstation, not w10.enterprise running community edition of vagent, thanks veeam for free edition!

the source files of the backup are stored on a refs filesystem.
the veeam backup files, stored on a server, use refs filesystem.
the backup was performed using community edition of vbar, again, thanks veeam for the free edition!
so it is end to end, refs on the source and refs on the dest.

to be clear, my w10.pro agent was able to mount that backup, as a shared folder.
and i was able to recover files.

[asdffdsa6131]

i take back my last post.
to be clear, that backup, of files on refs, was a file backup, not a volume backup.

i better understand that the issue is restore from a backup containg a refs volume.

sorry for my last post, i tried to delete it but could not.

[Mildur]

„i better understand that the issue is restore from a backup containg a refs volume.„

Yes
My Hope is in Veeam V10 and centralized restore point mounting.

[Leitner]

Hi Folks,

Thanks for your answer.

@asdffdsa6131:
At the end of FLR I click on "open". Then the BackupBrowser opens and I can see a Popup with the message "Please wait". After few seconds the computer crash with the BSOD.
Server 2019 and Win 10 using the exact same version of Veeam Agent.
Server 2019 can restore the files with FLR without any issues.
I’m almost sure, that ReFS is responsible for the BSOD.

@Dima P.:
This was my first suspicion.
But disabling AV Software doesn't fix it.

@Mildur:
could you tell me the Windows KB number?

Best Regards
Patrick

[Mildur]

@Leitner
My testing was with two windows builds last august.

1809 --> KB4505658
1903 --> no update available

Windows 10 1909 has the same BSOD problem now. No solution until now.

[Leitner]

Thanks for this information.

Did you get any reasonable answer from veeam about this issue?

Regards
Patrick

[Mildur]

No problem.
Support told me only about the new "mounting process". It should be released with Veeam V10 today/next few days.
I will test the process sometime after upgrading our backup environment.

Veeam Support:

From Veeam side - in internal thread I've mentioned while on phone call - there is a discussion that development team is planning to rewrite Veeam driver that is in charge for that mount operation, which could possibly improve state. But I don't have any ETA, sorry.
So current plan is to check KB4505658 MS part.

[Dima P.]

Hello folks,

Thank you for sharing the updates. We'll check both cases with the QA folks, meanwhile please download the latest version of Veeam Agent for Windows v4 (should become available at the website today) and check if the issue persists while using the FLR with the latest version. Cheers!

[adault]

Hello,

I had the same problem, I will explain what my setup and how I ended up restoring my files.

Workstation: Windows 10 Ent 1909 (Build 18363.657)
reFS.sys driver version: 10.0.18362.449
Two volumes: one NTFS and one ReFS (going to be changing that)
Client version: was 3.0.2, now is 4.0.0.1811
Backup target: Windows Share

When I tried to do a FLR, I get the same blue screen everyone seems to be getting.

Steps I tried:

• disable AV, get a BSOD.
• tried to do a FLR from a fresh install of Windows with same version of client, same BSOD.
• upgraded client to version 4, did same 2 tests again with same outcome.
• Tried on a Windows 7 Pro workstation was able to mount the backup but could not see drive that was formatted reFS.

How I managed to restore files:

• Installed client version 4 on Windows Server 2019 1809 (build 17763.973)
• refs.sys driver version: 10.0.17763.831
• system was able to mount and restore files from reFS formatted volume.

So it seems that the problem might be related to changes in the newer reFS driver.
Hope this can be helpful to others.

[Dima P.]

Thanks for the details Andre!

UPDATE: Based on the discussion with RnD folks the issue is related to ReFS driver version (specifically the latest version of the driver) and since our file level recovery relies on ReFS functionality during recovery we got this problem. For now it looks like the workaround would be, indeed, to perform a file level recovery on 1809 machine with the 10.0.17763.831 ReFS driver version. I'll update this thread once I have more details. Stay tuned.

[Leitner]

@Dima P.
Is Veeam now getting in touch with MS or should we report this MS (last probably won't help us to fix this)?

Regards
Patrick

[Leitner]

Answer from Support at my ticket:

It looks as if the behavior seems quite similar to one known issue found in the FLR VDK driver we rely one - Bug 180011 - WinFLR (Item restore) cause BSOD of Console server where restore is initiated.

It's been reported for Backup & Replication, whereas it seems that our standalone Agent driver may have the same behavior. At the moment we're clarifying if there are any possible workarounds.
I'll let you know those as soon as we get them.

@Dima P.:
Now I'm confused.
You told, that the problem is the windows ReFS driver. Your colleague told me, that this is a Veeam bug.

[Dima P.]

Leitner,

The deal is that during file level recovery ReFS driver of the Operating System is used. Stay tuned, still checking the details with QA folks.

[Leitner]

Fix-it/Workaround:

1) Update VAW to the most recent version: https://www.veeam.com/windows-backup-do ... =downloads

2) Open registry branch:

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam EndPoint Backup

and add key:
Type DWORD
Name UseVHDMount
Value 1

Thanks a lot to Veeam Support!

Regards
Patrick

[Dima P.]

Patrick,

Thank you for the confirmation that the mentioned workaround helps!