CHKDSK During SureBackup Boot

[foggy]

Does the corresponding backup job have AAIP enabled?

[JoshuaPostSAMC]

Yes, this particular VM is in a job with AAIP enabled, although this server does not have SQL running on it.

All of my proxies that this job uses are set to Automatic Transport Mode. This VM did use Hotadd last night, but nearly all of my VM do except for my Exchange servers which are set to use a proxy that is NBD.

[foggy]

Do you see similar behavior on the original VM, if you try to reboot it?

[JoshuaPostSAMC]

In previous instances, no. The original VM reboot successfully. At this point I went ahead and scheduled a chkdsk on next reboot of the original VM to be safe so until it does a reboot and runs that I won't have an accurate test in SureBackup

[foggy]

Don't forget to take a fresh backup after that.

[mcz]

Hi everybody,

my SureBackup jobs are failing the last days and the reason is obvious: My DC is running a checkdisk during startup and therefore the job runs into the timeout. Now you could easily just increase the time in the settings but that's not the question. The question is why does it run checkdisk?? When I restart my DC in the production, no checkdisk runs there - so the filesystem should be ok. The backup itself has healthchecks after every SureBackup run and once a week - I assume the backup is fine.

So something with the filesystem on the vm in the lab seems to be wrong - but why could it ever happen?

Thanks in advance!

[HannesK]

Hello,
application aware backup creates a VSS snapshot. I guess it has something to do with that. I also guess that when you rebooted your production DC, that you did not do authoritative / non-authoritative restore (which happens when you select the roles in SureBackup).

For definite answer, please contact support.

Best regards,
Hannes

[mcz]

Hi Hannes,

investigation is still going on, case # 04326276. When I do a Instand VM recovery or even start a replica manually, everything is fine. As soon as SureBackup starts that vm, checkdisk is run. Looks like the DC preparation is doing something which breaks the filesystem.

[foggy]

Hi Michael, I've merged your thread into an existing one discussing similar issues, please review as it contains a few hints that might be helpful. Thanks!

[mcz]

Hi Alexander,

thanks for the merge. Changing transport mode to nbd (from hotadd) hasn't helped. Source filesystem is clean. There's nothing I could do right now, so I'll wait for the response from support.

[foggy]

Since it has started out of nowhere - does AAIP complete successfully during the last backups?

[mcz]

yeah, sure. That was one of the first things I've checked. It doesn't make sense to me at all why it came all of a sudden.

[nitramd]

Michael,

It sounds like the disk's dirty bit has been set for some reason; it's usually set when a disk is improperly shut down.

Follow this link for strategies: https://superuser.com/questions/469521/ ... every-boot

Maybe experimentation and testing are in order.

Hope this helps and good luck.

[mcz]

Hi foggy,

case # 04326276. Looks like since Windows 10 2004 (and also in the latest october release) the ntfs driver is corrupting the disk. Since veeam has a direct link to microsoft, I hope that there's a conversation going on??

[Gostev]

The direct link we (R&D) have is to the ReFS dev team only. But normally, our support opens cases with the Microsoft Premier Support automatically as soon as the issue is confirmed to be on their side. It's actually very common for our support to open cases with other vendors because so many issues our users run into have nothing to deal with Veeam. And we in R&D simply don't track all these cases, but the engineer on your case will be able to tell you if they already opened a case.

[mcz]

Thanks Anton. The engineer told me that he's waiting for a response from T3 and I have already asked if they talked to Microsoft. Maybe they did but at least it wasn't confirmed to me.

[mcz]

Update: My engineer wrote me the following:

The MS case was opened. Title: "NTFS corruption when mounting a disk to a different Windows version"

So finally the case has reached Microsoft. I'm still surprised that they didn't fix it (or maybe even didn't recognize it???) because the latest Windows 10 version has the same bug, so it exists since may...

The ugly part here is that you're limited when you're running such an affected version. Engineer told me the following:

The data is not safe only during SureBackup DC configuration, or replica Re-Ip (in that case Veeam is mounting replicated VM hard disk to the proxying server), or Windows FLR. In those cases, it would be better to avoid assign Veeam server or proxy server role to the Windows 10 2004 servers. In all other cases everything should be OK (Backup, restore, replica, other OS FLR,etc... ).

I'm sure I don't know every detail of this case (how should I) but it looks like this NTFS bug is quite severe for a properly working installation and so it still surprises me that it takes so long overall...

[helpdeskvsg]

Is this issue resolved somehow? I've just migrated from a Windows Server 2016 Veeam B&R server to Windows Server 2022 and as of that moment every Windows Server 2012R2 DC starts a chkdks when run from SureBackup. SureBackup with a Windows Server 2012R2 server or higher (no DC) just completes successful. I've created a ticket for this.

[mcz]

I've given up and migrated to a higher OS version, that's why I can't tell you anything about it. But I'm not surprised that you still see it...

[PetrM]

Hello,

I believe you would see the same behavior if you rebooted one of the affected servers but anyway I recommend you continue working with our support team to sort out this problem.

Thanks!