Comprehensive data protection for all workloads
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Check Disk During Boot

Post by foggy »

Does the corresponding backup job have AAIP enabled?
JoshuaPostSAMC
Expert
Posts: 124
Liked: 22 times
Joined: Jul 30, 2015 7:32 pm
Contact:

Re: Check Disk During Boot

Post by JoshuaPostSAMC »

Yes, this particular VM is in a job with AAIP enabled, although this server does not have SQL running on it.

All of my proxies that this job uses are set to Automatic Transport Mode. This VM did use Hotadd last night, but nearly all of my VM do except for my Exchange servers which are set to use a proxy that is NBD.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Check Disk During Boot

Post by foggy »

Do you see similar behavior on the original VM, if you try to reboot it?
JoshuaPostSAMC
Expert
Posts: 124
Liked: 22 times
Joined: Jul 30, 2015 7:32 pm
Contact:

Re: Check Disk During Boot

Post by JoshuaPostSAMC »

In previous instances, no. The original VM reboot successfully. At this point I went ahead and scheduled a chkdsk on next reboot of the original VM to be safe so until it does a reboot and runs that I won't have an accurate test in SureBackup
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Check Disk During Boot

Post by foggy »

Don't forget to take a fresh backup after that.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

[MERGED] DC is running checkdisk in SureBackup Lab

Post by mcz »

Hi everybody,

my SureBackup jobs are failing the last days and the reason is obvious: My DC is running a checkdisk during startup and therefore the job runs into the timeout. Now you could easily just increase the time in the settings but that's not the question. The question is why does it run checkdisk?? When I restart my DC in the production, no checkdisk runs there - so the filesystem should be ok. The backup itself has healthchecks after every SureBackup run and once a week - I assume the backup is fine.

So something with the filesystem on the vm in the lab seems to be wrong - but why could it ever happen?

Thanks in advance!
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: DC is running checkdisk in SureBackup Lab

Post by HannesK »

Hello,
application aware backup creates a VSS snapshot. I guess it has something to do with that. I also guess that when you rebooted your production DC, that you did not do authoritative / non-authoritative restore (which happens when you select the roles in SureBackup).

For definite answer, please contact support.

Best regards,
Hannes
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: DC is running checkdisk in SureBackup Lab

Post by mcz »

Hi Hannes,

investigation is still going on, case # 04326276. When I do a Instand VM recovery or even start a replica manually, everything is fine. As soon as SureBackup starts that vm, checkdisk is run. Looks like the DC preparation is doing something which breaks the filesystem.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CHKDSK During SureBackup Boot

Post by foggy »

Hi Michael, I've merged your thread into an existing one discussing similar issues, please review as it contains a few hints that might be helpful. Thanks!
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: CHKDSK During SureBackup Boot

Post by mcz »

Hi Alexander,

thanks for the merge. Changing transport mode to nbd (from hotadd) hasn't helped. Source filesystem is clean. There's nothing I could do right now, so I'll wait for the response from support.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CHKDSK During SureBackup Boot

Post by foggy »

Since it has started out of nowhere - does AAIP complete successfully during the last backups?
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: CHKDSK During SureBackup Boot

Post by mcz »

yeah, sure. That was one of the first things I've checked. It doesn't make sense to me at all why it came all of a sudden.
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: CHKDSK During SureBackup Boot

Post by nitramd »

Michael,

It sounds like the disk's dirty bit has been set for some reason; it's usually set when a disk is improperly shut down.

Follow this link for strategies: https://superuser.com/questions/469521/ ... every-boot

Maybe experimentation and testing are in order.

Hope this helps and good luck.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: CHKDSK During SureBackup Boot

Post by mcz »

Hi foggy,

case # 04326276. Looks like since Windows 10 2004 (and also in the latest october release) the ntfs driver is corrupting the disk. Since veeam has a direct link to microsoft, I hope that there's a conversation going on??
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CHKDSK During SureBackup Boot

Post by Gostev »

The direct link we (R&D) have is to the ReFS dev team only. But normally, our support opens cases with the Microsoft Premier Support automatically as soon as the issue is confirmed to be on their side. It's actually very common for our support to open cases with other vendors because so many issues our users run into have nothing to deal with Veeam. And we in R&D simply don't track all these cases, but the engineer on your case will be able to tell you if they already opened a case.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: CHKDSK During SureBackup Boot

Post by mcz »

Thanks Anton. The engineer told me that he's waiting for a response from T3 and I have already asked if they talked to Microsoft. Maybe they did but at least it wasn't confirmed to me.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: CHKDSK During SureBackup Boot

Post by mcz » 1 person likes this post

Update: My engineer wrote me the following:
The MS case was opened. Title: "NTFS corruption when mounting a disk to a different Windows version"
So finally the case has reached Microsoft. I'm still surprised that they didn't fix it (or maybe even didn't recognize it???) because the latest Windows 10 version has the same bug, so it exists since may...

The ugly part here is that you're limited when you're running such an affected version. Engineer told me the following:
The data is not safe only during SureBackup DC configuration, or replica Re-Ip (in that case Veeam is mounting replicated VM hard disk to the proxying server), or Windows FLR. In those cases, it would be better to avoid assign Veeam server or proxy server role to the Windows 10 2004 servers. In all other cases everything should be OK (Backup, restore, replica, other OS FLR,etc... ).
I'm sure I don't know every detail of this case (how should I) but it looks like this NTFS bug is quite severe for a properly working installation and so it still surprises me that it takes so long overall...
helpdeskvsg
Novice
Posts: 3
Liked: never
Joined: Jun 26, 2015 9:46 am
Full Name: Helpdesk Voortman
Contact:

Re: CHKDSK During SureBackup Boot

Post by helpdeskvsg »

Is this issue resolved somehow? I've just migrated from a Windows Server 2016 Veeam B&R server to Windows Server 2022 and as of that moment every Windows Server 2012R2 DC starts a chkdks when run from SureBackup. SureBackup with a Windows Server 2012R2 server or higher (no DC) just completes successful. I've created a ticket for this.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: CHKDSK During SureBackup Boot

Post by mcz »

I've given up and migrated to a higher OS version, that's why I can't tell you anything about it. But I'm not surprised that you still see it...
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: CHKDSK During SureBackup Boot

Post by PetrM »

Hello,

I believe you would see the same behavior if you rebooted one of the affected servers but anyway I recommend you continue working with our support team to sort out this problem.

Thanks!
Post Reply

Who is online

Users browsing this forum: ante_704, Baidu [Spider], Semrush [Bot] and 249 guests