Comprehensive data protection for all workloads
markfellows
Expert
Posts: 121
Liked: 6 times
Joined: Mar 24, 2010 12:15 pm
Full Name: Mark Fellows
Contact:

CHKDSK During SureBackup Boot

Post by markfellows »

When running a SureBackup job (on an Application group and using ‘Keep the application group running once the job completes’ selected) one of our servers run a chkdsk during boot.

The backup image boots fine using Instant Recovery and our overnight SureBackup job (using ‘Link Backip Jobs’) returns a successful status.

Is the check disk ok are should I run a full backup?

Mark.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Check Disk During Boot

Post by Gostev »

I believe chkdsk automatically kicks in after a few crashes... try rebooting the original server, and see if it gets chkdsk on reboot as well. Thanks!
markfellows
Expert
Posts: 121
Liked: 6 times
Joined: Mar 24, 2010 12:15 pm
Full Name: Mark Fellows
Contact:

Re: Check Disk During Boot

Post by markfellows »

Anton,

I'd already tried this. It re-booted fine.

Mark.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Check Disk During Boot

Post by Gostev »

Strange... but it looks like disk in backup has dirty bit set for some reason. Does chkdsk find any errors?
markfellows
Expert
Posts: 121
Liked: 6 times
Joined: Mar 24, 2010 12:15 pm
Full Name: Mark Fellows
Contact:

Re: Check Disk During Boot

Post by markfellows »

Don't think it did no. If a dirty bit was set would this not perform a chkdsk during instant recover too?
markfellows
Expert
Posts: 121
Liked: 6 times
Joined: Mar 24, 2010 12:15 pm
Full Name: Mark Fellows
Contact:

Re: Check Disk During Boot

Post by markfellows »

Tried it again with the latest backup and everything worked fine.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Check Disk During Boot

Post by Gostev »

markfellows wrote:Don't think it did no. If a dirty bit was set would this not perform a chkdsk during instant recover too?
Yes, if dirty bit is set in backed up image, chkdsk would be performed no matter what you do - regular restore, instant restore, or run it in virtual lab as a part of SureBackup/U-AIR/Sandbox functionality. There could not be any differences in behavior, because it is the same VM booting from the same disks.

What OS and applications does VM in question run?
markfellows
Expert
Posts: 121
Liked: 6 times
Joined: Mar 24, 2010 12:15 pm
Full Name: Mark Fellows
Contact:

Re: Check Disk During Boot

Post by markfellows »

Windows 2008 Server R2 running a couple of small apps at present, WSUS & Kaspersky Admin.
markfellows
Expert
Posts: 121
Liked: 6 times
Joined: Mar 24, 2010 12:15 pm
Full Name: Mark Fellows
Contact:

Re: Check Disk During Boot

Post by markfellows »

Seem to be having a similar issue with a different server. Every 1-2 weeks the SureBackup fails as it's taking too long to boot (Chkdsk is running). Running the backup from Instant Recovery confirms this as Chkdsk to run at start up. No errors are reported and once the checks are complete the server boots fine.

File level restore etc. works fine.

Rebooting the live VM seems to clear this problem for another 1-2 weeks. When the VM is rebooted a check disk doesn’t automatically run.

Could it be an issue with the disks on the backup destination (ReadyNas NVX)?

Would it be worth actually running a chkdsk on the live VM?

Any suggestions welcome.

Thanks,

Mark.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Check Disk During Boot

Post by Gostev »

Yes, I would try to reboot live VM and see if chkdsk will run first thing before looking at anything else. Thanks.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

I realize this is an old thread, but I have a similar issue on some of my backups.

SureBackup fails because of "Cannot detect VM staring becaue of timeout".
If I monitor the boot process during surebackup it shows CheckDisk running. Only “problem” it finds is “Deleting corrupt attribute record (128,$T)”.
This also happens if I do an Instant VM restore or Full restore
It only happens on the first boot. (not on restart or power down/up)¨

The production VM has no issues on boot. I have restarted them several times to see if CheckDisk runs.
I have aslo run checkdisk manually on the production VM's and taken new backup.

The issue is a bit random, bacause some times (not often) it boots fine from surebackup,instant VM Recovery or Full restore.

Any bright ideas?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Check Disk During Boot

Post by Dima P. »

Hello Espen,

What type of storage you use for the backups in the question? Could that be deduplication storage? Regarding time out error - have you tried to modify the boot delay parameter? Thank you.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

Thanx for the quick reply Dmitry

The storage is a windows server, no dedupe.
Offcause i could increace the boot time, but that does not fix the underlaying issue. That would just let the checkdisk finish and boot normal.
There is stil a potential issue somewhere.

If that was not clear: The VM's boot fine after checkdisk is finished, both when instant vm restore and full restore are used.
nefes
Veeam Software
Posts: 643
Liked: 162 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: Check Disk During Boot

Post by nefes »

Does the backup in question use AAIP or it is crash-consistent?
For crash-consistent backup checkdisk on restore is expected.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

AAIP is checked for the job. The job contains 24 VM's, where there is this "checkdisk-boot issue" with two VMs at the moment.
The others behave boot fine from SureBackup/Instant VM restore or full restore.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Check Disk During Boot

Post by foggy »

Espen, what kind of VMs the affected ones are? Is there something they have in common that is not related to other VMs? Have you already contacted support with this?
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

No, I have not contacted support yet about this issue. Seems like it is time to do that.

Nothing very particular stands out on these VMs.
The only thing with this VM's is that i have used snapshots and reverted to snapshot quite much lately. (Offcuase I clean the snapshots up the same day)
(One is a printserver, where I have used snapshots very much lately testing some Canon drivers..)

Some aditional info:
Just to make sure there is nothing fishy with the backup file on disk, i have removed it from backups, and started a fresh new full backup, still the same problem with the same VMs.
If i boot the production VM's they still boot without any issues.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Check Disk During Boot

Post by veremin »

Seems like the best idea, as we've run out of the ideas about potential causes of the behavior you experience. Once opened, please, post your ticket number here, so that, we can follow the case. Thanks.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

I did some more testing, and narrowed it down a bit. It seems like this problem only happens if the backup runs on a specific proxy and it uses HOTADD.

If I run the same backupjob (just change to a proxy that uses NBD) restores/surebackup runs fine. Support ticket: 00575016
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Check Disk During Boot

Post by veremin »

It seems like this problem only happens if the backup runs on a specific proxy and it uses HOTADD.
Is that the only one Hot Add proxy you have in your environment? I'm wondering whether the issue is related to particular proxy mode or to particular proxy itself. Thanks.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Check Disk During Boot

Post by Gostev »

Is it plain Windows OS with Veeam proxy components, or does it perhaps have some 3rd party software as well? Namely, antivirus would be my first suspect.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

I think some info about the environment is needed.

We have two ESXi hosts protected by Veeam, with local storage.
So there is one dedicated VM that acts as proxy on each host.
These two VM has no other tasks, no antivirus, and no third party software. They are both plain, up to date, Windows 2012R2, only “third party software” is the proxy components installed by Veeam and vmtools. They are standalone server as well, no domain.

HOST1 (5 VMs + 1 proxy) have no problems (using “its own” proxy) /hotadd.
HOST2 (24 VM’s + 1 proxy) runs checkdisk on restore on some VMs (using “its own” proxy) /hotadd on backup job.

We also have three other proxies (Also dedicated VMs) on three other hosts. When we use them (with NBD) we have no problem.
But the backup is about 40% slower, that was why we swapped to hotadd. It served us well for a few months, and I really don’t want to go back to NBD.

I am thinking of reinstalling the proxy on HOST2, or copy the proxy on HOST1 to HOST2.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding » 1 person likes this post

Well. Since all pointed to the proxy, i removed it, deployed a new windows proxy and ran the backup jobs with the new proxy.
Voila, the Surebackup job afterwards went fine. Need to test this over time, but so far, so good.

If the problem does not resurface, I kept a VeeamZIP of the old proxy if Veeam support would like to investigate further.
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding »

Guess I was too optimistic here. All went fine yesterday. Today it acted up again. To be continued :)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Check Disk During Boot

Post by foggy »

Is the "pattern" still the same with the new proxy (same VMs, random boot issues)?

Anyway, please continue working with our support on this.
ksauer
Veeam Software
Posts: 7
Liked: 1 time
Joined: Jul 24, 2013 6:16 pm
Full Name: Ken Sauer
Contact:

Re: Check Disk During Boot

Post by ksauer »

Based on your responses you have:
Run CHKDSK on the production VM - no issues
Created a new backup - was it an active full or incremental? (I'm assuming active full, but want to make sure)
Ensured AAIP was enabled for the job

For testing purposes, do you have the availability to clone the production machine(s) that are affected and backup the clone to isolate if the issue persists from the production machine or its resolved from the clone?
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Check Disk During Boot

Post by Fiskepudding » 1 person likes this post

There patterns are different, now the issue is on other VMs.

I am working with support, installed patch 4 and curringently seting up a test job with many different settings then were in the original job (support suggestions for trouble shooting).

Yes, testet full backup (even created new test job, so it had to be full)
Cloing the source could be a good idea, but as of now the there is no pattern, new VM's this time around.

Will report back if we get to the bottom of this :)
JoshuaPostSAMC
Expert
Posts: 124
Liked: 22 times
Joined: Jul 30, 2015 7:32 pm
Contact:

[MERGED] SureBackup performing chkdsk

Post by JoshuaPostSAMC »

I have a small handful of VMs that when SureBackup is starting them up, they run a chkdsk while booting and takes so long they error out as not booting in the allotted time.

I've gone to several of the VMs and ran an online chkdsk c: and it comes back clean. I went ahead and scheduled it to run a fix on next boot just to make sure, but why would SureBackup think it needs to run a check when the VM is showing clean? Any way to prevent it?
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: SureBackup performing chkdsk

Post by PTide »

Hi,

SureBackup does not trigger chkdisk on its own. In fact, it is Windows itself what starts chkdisk on boot in case there is a dirty bit set on the file system.

You can increase the timeout to make SureBackup wait longer. However you should find out why is guest OS starting chkdisk. What do you use - Hyper-V or VMware?

Thank you
JoshuaPostSAMC
Expert
Posts: 124
Liked: 22 times
Joined: Jul 30, 2015 7:32 pm
Contact:

Re: SureBackup performing chkdsk

Post by JoshuaPostSAMC »

VMware here. In SureBackup it often finds file fragments it wants to cleanup, and the reduced performance through SureBackup makes it take longer, but doing it with the production machine finds nothing and doesn't take nearly as long to run.

I have increased the timeout and able to get machines to start, but when the scheduled job run all I see are errors about the VM failed to boot in the allotted time and I'm unable to determine if it was chkdsk or something else by booting it up in Troubleshooting mode.
Post Reply

Who is online

Users browsing this forum: Google [Bot], Tesla2k and 169 guests