CHKDSK During SureBackup Boot

[markfellows]

When running a SureBackup job (on an Application group and using ‘Keep the application group running once the job completes’ selected) one of our servers run a chkdsk during boot.

The backup image boots fine using Instant Recovery and our overnight SureBackup job (using ‘Link Backip Jobs’) returns a successful status.

Is the check disk ok are should I run a full backup?

Mark.

[Gostev]

I believe chkdsk automatically kicks in after a few crashes... try rebooting the original server, and see if it gets chkdsk on reboot as well. Thanks!

[markfellows]

Anton,

I'd already tried this. It re-booted fine.

Mark.

[Gostev]

Strange... but it looks like disk in backup has dirty bit set for some reason. Does chkdsk find any errors?

[markfellows]

Don't think it did no. If a dirty bit was set would this not perform a chkdsk during instant recover too?

[markfellows]

Tried it again with the latest backup and everything worked fine.

[Gostev]

Don't think it did no. If a dirty bit was set would this not perform a chkdsk during instant recover too?

Yes, if dirty bit is set in backed up image, chkdsk would be performed no matter what you do - regular restore, instant restore, or run it in virtual lab as a part of SureBackup/U-AIR/Sandbox functionality. There could not be any differences in behavior, because it is the same VM booting from the same disks.

What OS and applications does VM in question run?

[markfellows]

Windows 2008 Server R2 running a couple of small apps at present, WSUS & Kaspersky Admin.

[markfellows]

Seem to be having a similar issue with a different server. Every 1-2 weeks the SureBackup fails as it's taking too long to boot (Chkdsk is running). Running the backup from Instant Recovery confirms this as Chkdsk to run at start up. No errors are reported and once the checks are complete the server boots fine.

File level restore etc. works fine.

Rebooting the live VM seems to clear this problem for another 1-2 weeks. When the VM is rebooted a check disk doesn’t automatically run.

Could it be an issue with the disks on the backup destination (ReadyNas NVX)?

Would it be worth actually running a chkdsk on the live VM?

Any suggestions welcome.

Thanks,

Mark.

[Gostev]

Yes, I would try to reboot live VM and see if chkdsk will run first thing before looking at anything else. Thanks.

[Fiskepudding]

I realize this is an old thread, but I have a similar issue on some of my backups.

SureBackup fails because of "Cannot detect VM staring becaue of timeout".
If I monitor the boot process during surebackup it shows CheckDisk running. Only “problem” it finds is “Deleting corrupt attribute record (128,$T)”.
This also happens if I do an Instant VM restore or Full restore
It only happens on the first boot. (not on restart or power down/up)¨

The production VM has no issues on boot. I have restarted them several times to see if CheckDisk runs.
I have aslo run checkdisk manually on the production VM's and taken new backup.

The issue is a bit random, bacause some times (not often) it boots fine from surebackup,instant VM Recovery or Full restore.

Any bright ideas?

[Dima P.]

Hello Espen,

What type of storage you use for the backups in the question? Could that be deduplication storage? Regarding time out error - have you tried to modify the boot delay parameter? Thank you.

[Fiskepudding]

Thanx for the quick reply Dmitry

The storage is a windows server, no dedupe.
Offcause i could increace the boot time, but that does not fix the underlaying issue. That would just let the checkdisk finish and boot normal.
There is stil a potential issue somewhere.

If that was not clear: The VM's boot fine after checkdisk is finished, both when instant vm restore and full restore are used.

[nefes]

Does the backup in question use AAIP or it is crash-consistent?
For crash-consistent backup checkdisk on restore is expected.

[Fiskepudding]

AAIP is checked for the job. The job contains 24 VM's, where there is this "checkdisk-boot issue" with two VMs at the moment.
The others behave boot fine from SureBackup/Instant VM restore or full restore.

[foggy]

Espen, what kind of VMs the affected ones are? Is there something they have in common that is not related to other VMs? Have you already contacted support with this?

[Fiskepudding]

No, I have not contacted support yet about this issue. Seems like it is time to do that.

Nothing very particular stands out on these VMs.
The only thing with this VM's is that i have used snapshots and reverted to snapshot quite much lately. (Offcuase I clean the snapshots up the same day)
(One is a printserver, where I have used snapshots very much lately testing some Canon drivers..)

Some aditional info:
Just to make sure there is nothing fishy with the backup file on disk, i have removed it from backups, and started a fresh new full backup, still the same problem with the same VMs.
If i boot the production VM's they still boot without any issues.

[veremin]

Seems like the best idea, as we've run out of the ideas about potential causes of the behavior you experience. Once opened, please, post your ticket number here, so that, we can follow the case. Thanks.

[Fiskepudding]

I did some more testing, and narrowed it down a bit. It seems like this problem only happens if the backup runs on a specific proxy and it uses HOTADD.

If I run the same backupjob (just change to a proxy that uses NBD) restores/surebackup runs fine. Support ticket: 00575016

[veremin]

It seems like this problem only happens if the backup runs on a specific proxy and it uses HOTADD.

Is that the only one Hot Add proxy you have in your environment? I'm wondering whether the issue is related to particular proxy mode or to particular proxy itself. Thanks.

[Gostev]

Is it plain Windows OS with Veeam proxy components, or does it perhaps have some 3rd party software as well? Namely, antivirus would be my first suspect.

[Fiskepudding]

I think some info about the environment is needed.

We have two ESXi hosts protected by Veeam, with local storage.
So there is one dedicated VM that acts as proxy on each host.
These two VM has no other tasks, no antivirus, and no third party software. They are both plain, up to date, Windows 2012R2, only “third party software” is the proxy components installed by Veeam and vmtools. They are standalone server as well, no domain.

HOST1 (5 VMs + 1 proxy) have no problems (using “its own” proxy) /hotadd.
HOST2 (24 VM’s + 1 proxy) runs checkdisk on restore on some VMs (using “its own” proxy) /hotadd on backup job.

We also have three other proxies (Also dedicated VMs) on three other hosts. When we use them (with NBD) we have no problem.
But the backup is about 40% slower, that was why we swapped to hotadd. It served us well for a few months, and I really don’t want to go back to NBD.

I am thinking of reinstalling the proxy on HOST2, or copy the proxy on HOST1 to HOST2.

[Fiskepudding]

Well. Since all pointed to the proxy, i removed it, deployed a new windows proxy and ran the backup jobs with the new proxy.
Voila, the Surebackup job afterwards went fine. Need to test this over time, but so far, so good.

If the problem does not resurface, I kept a VeeamZIP of the old proxy if Veeam support would like to investigate further.

[Fiskepudding]

Guess I was too optimistic here. All went fine yesterday. Today it acted up again. To be continued

[foggy]

Is the "pattern" still the same with the new proxy (same VMs, random boot issues)?

Anyway, please continue working with our support on this.

[ksauer]

Based on your responses you have:
Run CHKDSK on the production VM - no issues
Created a new backup - was it an active full or incremental? (I'm assuming active full, but want to make sure)
Ensured AAIP was enabled for the job

For testing purposes, do you have the availability to clone the production machine(s) that are affected and backup the clone to isolate if the issue persists from the production machine or its resolved from the clone?

[Fiskepudding]

There patterns are different, now the issue is on other VMs.

I am working with support, installed patch 4 and curringently seting up a test job with many different settings then were in the original job (support suggestions for trouble shooting).

Yes, testet full backup (even created new test job, so it had to be full)
Cloing the source could be a good idea, but as of now the there is no pattern, new VM's this time around.

Will report back if we get to the bottom of this

[JoshuaPostSAMC]

I have a small handful of VMs that when SureBackup is starting them up, they run a chkdsk while booting and takes so long they error out as not booting in the allotted time.

I've gone to several of the VMs and ran an online chkdsk c: and it comes back clean. I went ahead and scheduled it to run a fix on next boot just to make sure, but why would SureBackup think it needs to run a check when the VM is showing clean? Any way to prevent it?

[PTide]

Hi,

SureBackup does not trigger chkdisk on its own. In fact, it is Windows itself what starts chkdisk on boot in case there is a dirty bit set on the file system.

You can increase the timeout to make SureBackup wait longer. However you should find out why is guest OS starting chkdisk. What do you use - Hyper-V or VMware?

Thank you

[JoshuaPostSAMC]

VMware here. In SureBackup it often finds file fragments it wants to cleanup, and the reduced performance through SureBackup makes it take longer, but doing it with the production machine finds nothing and doesn't take nearly as long to run.

I have increased the timeout and able to get machines to start, but when the scheduled job run all I see are errors about the VM failed to boot in the allotted time and I'm unable to determine if it was chkdsk or something else by booting it up in Troubleshooting mode.