Comprehensive data protection for all workloads
mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 30, 2019 10:00 am

Hello,

Case 03681744
We are using Veeam for many years now and never had issues with restores - until this month.

We now have inconsistent filesystem on every second backup of our central file server and totally corrupt filesystem (not fixable via chkdsk!) on every 10 backups of that server.
We tried:
- Using different backup storage (we are just switching to REFS on 1903)
- Using different repo server (one with 2019, one with 1903)
- Doing active full
- Doing full restore, FLR, instant recovery
- Using VMware quiescence and/or Veeam VSS (Veeam support verified that snapshot was ok!)

Nothing helped. Since the issues happen in the middle of the backup chain and newer backups often are 100 % consistent it is unlikely that it is a generic backup chain corruption.

Especially one drive got corrupted again and again: The drive is 3 TB big, uses GPT and had dedup enabled. But also other filesystems of that server showed issues.

We then thought about what was changes in the last few months. One thing we changed was backup method: We switched from direct SAN to NBD and then to hotadd (now on Windows 2019 proxies). For that reason we switched to our old W2016 proxies which do only NBD. In that configuration we had no filesystem issues anymore in our backup files!

Is there any known issue backing up W2012R2 filesserver with GPT, dedup with W2019 proxies via Hot-Add in Update 4b with hotfix from kb2981 installed?

Markus

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Jul 30, 2019 12:22 pm 1 person likes this post

No, definitely no known issues like that at the moment. Thanks for opening the support case!

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Jul 30, 2019 12:31 pm

I've been told we had a support case a while back when a similar issue was found to be caused by the corruption of the file system on the source VM. The production VM was behaving fine, because it was still using the valid data cached in memory (file system metadata and system cache), however the file system alone was already not mountable from backup. Just something to consider during the troubleshooting.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 30, 2019 12:47 pm

Gostev,

would chkdsk /scan not find issues? Would there not be an issue shown in event log when the daily VSS snapshot is created and the volumes are re-mounted?
Would the issue just go away in a new incremental backup?

Markus

skrause
Expert
Posts: 444
Liked: 93 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by skrause » Jul 30, 2019 1:13 pm

I have had a similar issue with a few 2008R2 machines built off an old template that exhibit this kind of behavior when replicated or (less frequently) restored from a backup. I had assumed it had something to do with thin provisioning going back to ESXi 5.0 which the template was built on.

I have never seen it with 2012R2 though and almost all of my machines are 2012R2.

None of the disks I have had issues with were GPT, they were all MBR.

Note that in my case, the machines work. They just take a REALLY long time to reboot after patches because it does a chkdsk scan every time it reboots. I actually have one exhibiting this behavior now but it is slated to be retired in the next couple of months so I didn't put in much work past trying to restore from backup rather than replicating it when I moved it between datacenters.
Steve Krause
Veeam Certified Architect

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by foggy » Jul 30, 2019 1:34 pm

Another thought - I'd check SAN Policy on the proxy server involved in the affected VMs processing. In case it allows bringing mounted disks online, Windows might corrupt the file system on those (by default, Veeam B&R ensures SAN Policy is set to OfflineShared).

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 30, 2019 1:36 pm

No they are set to offline shared....

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 30, 2019 6:38 pm

I did FLR mount of every backup today (i set the backup to run every hour) and there was no NTFS "error". It really seems connected to HotAdd. I just hope Veeam support finds anything...

oliverL
Enthusiast
Posts: 63
Liked: 9 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by oliverL » Jul 31, 2019 2:02 pm

Have you disabled automount and used automount scrub on your new proxies to ensure that windows does not take a look into the disk?

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 31, 2019 2:12 pm

Huh? Automount is enabled!

I am 99,99 % sure i disabled it on all proxies. I do not understand why it is enabled!

Thank you very much - interestingly we have no corruptions in production just on the backups. Is that normal?

Is this only relevant for hot-add?

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 31, 2019 2:30 pm

One second: with 2019 San policy is relevant and not automount correct?

oliverL
Enthusiast
Posts: 63
Liked: 9 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by oliverL » Jul 31, 2019 2:41 pm

I would say automount is relevant for hot-add and SAN-based backup as the proxy will see the disks.

Windows can be messy with disks. I've needed to do some CPU-whitelisting on an ESXi. After adding the SSD (where esxi was installed) to my windows os, it started writing some stuff on the ssd which made ESXi unbootable (yay). (Which obviously doesn't have anything todo with Veeam, but that again showed me how good it is to use automount disable).

And since hot-add-Proxys might not release vmdks after the backup it is even more necessary to use automount disable (there also exists a Veeam-KB about it).

automount scrub will clear the list of drives which windows have seen.
It's just like clearing the recycle-bin which is filled due to automount disable ;-)


I've nether bothered with the san policy to be honest, because at customers and on my own lab (full windows server 2019 environment) i've disabled automount to ensure that my proxies release the vmdks after the backup.

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by foggy » Jul 31, 2019 4:09 pm

mkretzer wrote:
Jul 31, 2019 2:30 pm
One second: with 2019 San policy is relevant and not automount correct?
Correct, SAN Policy is what plays the main role here.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 31, 2019 4:09 pm

What is Veeams view on this?

Edit: Support also told me that Automount can be the cause. If so: Why does Veeam not disable Automount on proxys automatically?

skrause
Expert
Posts: 444
Liked: 93 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by skrause » Jul 31, 2019 6:51 pm 1 person likes this post

It does when a new proxy is added to the infrastructure. I don't think it goes back and turns it off if it was enabled after the Veeam components are installed though.

Is it possible that somehow it got turned on after you added the proxies to your setup? Were you using VM proxies for Direct SAN using iSCSI initiators?
Steve Krause
Veeam Certified Architect

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Jul 31, 2019 7:27 pm

We installed new windows 2016 and now windows 2019 VMs dedicated as proxy and then installed the veeam components. We never changed outmount settings as we believed SAN policy would be enough.

It seems automount was never disabled. But from my point of view Veeam should check it before every backup and warn or disable it.

If automount really can cause this i CANNOT understand Veeam. How is the risk of having inconsistent backups acceptable?

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Jul 31, 2019 10:07 pm

Sorry, I cannot understand what you're saying/asking.

As far as disabling automount, Steve is absolutely correct that Veeam already does this, primarily to prevent issues with Direct SAN transport mode. This process is reflected in the backup proxy deployments logs, and our support can verify based on them if it was correctly disabled, or if something prevented it. By the way, as far as I know, Veeam is still the only VMware backup vendor that will do this automatically for you.

As far as support response, they are likely confused. As noted above, there is a known issue around Direct SAN Access transport mode, where Windows OS would re-signature production VMFS volumes by attempting to automatically mount them. For hot add transport mode though, I don't see how any impact at all on production VM is possible, considering that what is being attached to the backup proxy is a read-only snapshot file.

Overall, this is clearly not an issue that could somehow go unnoticed for 10 years and across hundreds of thousands of deployments, so I am positive we're dealing with some environment-specific problem here. As such, instead of jumping to conclusions, let's give it a chance to be properly investigated by R&D.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 01, 2019 1:34 am

Gostev,

Thank you very much. But with that information or is clear why our production vm is not showing inconsistency: the snapshot is gladly read only.

The question now ist: can Backup be at all inconsistent thanks to auto mount?

I will ask support to check the deployment logs

oliverL
Enthusiast
Posts: 63
Liked: 9 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by oliverL » Aug 01, 2019 7:18 am

In my case and with some customers we could see that the proxys sometimes aren't releasing the VMDKs after the Backup in Hot-Add mode if automount wasn't disabled which meant that some proxys had multiple vmdks from different VMs attached. The same goes for Veeam Replication Proxys.
Disabling automount on those proxies helped with that problem.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 01, 2019 7:31 am

Hanging vmdks was never an issue for us...

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 01, 2019 11:33 am

@gostev support just told me that automount is no longer disabled automatically...

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Aug 02, 2019 12:20 am

Automount is disabled through diskpart san policy=offlineall, as per KB1446.

diskpart automount disable was the legacy approach use to disable automount on Windows versions prior to Windows 2008, where it started causing various issues, for example with installing updates. So, for Windows 2008 or later we switched Microsoft-recommended approach of using san policy.

Anyway, this stuff is completely irrelevant for hot add mode in any case, as per my earlier explanation... suggest we stop beating this dead horse ;)

mkretzer wrote:
Aug 01, 2019 1:34 am
The question now is: can Backup be at all inconsistent thanks to auto mount?
No, because what is backed up is a content of the snapshot, and the snapshot is read-only - so it is simply impossible for anything at all to change a content of the snapshot.

The way these sort of issues are normally troubleshoot by devs is by blocking snapshot removal, and running a binary comparison of snapshot content with what was saved into the backup file. Then, go from there based on results... but first, need to exclude potential CBT issues, data corruption in flight, data processing conveyor bugs etc.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 02, 2019 4:58 am

Aggreed. But support helped me beating the dead horse quite a bit :-(

Which bugs/issues are affected by the switch from Hotadd to NBD? If CBT was at fault it would happen no matter which transport mode.

The next strange thing is: The corruption seems to be always at the basic foundation of the filesystem (MFT corrupt, errors which are detected in the first seconds of a chkdsk run, never at the later stages).

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Aug 02, 2019 1:59 pm

Right, I don't get it myself why a potential data corruption case was sitting with junior T1 engineer for 3 days. So I already asked support VP yesterday to escalate this to T3.

As far as strange things - there are potential explanations, but don't want to speculate further at the moment, until the appropriate troubleshooting tests are done (such as one described above).

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Aug 06, 2019 1:48 pm

Hi, Michael. I think we found the reason for this issue. To make sure our theory is right, can you confirm that in your case, both the backed up VM and the backup proxy VM were cloned from the same template? Thanks!

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 06, 2019 1:53 pm

Hello Gostev,

its still Markus :-)
I can confirm that it is 100 % sure that the machines were not from a template. The backed up machine is very old, the proxy was a fresh setup.

Markus

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 06, 2019 2:05 pm

Anyway, i just confirmed: Half of our volumes get mounted on the proxy (including getting a drive letter) while backup is running! There are LOTS of errors in event log.

It really seems SAN Policy offline-shared does not work and that that corrupts the backup. But we should find out why - other customers might have the same issue without noticing it!

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by Gostev » Aug 06, 2019 2:48 pm 1 person likes this post

Actually, Veeam backup proxy setup sets SAN Policy to offlineAll as opposed to offlineShared.

So if your backup proxy indeed has offlineShared setting set now (or had it enabled at some point), then this would indeed be the root cause of the problem. Moreover, simply switching it back to offlineAll now will no longer help, because those disks are already "known" to the OS, which apparently makes offlineAll policy ignored for them (some sort of bug in Windows). So they will be mounted by backup proxy, which in turn may cause a corruption in case the mounted disks experienced write activity.

[EDIT] I got it all wrong, never mind.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 06, 2019 4:20 pm

But corruption just in the backup correct since the snapshot is read only?

Still, thats one thing i do not understand: since the read only disk cannot be modified how can this cause corruption?

Edit: I checked all our proxies - all are set to "offline, shared". We never modified that. Are you 100 % sure Veeam sets the correct mode?

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Inconsistent Filesystems on restores when backing up with Hotadd

Post by mkretzer » Aug 06, 2019 4:24 pm

Gostev,

i checked documenation:
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

"Veeam Backup & Replication automatically sets the SAN Policy within each proxy to Offline Shared."

If that is true (and i belive it is as we did not change it) and that can cause the volumes to become online this should be treated as a bug/fault!

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], mkretzer and 49 guests