Inconsistent Filesystems on restores when backing up with Hotadd

[mkretzer]

Hello,

Case 03681744
We are using Veeam for many years now and never had issues with restores - until this month.

We now have inconsistent filesystem on every second backup of our central file server and totally corrupt filesystem (not fixable via chkdsk!) on every 10 backups of that server.
We tried:
- Using different backup storage (we are just switching to REFS on 1903)
- Using different repo server (one with 2019, one with 1903)
- Doing active full
- Doing full restore, FLR, instant recovery
- Using VMware quiescence and/or Veeam VSS (Veeam support verified that snapshot was ok!)

Nothing helped. Since the issues happen in the middle of the backup chain and newer backups often are 100 % consistent it is unlikely that it is a generic backup chain corruption.

Especially one drive got corrupted again and again: The drive is 3 TB big, uses GPT and had dedup enabled. But also other filesystems of that server showed issues.

We then thought about what was changes in the last few months. One thing we changed was backup method: We switched from direct SAN to NBD and then to hotadd (now on Windows 2019 proxies). For that reason we switched to our old W2016 proxies which do only NBD. In that configuration we had no filesystem issues anymore in our backup files!

Is there any known issue backing up W2012R2 filesserver with GPT, dedup with W2019 proxies via Hot-Add in Update 4b with hotfix from kb2981 installed?

Markus

[Gostev]

No, definitely no known issues like that at the moment. Thanks for opening the support case!

[Gostev]

I've been told we had a support case a while back when a similar issue was found to be caused by the corruption of the file system on the source VM. The production VM was behaving fine, because it was still using the valid data cached in memory (file system metadata and system cache), however the file system alone was already not mountable from backup. Just something to consider during the troubleshooting.

[mkretzer]

Gostev,

would chkdsk /scan not find issues? Would there not be an issue shown in event log when the daily VSS snapshot is created and the volumes are re-mounted?
Would the issue just go away in a new incremental backup?

Markus

[skrause]

I have had a similar issue with a few 2008R2 machines built off an old template that exhibit this kind of behavior when replicated or (less frequently) restored from a backup. I had assumed it had something to do with thin provisioning going back to ESXi 5.0 which the template was built on.

I have never seen it with 2012R2 though and almost all of my machines are 2012R2.

None of the disks I have had issues with were GPT, they were all MBR.

Note that in my case, the machines work. They just take a REALLY long time to reboot after patches because it does a chkdsk scan every time it reboots. I actually have one exhibiting this behavior now but it is slated to be retired in the next couple of months so I didn't put in much work past trying to restore from backup rather than replicating it when I moved it between datacenters.

[foggy]

Another thought - I'd check SAN Policy on the proxy server involved in the affected VMs processing. In case it allows bringing mounted disks online, Windows might corrupt the file system on those (by default, Veeam B&R ensures SAN Policy is set to OfflineShared).

[mkretzer]

No they are set to offline shared....

[mkretzer]

I did FLR mount of every backup today (i set the backup to run every hour) and there was no NTFS "error". It really seems connected to HotAdd. I just hope Veeam support finds anything...

[oliverL]

Have you disabled automount and used automount scrub on your new proxies to ensure that windows does not take a look into the disk?

[mkretzer]

Huh? Automount is enabled!

I am 99,99 % sure i disabled it on all proxies. I do not understand why it is enabled!

Thank you very much - interestingly we have no corruptions in production just on the backups. Is that normal?

Is this only relevant for hot-add?

[mkretzer]

One second: with 2019 San policy is relevant and not automount correct?

[oliverL]

I would say automount is relevant for hot-add and SAN-based backup as the proxy will see the disks.

Windows can be messy with disks. I've needed to do some CPU-whitelisting on an ESXi. After adding the SSD (where esxi was installed) to my windows os, it started writing some stuff on the ssd which made ESXi unbootable (yay). (Which obviously doesn't have anything todo with Veeam, but that again showed me how good it is to use automount disable).

And since hot-add-Proxys might not release vmdks after the backup it is even more necessary to use automount disable (there also exists a Veeam-KB about it).

automount scrub will clear the list of drives which windows have seen.
It's just like clearing the recycle-bin which is filled due to automount disable


I've nether bothered with the san policy to be honest, because at customers and on my own lab (full windows server 2019 environment) i've disabled automount to ensure that my proxies release the vmdks after the backup.

[foggy]

One second: with 2019 San policy is relevant and not automount correct?

Correct, SAN Policy is what plays the main role here.

[mkretzer]

What is Veeams view on this?

Edit: Support also told me that Automount can be the cause. If so: Why does Veeam not disable Automount on proxys automatically?

[skrause]

It does when a new proxy is added to the infrastructure. I don't think it goes back and turns it off if it was enabled after the Veeam components are installed though.

Is it possible that somehow it got turned on after you added the proxies to your setup? Were you using VM proxies for Direct SAN using iSCSI initiators?

[mkretzer]

We installed new windows 2016 and now windows 2019 VMs dedicated as proxy and then installed the veeam components. We never changed outmount settings as we believed SAN policy would be enough.

It seems automount was never disabled. But from my point of view Veeam should check it before every backup and warn or disable it.

If automount really can cause this i CANNOT understand Veeam. How is the risk of having inconsistent backups acceptable?

[Gostev]

Sorry, I cannot understand what you're saying/asking.

As far as disabling automount, Steve is absolutely correct that Veeam already does this, primarily to prevent issues with Direct SAN transport mode. This process is reflected in the backup proxy deployments logs, and our support can verify based on them if it was correctly disabled, or if something prevented it. By the way, as far as I know, Veeam is still the only VMware backup vendor that will do this automatically for you.

As far as support response, they are likely confused. As noted above, there is a known issue around Direct SAN Access transport mode, where Windows OS would re-signature production VMFS volumes by attempting to automatically mount them. For hot add transport mode though, I don't see how any impact at all on production VM is possible, considering that what is being attached to the backup proxy is a read-only snapshot file.

Overall, this is clearly not an issue that could somehow go unnoticed for 10 years and across hundreds of thousands of deployments, so I am positive we're dealing with some environment-specific problem here. As such, instead of jumping to conclusions, let's give it a chance to be properly investigated by R&D.

[mkretzer]

Gostev,

Thank you very much. But with that information or is clear why our production vm is not showing inconsistency: the snapshot is gladly read only.

The question now ist: can Backup be at all inconsistent thanks to auto mount?

I will ask support to check the deployment logs

[oliverL]

In my case and with some customers we could see that the proxys sometimes aren't releasing the VMDKs after the Backup in Hot-Add mode if automount wasn't disabled which meant that some proxys had multiple vmdks from different VMs attached. The same goes for Veeam Replication Proxys.
Disabling automount on those proxies helped with that problem.

[mkretzer]

Hanging vmdks was never an issue for us...

[mkretzer]

@gostev support just told me that automount is no longer disabled automatically...

[Gostev]

Automount is disabled through diskpart san policy=offlineall, as per KB1446.

diskpart automount disable was the legacy approach use to disable automount on Windows versions prior to Windows 2008, where it started causing various issues, for example with installing updates. So, for Windows 2008 or later we switched Microsoft-recommended approach of using san policy.

Anyway, this stuff is completely irrelevant for hot add mode in any case, as per my earlier explanation... suggest we stop beating this dead horse

The question now is: can Backup be at all inconsistent thanks to auto mount?

No, because what is backed up is a content of the snapshot, and the snapshot is read-only - so it is simply impossible for anything at all to change a content of the snapshot.

The way these sort of issues are normally troubleshoot by devs is by blocking snapshot removal, and running a binary comparison of snapshot content with what was saved into the backup file. Then, go from there based on results... but first, need to exclude potential CBT issues, data corruption in flight, data processing conveyor bugs etc.

[mkretzer]

Aggreed. But support helped me beating the dead horse quite a bit

Which bugs/issues are affected by the switch from Hotadd to NBD? If CBT was at fault it would happen no matter which transport mode.

The next strange thing is: The corruption seems to be always at the basic foundation of the filesystem (MFT corrupt, errors which are detected in the first seconds of a chkdsk run, never at the later stages).

[Gostev]

Right, I don't get it myself why a potential data corruption case was sitting with junior T1 engineer for 3 days. So I already asked support VP yesterday to escalate this to T3.

As far as strange things - there are potential explanations, but don't want to speculate further at the moment, until the appropriate troubleshooting tests are done (such as one described above).

[Gostev]

Hi, Michael. I think we found the reason for this issue. To make sure our theory is right, can you confirm that in your case, both the backed up VM and the backup proxy VM were cloned from the same template? Thanks!

[mkretzer]

Hello Gostev,

its still Markus
I can confirm that it is 100 % sure that the machines were not from a template. The backed up machine is very old, the proxy was a fresh setup.

Markus

[mkretzer]

Anyway, i just confirmed: Half of our volumes get mounted on the proxy (including getting a drive letter) while backup is running! There are LOTS of errors in event log.

It really seems SAN Policy offline-shared does not work and that that corrupts the backup. But we should find out why - other customers might have the same issue without noticing it!

[Gostev]

Actually, Veeam backup proxy setup sets SAN Policy to offlineAll as opposed to offlineShared.

So if your backup proxy indeed has offlineShared setting set now (or had it enabled at some point), then this would indeed be the root cause of the problem. Moreover, simply switching it back to offlineAll now will no longer help, because those disks are already "known" to the OS, which apparently makes offlineAll policy ignored for them (some sort of bug in Windows). So they will be mounted by backup proxy, which in turn may cause a corruption in case the mounted disks experienced write activity.

[EDIT] I got it all wrong, never mind.

[mkretzer]

But corruption just in the backup correct since the snapshot is read only?

Still, thats one thing i do not understand: since the read only disk cannot be modified how can this cause corruption?

Edit: I checked all our proxies - all are set to "offline, shared". We never modified that. Are you 100 % sure Veeam sets the correct mode?

[mkretzer]

Gostev,

i checked documenation:
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

"Veeam Backup & Replication automatically sets the SAN Policy within each proxy to Offline Shared."

If that is true (and i belive it is as we did not change it) and that can cause the volumes to become online this should be treated as a bug/fault!