Inconsistent Filesystems on restores when backing up with Hotadd

[mkretzer]

As before, many of our 4000 VM backups are not slightly inconsistent....

[HannesK]

Hello Markus,
I talked to support to move the case to the next tier for deeper investigations

Best regards,
Hannes

[mkretzer]

Thanks Hannes,

its really bad, we testest some other VMs and the proxy does not only mount the disk of one VM read-write.
I wonder how many customers have silently corrupted backups - without alot of restore-tests it would be impossible to find something like this.

This is super-annoying since this time we were absolutely paranoind when setting up the proxies! If only we would have switches to linux proxies (was planned for V11)...

Markus

[mkretzer]

I have one question about this: Will we have to re-transmit all our backup copies? I just saw that the backups copies of the VMs that did an active full transfered ALOT of data to the copy target which gives me the "feeling" that the copy jobs do not depend on the source CBT on ESXi side (which in fact does not show the changes that were done at backup time by the proxy which lead to the corruption) but "knows" which blocks really changed in the backup itself. Is that correct?

[HannesK]

The backup copy job knows which blocks changed in a backup file, yes.

In the easiest case, it's the same amount of data like you see in the VIB files.

For reverse incremental for example, the BCJ knows which parts of the VBK changed and only copies these parts.

If there is for example incremental backup every hour and BCJ only every 24h, then it will consolidate the data to the latest relevant data. Example: Block ABC was overwritten 24x during the day (once per hour), then the BCJ only copies the latest version of block ABC

Indirectly it's tied to VMware CBT via the BJ. But the BCJ only cares about the backup chain

[mkretzer]

"Indirectly it's tied to VMware CBT" - so because in V10 active full resets CBT the CBT on the copy job is reset as well? That would be very nice...

[HannesK]

there is no "CBT" inside a backup file. We read the metadata from the backup file(s) and transfer the blocks needed. VMware CBT reset is independent from the backup copy job. Only the backup job data is taken into account

The backup copy job copies a restore point (not a backup file). If the BJ restore point is consistent, then the BCJ restore is consistent

[mkretzer]

Interesting - how do you do this? This means that the metadata must contain block checksums of every block and it compares this with every run?
This is really kind of magic as this goes extremly fast....

[foggy]

Veeam B&R indeed keeps the checksums of every block and compares those when needed.

[mkretzer]

Still nothing from support (never had a SEV1 without ANY reaction...).
But i have an enhancement request. Why does Veeam not check the mount state of a volume when doing the backup via hotadd? That is the only way to be 100 % sure the backups are not corrupted.

[Gostev]

Still nothing from support (never had a SEV1 without ANY reaction...)

Please check your SPAM filters, as I see plenty of reactions in the support system (to be precise, 8 emails from the Support system to you since the case was opened). Also, it would appear the case is progressing quite quickly through support tiers, which is what we want to see here.

[mkretzer]

Finally something is really happening.

Still, i think my enhancement request makes alot of sense!!

Are there problems like this known with the Linux Proxy?

[Gostev]

Need to understand the problem first (and if it actually exist) before we can answer that.

[mkretzer]

Short update: We can reproduce this...

When the backup runs with hotadd the disk is brought online (but the volumes do not get a drive letter). But diskpart shows volume "healthy" even if "offline" - from my point of view it should not know if the volume is "healthy".
We did include other VMs in the job and those disks stayed offline in the proxy.

Doing instant recovery the disk was instantly showing a corruption and chkdsk ran at start. This means the issue is most likely not an CBT issue or an issue which is fixable by active full (which is good because we put alot of trust in "forever incremental").

Doing the same with NBD proxies shows no corruption at all in several tests!

[Gostev]

OK! As I already explained earlier in the issue recap, the Online status for a disk by itself is never a problem as long as volumes from the disk are not mounted by the OS, which subsequently allows it to make changes to them.

Since you have confirmed that the backed up volumes remain Offline during backup, their content cannot be altered in principle - so whatever issue you may be having appears to be unrelated to the original issue that started this discussion and was worked around in V10. As such, I think it would be best to start a new topic about this new issue, not to confuse future readers that the original issue from 2019 might be back.

I think backing powered off VMs should help to isolate the issue, as then we will be able to see if there are any differences between the two transport methods reading the same content.

[mkretzer]

Gostev, you are right, and i was wrong with my assumptions: We now have found corruption in a backup which has only been done by NBD.

Also, we have stopped getting corruptions in our Hotadd backup when disabling VMware quiescing. Also no corruptions with powered off VMs.

It is now much more likely to be a localized VSS issue!

Thanks for your great support!

[Gostev]

Check out your Windows event log during the backup... you will see OS specifically complaining about data loss caused that the usage of VMware Tools quiescence.

Not surprised to be honest, as it has a history of such bugs spanning my entire time at Veeam. And NOT using VMware Tools quiescence was officially the first VMware backup best practice I documented, shortly after joining Veeam

[mkretzer]

Someone told me in the past that the messages can be "safely ignored" because the messages are all about the duplicate disks. Don't know if it was VMware or Veeam... We saw them on nearly all our 2012R2 VMs (it seems to be fixed in 2019) and never had issues. Strange that this one VM shows "chkdsk needed" so consistently...
We have some VMs which just cannot use the Veeam VSS - do you think it is safer to disable quiescence for such VMs (not databases, exchange or so on - those all do Veeam VSS)?

[Gostev]

From my perspective, yes. I personally would choose crash-consistent over VMware Tools quiescence any time.

[mkretzer]

@Gostev: Perhaps updating the documenation would make sense: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

"It is recommended to enable VMware Tools quiescence and application-aware processing when you add Microsoft Windows and Linux VMs to the same job. In this case, all VMs will be processed in a transactionally consistent manner — either with application-aware processing or VMware Tools quiescence."

[Gostev]

That's actually the correct recommendation because in such configuration (when both options are enabled) AAIP takes precedence over VMTQ, so the latter is never used.

[mkretzer]

I beg to differ. The link describes our sitation exactly: We mainly use AAIP but there are some VMs where this does not work - as described in the link quiescencing is a "fallback" for such situations. We have now learned (both from you, from support and from our inconsitent backups) that this is a very bad fallback option which could cause more harm then just using crash consistency! This information should be reflected there - after all, if we now disable quiescencing we go away from "best practice" that is documented.

[Gostev]

Not necessarily "a very bad failback option" as it depends on a few factors: OS type, OS version, VMware Tools version etc.

I definitely would not make it a blank statement just over one issue you have experienced in the particular configuration. Normally, you should have VMware Support troubleshoot your situation, so that they could understand and fix the bug with your configuration. All I'm saying, VMware Tools quiescence has a long history of such issues (and by the way, not just with Windows). Sorry if I wasn't clear.

Also, just to re-iterate, the best practice is to have both enabled, so you went away from the best practice the moment you disabled one of them (AAIP) which previously prevented VMware Tools quiescence to perform guest processing on the OS where Veeam does it much better natively.

Anyway, we're completely off-topic now