VAAP Linux vs. Tools quiescing

[mkretzer]

According to https://kb.vmware.com/s/article/2044492 VMware tools quiescing uses 3 methods to prepare linux guests for backup:
1. Using FIFREEZE/FITHAW ioctls build into the kernel
2. Using a VMware kernel module (vmsync)
3. Using the Linux sync command

Because of the problems we had with quiescing in windows we switched to VAAP. To get rid of the warnings in our quite mixed jobs we also use VAAP for linux without any scripts on general purpose VMs (web servers for example).
Does Veeam VAAP at all do any preperation on linux systems as quiescing would do? If not it would be better to disable quiescing again for all servers not using pre/post scripts...

Markus

[Gostev]

For Linux, host-based backup AAIP has logic for Oracle, and agent-based backup AAIP in addition to Oracle also has logic for PostgreSQL and MySQL.

If file system quiescing is desired, you can use VMware Tools quiescence. However, at least in early days, it was causing application data corruptions (I very vaguely remember this was from vmsync specifically) and customers had better results doing crash-consistent backups. This prompted us to go the pre-freeze/post-thaw scripts route for general-purpose Linux AAIP, which can include both application-specific logic and sync;sync;sync; at the end for the file system (if desired).

[mkretzer]

In other words - if we do not implement any scripting we are (for newer systems now all supporting FIFREEZE/FITHAW) better off taking the Linux credentials out and thus using VMware tools for that? Why did you not at least implement FIFREEZE/FITHAW for AAIP?

[Andreas Neufert]

There is no similar tool like VSS within Linux and the only way is to implement this yourself.

VMware Tools quiescence has 2 options. Run scripts and run file system freeze (which is disabled by default for good reasons).

So our processing has Oracle consistency, Indexing and own script processing.

In most of the cases you need non of them. It is the same with any file backup tool. File system freezes will usually lead to any application downtime or corruption. As well open files can be damaged when the applications are in the write processes. Not a good idea.

[mkretzer]

"Run scripts and run file system freeze (which is disabled by default for good reasons)."

it is disabled by default? Where is this documented? According to https://kb.vmware.com/s/article/2044492 3 different methods are always used.

Edit: I enabled debug logging on a debian 10 system with default open-vm-tools installed. As described in the article above it freezes the OS!
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupAsyncCallback
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupPostProcessCurrentOp
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] VmBackupPostProcessCurrentOp: checking VmBackupOnFreeze
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] Async request 'VmBackupOnFreeze' completed
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupEnableSyncWait
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] Submitted backup start task.
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupSyncDriverStart
...
[2021-04-30T07:18:00.633Z] [ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
...
[2021-04-30T07:18:00.854Z] [ debug] [vmsvc] SyncDriver: opening path '/boot'.
[2021-04-30T07:18:00.854Z] [ debug] [vmsvc] SyncDriver: freezing path '/boot' (fd=10).
[2021-04-30T07:18:00.903Z] [ debug] [vmsvc] SyncDriver: successfully froze '/boot' (fd=10).

what i find rather interesting:
[2021-04-30T07:18:00.903Z] [ debug] [vmsvc] SyncDriver: freezing path '/' (fd=11).
[2021-04-30T07:18:00.903Z] [ debug] [vmsvc] SyncDriver: freeze on '/' returned: 16 (Das Gerät oder die Ressource ist belegt) -> Device or resource is busy

[Andreas Neufert]

I did not find the default values for the VMware ones, but when you click through the wizard you see that it is disable by default.

Regarding Open VM Tools, the default is as well disabled sync: https://github.com/vmware/open-vm-tools ... tools.conf

[Andreas Neufert]

Please check as well this discussion from 2013 and the comments from tsightler there regarding your filesystem concerns.

[mkretzer]

Regarding Open VM Tools, the default is as well disabled sync: https://github.com/vmware/open-vm-tools ... tools.conf

#enableSyncDriver=true -> commented out does not mean disabled by default if i am not correct.

Futhermore, i found something even more interesting - i we enable VAAP and quescing the system is indeed trying to use sync even with Veeam not logging it - so this might be the "best of both worlds" for us....

[tsightler]

I believe that @mkretzer is correct and that open-vm-tools does default to using FIFREEZE/FITHAW and has for quite a while. It's been quite a long time since I dug into the source code of open-vm-tools at all, but the most interesting part when talking about quiescence is services/plugins/vmbackup/stateMachine.c.

I believe this file contains the only reference to the enableSyncDriver which is in the following code snippet:

Code: Select all

#elif defined(__linux__)
         /*
          * If quiescing is requested on linux platform,
          * only allow SyncDriver provider
          */
         if (gBackupState->quiesceFS &&
             VMBACKUP_CONFIG_GET_BOOL(ctx->config, "enableSyncDriver", TRUE)) {
            provider = VmBackup_NewSyncDriverOnlyProvider();
         }
#endif

Just looking at this your might think enableSyncDriver has to be TRUE for FIFREEZE/FITHAW to be enabled, i.e. the provider is only enabled if the config value is set to true. However, if you look at the surrounding code, you find that this code itself is inside of a conditional, specifically, it is only called if forceQuiesce = TRUE, which it should pretty much never be.

When forceQuiesce = FALSE or undefined, the provider is set to VmBackup_NewSyncDriverProvider instead of the VmBackup_NewSyncDriverOnlyProvider. The basic difference between these two is that the default behavior is to try to use FIFREEZE/FITHAW but, if it fails, it falls back to a NULL provider that basically just runs sync and reports success to the caller, while the VmBackup_NewSyncDriverOnlyProvider reports failure if it is unable to freeze the filesytem. The historical context for forceQuiesce is that it was used for VMware VADP and perhaps is still used for Host Based Replication when they are configured to required quiesce. I don't believe there were any other standard consumers of forceQuiesce=TRUE with VMware tools.

So, the default behavior of the code appears to match the VMware 2044492 except for the fact that, if it's Windows, it tries VSS first (it adds a VSS provider first in the list) and then attempts to use the VMware tools sync driver.

On Linux, there is no more sync driver (I don't believe the VMware sync driver has been included in VMware tools since the 3.x kernels), so it just tries FIFREEZE/FITHAW and falls back to sync, which I think is very safe.

The old Linux vmsync driver was always a nightmare and I've ALWAYS recommended against it (crash consistent was more reliable), the Windows sync driver has certainly had it's share of issues over the years, but it's been quite a while since I had heard of corruption issues like you are seeing. On Linux, FIFREEZE/FITHAW has generally been reliable with the exception of a pretty good blip back in the kernel 3.x days when it was first becoming common. I don't remember the exact version but it was definitely buggy in RHEL7 kernel for a while and would causes system hangs, although I don't think it caused any filesystem corruption.

Are you seeing corruption issues with both Windows and Linux?

I believe the best practice recommendation to use AAIP and VMware tools in mixed jobs was so that AAIP would be used for Windows and VMware tools (and thus FIFREEZE/FITHAW) for Linux (or app specific stuff for Oracle).

[mkretzer]

Hi,

no, only corruption in specific Windows 2012R2 machine(s?). Never any bigger filesystem or data corruption issues on linux other than the normal "orphaned inode" at boot.
Since AAIP + VMware tools quiescence works together in Linux this seems to be a solution for 99,9 % of cases, we can live with this!

Markus