Discussions specific to the VMware vSphere hypervisor
Post Reply
mkretzer
Veeam Legend
Posts: 875
Liked: 263 times
Joined: Dec 17, 2015 7:17 am
Contact:

VAAP Linux vs. Tools quiescing

Post by mkretzer »

According to https://kb.vmware.com/s/article/2044492 VMware tools quiescing uses 3 methods to prepare linux guests for backup:
1. Using FIFREEZE/FITHAW ioctls build into the kernel
2. Using a VMware kernel module (vmsync)
3. Using the Linux sync command

Because of the problems we had with quiescing in windows we switched to VAAP. To get rid of the warnings in our quite mixed jobs we also use VAAP for linux without any scripts on general purpose VMs (web servers for example).
Does Veeam VAAP at all do any preperation on linux systems as quiescing would do? If not it would be better to disable quiescing again for all servers not using pre/post scripts...

Markus

Gostev
SVP, Product Management
Posts: 29118
Liked: 5364 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by Gostev »

For Linux, host-based backup AAIP has logic for Oracle, and agent-based backup AAIP in addition to Oracle also has logic for PostgreSQL and MySQL.

If file system quiescing is desired, you can use VMware Tools quiescence. However, at least in early days, it was causing application data corruptions (I very vaguely remember this was from vmsync specifically) and customers had better results doing crash-consistent backups. This prompted us to go the pre-freeze/post-thaw scripts route for general-purpose Linux AAIP, which can include both application-specific logic and sync;sync;sync; at the end for the file system (if desired).

mkretzer
Veeam Legend
Posts: 875
Liked: 263 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by mkretzer »

In other words - if we do not implement any scripting we are (for newer systems now all supporting FIFREEZE/FITHAW) better off taking the Linux credentials out and thus using VMware tools for that? Why did you not at least implement FIFREEZE/FITHAW for AAIP?

Andreas Neufert
VP, Product Management
Posts: 5486
Liked: 1123 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by Andreas Neufert »

There is no similar tool like VSS within Linux and the only way is to implement this yourself.

VMware Tools quiescence has 2 options. Run scripts and run file system freeze (which is disabled by default for good reasons).

So our processing has Oracle consistency, Indexing and own script processing.

In most of the cases you need non of them. It is the same with any file backup tool. File system freezes will usually lead to any application downtime or corruption. As well open files can be damaged when the applications are in the write processes. Not a good idea.

mkretzer
Veeam Legend
Posts: 875
Liked: 263 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by mkretzer »

"Run scripts and run file system freeze (which is disabled by default for good reasons)."

it is disabled by default? Where is this documented? According to https://kb.vmware.com/s/article/2044492 3 different methods are always used.

Edit: I enabled debug logging on a debian 10 system with default open-vm-tools installed. As described in the article above it freezes the OS!
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupAsyncCallback
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupPostProcessCurrentOp
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] VmBackupPostProcessCurrentOp: checking VmBackupOnFreeze
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] Async request 'VmBackupOnFreeze' completed
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupEnableSyncWait
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] Submitted backup start task.
[2021-04-30T07:18:00.632Z] [ debug] [vmbackup] *** VmBackupSyncDriverStart
...
[2021-04-30T07:18:00.633Z] [ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
...
[2021-04-30T07:18:00.854Z] [ debug] [vmsvc] SyncDriver: opening path '/boot'.
[2021-04-30T07:18:00.854Z] [ debug] [vmsvc] SyncDriver: freezing path '/boot' (fd=10).
[2021-04-30T07:18:00.903Z] [ debug] [vmsvc] SyncDriver: successfully froze '/boot' (fd=10).

what i find rather interesting:
[2021-04-30T07:18:00.903Z] [ debug] [vmsvc] SyncDriver: freezing path '/' (fd=11).
[2021-04-30T07:18:00.903Z] [ debug] [vmsvc] SyncDriver: freeze on '/' returned: 16 (Das Gerät oder die Ressource ist belegt) -> Device or resource is busy

Andreas Neufert
VP, Product Management
Posts: 5486
Liked: 1123 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by Andreas Neufert »

I did not find the default values for the VMware ones, but when you click through the wizard you see that it is disable by default.

Regarding Open VM Tools, the default is as well disabled sync: https://github.com/vmware/open-vm-tools ... tools.conf

Andreas Neufert
VP, Product Management
Posts: 5486
Liked: 1123 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by Andreas Neufert »

Please check as well this discussion from 2013 and the comments from tsightler there regarding your filesystem concerns.

mkretzer
Veeam Legend
Posts: 875
Liked: 263 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by mkretzer »

Andreas Neufert wrote: Apr 30, 2021 7:49 am Regarding Open VM Tools, the default is as well disabled sync: https://github.com/vmware/open-vm-tools ... tools.conf
#enableSyncDriver=true -> commented out does not mean disabled by default if i am not correct.

Futhermore, i found something even more interesting - i we enable VAAP and quescing the system is indeed trying to use sync even with Veeam not logging it - so this might be the "best of both worlds" for us....

tsightler
VP, Product Management
Posts: 5905
Liked: 2759 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by tsightler » 1 person likes this post

I believe that @mkretzer is correct and that open-vm-tools does default to using FIFREEZE/FITHAW and has for quite a while. It's been quite a long time since I dug into the source code of open-vm-tools at all, but the most interesting part when talking about quiescence is services/plugins/vmbackup/stateMachine.c.

I believe this file contains the only reference to the enableSyncDriver which is in the following code snippet:

Code: Select all

#elif defined(__linux__)
         /*
          * If quiescing is requested on linux platform,
          * only allow SyncDriver provider
          */
         if (gBackupState->quiesceFS &&
             VMBACKUP_CONFIG_GET_BOOL(ctx->config, "enableSyncDriver", TRUE)) {
            provider = VmBackup_NewSyncDriverOnlyProvider();
         }
#endif
Just looking at this your might think enableSyncDriver has to be TRUE for FIFREEZE/FITHAW to be enabled, i.e. the provider is only enabled if the config value is set to true. However, if you look at the surrounding code, you find that this code itself is inside of a conditional, specifically, it is only called if forceQuiesce = TRUE, which it should pretty much never be.

When forceQuiesce = FALSE or undefined, the provider is set to VmBackup_NewSyncDriverProvider instead of the VmBackup_NewSyncDriverOnlyProvider. The basic difference between these two is that the default behavior is to try to use FIFREEZE/FITHAW but, if it fails, it falls back to a NULL provider that basically just runs sync and reports success to the caller, while the VmBackup_NewSyncDriverOnlyProvider reports failure if it is unable to freeze the filesytem. The historical context for forceQuiesce is that it was used for VMware VADP and perhaps is still used for Host Based Replication when they are configured to required quiesce. I don't believe there were any other standard consumers of forceQuiesce=TRUE with VMware tools.

So, the default behavior of the code appears to match the VMware 2044492 except for the fact that, if it's Windows, it tries VSS first (it adds a VSS provider first in the list) and then attempts to use the VMware tools sync driver.

On Linux, there is no more sync driver (I don't believe the VMware sync driver has been included in VMware tools since the 3.x kernels), so it just tries FIFREEZE/FITHAW and falls back to sync, which I think is very safe.

The old Linux vmsync driver was always a nightmare and I've ALWAYS recommended against it (crash consistent was more reliable), the Windows sync driver has certainly had it's share of issues over the years, but it's been quite a while since I had heard of corruption issues like you are seeing. On Linux, FIFREEZE/FITHAW has generally been reliable with the exception of a pretty good blip back in the kernel 3.x days when it was first becoming common. I don't remember the exact version but it was definitely buggy in RHEL7 kernel for a while and would causes system hangs, although I don't think it caused any filesystem corruption.

Are you seeing corruption issues with both Windows and Linux?

I believe the best practice recommendation to use AAIP and VMware tools in mixed jobs was so that AAIP would be used for Windows and VMware tools (and thus FIFREEZE/FITHAW) for Linux (or app specific stuff for Oracle).

mkretzer
Veeam Legend
Posts: 875
Liked: 263 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: VAAP Linux vs. Tools quiescing

Post by mkretzer »

Hi,

no, only corruption in specific Windows 2012R2 machine(s?). Never any bigger filesystem or data corruption issues on linux other than the normal "orphaned inode" at boot.
Since AAIP + VMware tools quiescence works together in Linux this seems to be a solution for 99,9 % of cases, we can live with this!

Markus

Post Reply

Who is online

Users browsing this forum: No registered users and 39 guests