Filesystem consistent snapshots on Linux guest OS

[Vasko]

Hi Veeam Community,

during my last disaster recovery simulation I came across corrupted files in the backup which were not corrupt in the live version. This apparently happened because of the nature of crash consistent snapshots/backups of the corresponding Linux VM: Snapshoting a VM is freezing I/O in place until the redo logs have been created which is when I/O is resumed. When there was a pending file operation in the memory/buffer which hasn't finished, the file will be corrupt in the snapshot and consistent in the live system. This is no problem for Windows systems because VSS magic takes care of this. In Linux, VSS is not an option. Instead it uses pre-freeze and post-thaw scripts and a very mysterious plugin/driver that apparently no one has any information about.

The scripts are a fair solution for getting databases into a backup state (if they support it) but it doesn't help for file system level operations or applications that demand 100% up time. My hopes are with the driver/plugin that vmtools supposedly should make use of to get the file system in a consistent state prior to creating the snapshot. My problem with that thing is that I don't have any evidence that it is using said driver/plugin. Even when I turn on vmtools debug logging I get no information about the snapshot process at all. All I see in the vmtoolsd.log is:

Code: Select all

[Mar 26 11:54:19.378] [   debug] [vmtoolsd] Plugin 'hgfsServer' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'vix' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'guestInfo' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'powerops' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'timeSync' initialized.
---->[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'vmbackup' initialized.<----
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'synctime' to '0'.
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'copypaste' to '0'.
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'autohide' to '0'.
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'broadcastIP' to '1'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'enableDnD' to '0'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'synctime.period' to '0'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.enable' to '1'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.startup' to '1'.
[Mar 26 11:54:20.231] [   debug] [vmtoolsd] Setting option 'toolScripts.afterPowerOn' to '1'.
[Mar 26 11:54:20.231] [   debug] [vmtoolsd] Setting option 'toolScripts.beforePowerOff' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'toolScripts.afterResume' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'toolScripts.beforeSuspend' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.slewCorrection' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.percentCorrection' to '0'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'mapRootHgfsShare' to '0'.
[Mar 26 11:54:20.233] [   debug] [vmtoolsd] Setting option 'linkRootHgfsShare' to '0'.
[Mar 26 11:54:49.264] [   debug] [vmtoolsd] Setting option 'broadcastIP' to '1'.

The highlighted line is where vmtools load the vmbackup plugin that is supposed to be used when vmtools get the command to quiesce the file system. But is it doing it? Or is it not?

When I remove the VMtools with:

Code: Select all

apt-get autoremove open-vm-tools

The snapshot is also created without errors even though the ticked ckeckbox "Quiesce guest file system (Needs VMware tools installed)" explicitly says that it requires VMware tools (for testing purposes I trigger the snapshot out of vCenter Server Web UI). I would at least expect a log entry when he ignores the quiesce because of missing VMware tools.

Does anyone have advise on this?

I am on vCenter 5.5 Update 2.

Thanks and Best,
Vasko

[iwik]

Quiescing shoud put filesystem into freezed and consistent state. In this KB (bug report) http://kb.vmware.com/selfservice/micros ... Id=2038606 is shown the process

SyncDriver: Freezing using Linux ioctls

[Vasko]

Hi iwik,

I came across that KB article but this is not what is happening. The snapshots are created and no error is thrown in seither logs nor vCenter. The vmdk restore was just inkonsistent. I think I checked the kernel versions but don't recall their version number. I will check again and test a kernel update in case.

Thanks,
Vasko