Discussions specific to the VMware vSphere hypervisor
Post Reply
Vasko
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 31, 2013 12:11 pm
Full Name: Felix Geschwindner
Contact:

Filesystem consistent snapshots on Linux guest OS

Post by Vasko » Mar 26, 2015 12:23 pm 1 person likes this post

Hi Veeam Community,

during my last disaster recovery simulation I came across corrupted files in the backup which were not corrupt in the live version. This apparently happened because of the nature of crash consistent snapshots/backups of the corresponding Linux VM: Snapshoting a VM is freezing I/O in place until the redo logs have been created which is when I/O is resumed. When there was a pending file operation in the memory/buffer which hasn't finished, the file will be corrupt in the snapshot and consistent in the live system. This is no problem for Windows systems because VSS magic takes care of this. In Linux, VSS is not an option. Instead it uses pre-freeze and post-thaw scripts and a very mysterious plugin/driver that apparently no one has any information about.

The scripts are a fair solution for getting databases into a backup state (if they support it) but it doesn't help for file system level operations or applications that demand 100% up time. My hopes are with the driver/plugin that vmtools supposedly should make use of to get the file system in a consistent state prior to creating the snapshot. My problem with that thing is that I don't have any evidence that it is using said driver/plugin. Even when I turn on vmtools debug logging I get no information about the snapshot process at all. All I see in the vmtoolsd.log is:

Code: Select all

[Mar 26 11:54:19.378] [   debug] [vmtoolsd] Plugin 'hgfsServer' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'vix' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'guestInfo' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'powerops' initialized.
[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'timeSync' initialized.
---->[Mar 26 11:54:19.380] [   debug] [vmtoolsd] Plugin 'vmbackup' initialized.<----
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'synctime' to '0'.
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'copypaste' to '0'.
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'autohide' to '0'.
[Mar 26 11:54:20.383] [   debug] [vmtoolsd] Setting option 'broadcastIP' to '1'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'enableDnD' to '0'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'synctime.period' to '0'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.enable' to '1'.
[Mar 26 11:54:20.391] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.startup' to '1'.
[Mar 26 11:54:20.231] [   debug] [vmtoolsd] Setting option 'toolScripts.afterPowerOn' to '1'.
[Mar 26 11:54:20.231] [   debug] [vmtoolsd] Setting option 'toolScripts.beforePowerOff' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'toolScripts.afterResume' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'toolScripts.beforeSuspend' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.slewCorrection' to '1'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'time.synchronize.tools.percentCorrection' to '0'.
[Mar 26 11:54:20.232] [   debug] [vmtoolsd] Setting option 'mapRootHgfsShare' to '0'.
[Mar 26 11:54:20.233] [   debug] [vmtoolsd] Setting option 'linkRootHgfsShare' to '0'.
[Mar 26 11:54:49.264] [   debug] [vmtoolsd] Setting option 'broadcastIP' to '1'.
The highlighted line is where vmtools load the vmbackup plugin that is supposed to be used when vmtools get the command to quiesce the file system. But is it doing it? Or is it not?

When I remove the VMtools with:

Code: Select all

apt-get autoremove open-vm-tools
The snapshot is also created without errors even though the ticked ckeckbox "Quiesce guest file system (Needs VMware tools installed)" explicitly says that it requires VMware tools (for testing purposes I trigger the snapshot out of vCenter Server Web UI). I would at least expect a log entry when he ignores the quiesce because of missing VMware tools.

Does anyone have advise on this?

I am on vCenter 5.5 Update 2.

Thanks and Best,
Vasko

iwik
Influencer
Posts: 18
Liked: 2 times
Joined: Apr 02, 2014 11:10 am
Contact:

Re: Filesystem consistent snapshots on Linux guest OS

Post by iwik » Mar 27, 2015 2:03 pm

Quiescing shoud put filesystem into freezed and consistent state. In this KB (bug report) http://kb.vmware.com/selfservice/micros ... Id=2038606 is shown the process

SyncDriver: Freezing using Linux ioctls

Vasko
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 31, 2013 12:11 pm
Full Name: Felix Geschwindner
Contact:

Re: Filesystem consistent snapshots on Linux guest OS

Post by Vasko » Mar 31, 2015 6:48 am

Hi iwik,

I came across that KB article but this is not what is happening. The snapshots are created and no error is thrown in seither logs nor vCenter. The vmdk restore was just inkonsistent. I think I checked the kernel versions but don't recall their version number. I will check again and test a kernel update in case.

Thanks,
Vasko

Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests