Instant Recovery: "Filesystem not clean. Replaying journal"

[dcaunt]

Is it considered normal for a linux virtual machine that has been launched via Instant Recovery to give the error "Filesystem not clean" and then replay hundreds of transactions when booting? In case this makes a difference, I'm using "VMware tools quiescence" and backing up the linux virtual machines while they're running (they're production servers). Although the server does finish booting and seems normal, I'm wondering if this is an indication of an underlying problem.

[srebucci]

Hi dcaunt, if the linux machines are up for more than 180 days the autocheck procedure on ext3 run automatically to the next reboot: try to reboot the production machine, you should see the same behaviour. Next, try to backup it again.

[dcaunt]

Hi srebucci. The filesystem is reiserfs, not ext3, and the system has only been up for 114 days. I can't reboot the production machine at the moment, but I will the next time I can schedule downtime (probably tonight). Since it's not ext3 and it hasn't been online for 180 days, are there any other ideas? Given your initial answer, it seems to me that this is not normal behavior for an Instant Recovered machine.

[tsightler]

So "VMware tools quiescence" simply pauses I/O to the underlying file system to give time for a snapshot to be taken. In the end, the filesystem was still open when the snapshot is taken, and if there were any outstanding open files at the time then the journal replay is fully expected. The default journalling mode on most Linux distros and common filesystems (say ext3 or ext4) is called "ordered" journalling. This mode basically journals metadata only, but the file data must be confirmed as written to disk prior to the journalled metadata being marked as committed. In the case of a snapshot backup, when the system reboots, if there are any metadata transactions in the filesystem journal that are not mark as committed (meaning that the filesystem is not 100% sure the actual file data made it to disk) then the metadata is rolled back to it's previous state to provide a consistent file system state.

So in other words, it is normal to see these if there was any outstanding I/O at all on the system.

You can read all the details on ext3/4 journalling at the bottom of http://lwn.net/Articles/203915/ just look for the "Data Mode" section.

If your filesystem happens to be something like ReiserFS, XFS, or JFS they use similar methods of journalling so similar messages would be expected as, no matter the journalling filesystem, the concept behind journalling is to prevent the requirement to run fsck after a restart from an improperly closed drive. Since taking a snapshot of an active filesystem is not a closed drive, they would all need to perform the "journal replay" feature on startup, but I'm not as familiar with their specific messages.

[dcaunt]

Thanks for the detailed answer, tsightler! Based on your reply, it seems I don't have to worry about the messages at the moment (especially since no other problems have arisen). I really appreciate the help.