CBT issue? - missing files

[mzmultics]

Howdy all,
first of all I apologize for the kinda cryptic subject, I'll try to explain what is happening in a bit.

This is the third time I'm experiencing such issue in our environment. Linux vm running pre-freeze and post-thaw scripts to create MySQL backup.
The pre-freeze script creates a dump of MySQL DB and then logs the presence of the file into default logger (/var/log/messages), once that is completed snapshot of this server is created and post-thaw script is executed. Again at this point I'm logging presence of the dump file to default logger, and after that is done script removes the dump file.

The issue I'm seeing is that when I browse RPS for the MySQL dump file, its not there, as if the snapshot was executed before pre-freeze was completed or after post-thaw deleted the file.

OR there is 3rd option - CBT is not registering correctly presence of the file, and Veeam is not reading it while creating backup.

I've created support case for that - Case # 04243404

Anyone experienced similar behaviour in their environment? Any ideas what to check here?

kind regards,
mzmultics

[PetrM]

Hello,

I think you could isolate the issue and to exclude CBT from the list of potential root causes by checking the behavior with a restore point created within an active full backup run.

Thanks!

[tsightler]

I think it's pretty unlikely to be caused by CBT since CBT isn't about files, but about blocks.

I believe the issue is more likely related to OS/filesystem buffering. I've written some similar scripts, for example the SAP HANA snapshot scripts, and have seen the same issue. When you take a SAP HANA snapshot a number of files are written to disk and, when I first started working with them these files would quite often be missing from the backup. This is because modern journaling filesystems typically write file data directly to disk, but only flush the journal to update metadata on intervals (ext4 is 5 seconds by default). This is how they maintain filesystem consistency, but don't guarantee that you may not lose data after a reboot.

For a running system this is a non-issue as the updated metadata is in OS buffers in RAM so the running OS has a consistent view of the filesystem, but if you take a snapshot at that time and back that up, the inodes on the filesystem table do not match the actual data on disk. When the disk is mounted, your data may be there, but the directory doesn't reflect it because it wasn't updated yet.

The general solution is to simply add a call to "sync" in the pre-freeze script which forces the OS to flush all buffered data immediately. Try adding that after your dump and see if you still have the problem. Obviously I don't know for sure if this is your issue, but it sounded close enough to my experience that I thought it was worth sharing.

[Gostev]

Ha, this made me remember good old times... I will never forget those sync;sync;sync; statements at the end of VMware pre-freeze scripts!

To clarify, the old time classics was to call sync multiple times in a row at a bare minimum 3 times, I have absolutely no idea why - but almost everyone did that, so I assume there was some merit behind it?

[mzmultics]

Hi,

@PetrM, true, didn't think about that.

@tsightler , Gostev - thanks for sharing, I already have sync command (although only 2 times) in the pre-freeze script. For "debug" purposes I've also added 20sec sleep at the end of the pre-freeze (but before sync call) just to make sure OS have enough time to write all the data to disk. I'll make sure to include at least 3 sync calls just to be on a safe side.

Though at some point yesterday, all of a sudden, I can see the file in RPS. Now question is how "reliable" it will be in the future. I'm just wondering if from now on the backup will always include the MySQL dump or I'll have to check periodically for the file in RPS.

Much appreciate all your help, and I'm, well not sure if that is a correct term, happy I'm not the special snowflake that encounter this issue.