Backing up Linux instances using XFS filesystem

[nusol_sc]

We have been using Veeam B&R in our on-premise environment for a few years, and given our experience with it we are eager to use Veeam Backup for AWS. After reading through the documentation I have a few points of concern:

[*] I see no discussion on the topic of filesystem quiescence.
[*] There doesn't appear to be a way to trigger pre/post scripts, meaning we can't trigger the freeze/unfreeze of an XFS filesystem before/after a live snapshot.
[*] There doesn't appear to be an option to shut down an instance prior to taking the snapshot and then restart it after the snapshot has been completed.

Given that we have experienced catastrophic filesystem corruption on the same server twice in 2 months, I am eager to get reliable backups going. None of the other servers have had this problem, but now I am extremely uneasy. The snapshot I created of this server after it failed the first time turned out to be corrupt, perhaps due to taking the snapshot when the server was running. As such, I no longer trust live snapshots, at least when using an XFS filesystem.

Is filesystem quiescence performed automatically? I don't want to jump in head-first until I understand the details. What is the recommended approach for backing up Linux servers that use XFS?

[nielsengelen]

Hi, you are correct that there is no integration with Linux filesystems right now. We have this on our roadmap but I can't provide an ETA.

Could you describe the use case for shutting down the instance and then taking the snapshot? Would it be sufficient for you to benefit from using pre/post scripts for your requirements?

On the topic of creating snapshots, snapshots created by Veeam Backup for AWS are outside of the OS (via AWS API's). The issue you describe which has happened in the past 2 months, is that one related to in-guest snapshots or also external snapshots?

[nusol_sc]

Shutting down the servers would be a crude way to guarantee file system consistency when the snapshot is taken, as opposed to freezing/unfreezing the filesystem before/after the snapshot has been taken. We haven't yet been able to determine what caused the file system corruption and subsequent failure of the server, nor why the snapshot we took was bad. We suspect that the snapshot was bad due to taking it while the server was running but it just a guess. Further research needs to be performed to get to the bottom of it.

Being able to trigger pre/post scripts would be very helpful, and definitely preferable as it would provide a mechanism by which tasks could be executed to help prepare the OS for a snapshot without having to shut down the server. For example, a pre-snapshot script could freeze an XFS/EXT3/EXT4/GFS2/BTRFS file system using "xfs_freeze -f" in addition to any application-specific tasks that may be necessary, and once the snapshot is complete the file system could then be unfrozen via "xfs_freeze -u". When I get the chance I am going to test the filesystem freeze -> snapshot -> filesystem unfreeze workflow.

Snapshots were taken in the standard way using EC2 console.

Perhaps also worth noting is that we are using CentOS 8, which is technically not a supported OS (yet). RedHat 8 had been supported for a while, but support was removed. It could also be that there is a fundamental issue related to RedHat 8/CentOS 8. Having said that, we haven't had an issue with any of the other CentOS 8 servers we are running, including our Oracle database server.

[nielsengelen]

Normally, a snapshot created in AWS shouldn't impact the server or filesystem in the way you describe. I understand the request for pre/post scripts but again, no ETA on it yet. We may have to do some research on CentOS running XFS (all be it unsupported as you state). If something like this happens, I guess the best thing you can do is try to open the AWS CLI console.