Incremental backups larger than expected? How I fixed the problem

[aj_potc]

Summary

I've long had a problem with Linux-based Web servers producing incremental backups much larger than expected, despite minimal actual file changes.

After some investigation, I discovered that Apache was updating atime (access time) whenever it served a file, causing unnecessary metadata changes across many files. Because Veeam’s Changed Block Tracking (CBT) detects any modified blocks, this resulted in massive backup sizes—even though the actual file content remained unchanged.

This is a description of my troubleshooting process, which I'm sharing in the hope it can help someone else. This may also be a useful tip for Veeam support to keep in its toolkit of possible suggestions for people with similar problems.


The problem: Backups much larger than expected

For quite some time, I've been experiencing much larger than expected incremental backups on a Web server that is heavy on media content. Even on days in which just a few GB of images were changed or added, my daily backups would consistently be in the neighborhood of 60-80 GB (or more).

Attempts to find a solution: Veeam support

I reached out to Veeam support a couple of times hoping for assistance, but they had no ideas and generally seemed very uninterested in helping me investigate. Their refrain was always to restate how Veeam CBT works, reiterating that something must be causing blocks to change on disk, or else Veeam's changed block tracking wouldn't be backing them up.

But I knew that such a large volume of data was not changing from one backup to the next. This is a Web server doing primarily read-only operations. It's serving files, not modifying them! To verify this, I checked which files were modified in the last 24 hours:

Code: Select all

find /data -type f -mtime -1

This resulted in exactly what I expected: newly added or modified files with an aggregate size way smaller than my daily backups. So no help there.

The reason for block changes: metadata updates

Recently, I spun up a new Web server. Before going into production, I copied over the static media files and started Veeam backups. During this pre-production phase, daily incremental backups were very small, in line with expectations. However, on the very day the new server went into production, my backups suddenly ballooned in size.

I started troubleshooting again. If the file contents were not changing, then what could my Web server be doing that would cause blocks on disk to change? The answer is that it was changing metadata, specifically the file access times.

Each time Apache serves up a file, it updates the access time for that file. Since files are being served from all over the volume essentially randomly, this means that blocks were being changed all over the place. Veeam CBT was dutifully picking up these changes and marking those blocks for backup.

You can verify this by running the following command, which finds all files in which the access time is within the last 24 hours:

Code: Select all

find /data -type f -atime -1 -exec ls -lh --time=atime {} +

When I ran that on my system, it returned hundreds of thousands of files, each one stamped with an access time matching the time it was retrieved according to the Apache access log.


The solution: Mount the data volume with option noatime

The solution to the problem is rather simple. By default, RHEL and Rocky Linux (and probably other modern distros) mount volumes with a set of default options, one of which is relatime. Instead, we want to mount the Web data volume with the flag noatime, which disables the recording of access time altogether.

Here's an example of /etc/fstab with that change:

Code: Select all

UUID=[UUID] /data                   xfs     defaults,noatime        0 0

After making this change and remounting the data volume, my incremental backups are now normal sized.

Of course, if you use the file access time for something specific in your application, then this fix may not be applicable. However, so far I have found no drawbacks.

[rovshan.pashayev]

Hi,

Thanks for sharing this!
And we will review this from outside.

[rovshan.pashayev]

Hello @aj_potc ,

In upcoming releases, we will address this problem.

[aj_potc]

Hi Rovshan,

That's good news!

How do you anticipate this will work? Will you change the behavior of the CBT module to ignore atime?

The very interesting thing about atime is that several common server applications modify it, and yet virtually none actually use it! I'm sure there must be organizations out there using atime to generate detailed audit logs, but otherwise it seems to have little practical use.

[rovshan.pashayev]

Hi,

It is too early to answer now, so please stay tuned for upcoming releases.
However, I can say that the module will be able to handle atime changes properly.