Linux incremental volume backups larger than expected

[aj_potc]

Case #05485189

Veeam support refuses to look into this case due to my OS version (Rocky Linux 8.6), which they say is unsupported. I don't understand this refusal, as the support issue I'm raising has not been affected by changes in the OS, and the OS I'm using is simply a rebuild of RHEL 8.6, which is the latest available.

In any case, my issue is that daily incremental backups appear larger than I would expect (and always have for the system in question). Is there any method in which I can determine the reason for their size? I realize this may be difficult since Veeam uses CBT to do the backup, but I'd really like to get to the bottom of it and hopefully save some space on my backup repositories.

Thank you very much for any suggestions.

Secondly, is it official Veeam support policy to deny any support for clones of RHEL such as Rocky and Alma? Or am I denied support because I'm using version 8.6, which is not yet on your list of supported versions of RHEL? Some clarification here would be helpful, as I don't see the point in paying for support if Veeam will refuse to cover the most popular CentOS 8 replacements.

Thank you.

[Mildur]

Hi AJ

Is there any method in which I can determine the reason for their size?

There isn't a direct method built-in in our products. We scan the changed data which CBT is telling us to scan.
Our support would start with the debug logs. They are stored at this location: /var/log/veeam/

Secondly, is it official Veeam support policy to deny any support for clones of RHEL such as Rocky and Alma?

Indeed, we cannot claim support for all clone-distributions, as no matter how close to the original distribution they may be, we do not perform QA testing on them and the nuances that make these clones different than the original often introduce edge-cases that need to be accommodated for.

I'll note your request for support, but keep in mind that there are many such clone and each has its own devoted following; whether we will invest resources into testing them for support depends on many factors besides just popularity, so unfortunately I cannot mention if or when it would be supported.
In most cases it should "just work", but keep in mind taking this route may mean you need to handle some compatibility issues on your own.

Thanks
Fabian

[aj_potc]

Thanks for your reply.

Unfortunately I haven't had any luck trying to discover the reason for the unexpectedly large incremental backups. Interestingly, for other Linux systems I'm backing up (mostly VMs), I'm not seeing the same problem I am on this physical server.

If it makes any difference, the system is using the XFS file system on top of md (software) RAID, and its role is a Web server. I suppose the software RAID wouldn't make a difference, as I don't think Veeam cares about the underlying storage architecture.

If anyone has had experience troubleshooting a similar issue, and was able to pinpoint a cause, I'd be happy to hear about it.
Any ideas about identifying the source of the changed blocks would also be appreciated.

Thank you.

[PTide]

@aj_potc,

Please share your logs via any fileshare of your choice.

Also please tell us more details:

1. Was it always like that (large increments), or it just started recently?
2. How exactly 'large increments' look like?

I mean, had you just dropped a 2GB file on the system and expected the backup to be 2GB in size but it turned out to be 3+ GBs? Please describe your scenario and observations.

3. Is there any kind of deduplication mechanism that might be shuffling blocks around on the disk thus causing the increment to grow?

4. What are your processed/read/transferred numbers from the job session?

Thanks!

[aj_potc]

Hi @PTide,

Thanks for your reply.

I did include logs in the case mentioned in my first post, and these were eventually analyzed by one of your engineers and his support manager. No issue was found with the Linux agent, and all appears to be running normally in that regard.

1. The incremental backup sizes have always been larger than expected for this physical server.
2. Daily incremental backups are typically 40-50 GB, even if little or no extra content is added to the server. Veeam's compression helps a bit. But, within the span of a week, my Veeam incremental backups for this server will take up about 250+ GB on my backup repository. This represents more than 10x the actual data growth rate for the server. So, as you can imagine, it certainly puts a dent in my repository capacity.
3. I don't use any deduplication. It's just XFS on top of software RAID. I also don't know of any processes that would be reading and writing into the filesystem to create changes. This is a Web server, so there's plenty of reading going on, but very little writing.
4. Normal numbers for processed/read/transferred from a day in which not much content was added (maybe a few GB) look like this:
   3.5 TB read, 68.2 GB processed, 39.1 GB (1.7x) transferred
   The compression for each day is always about 1.6-1.7x.

Thanks very much for any ideas!

[PTide]

Do you mind if I keep shooting blindly? : ) Any round-robin databases on that web server? Monitoring tools can use those.

As for "very little writing" - overwriting the existing data with the same data also counts as "write".

Thanks

[aj_potc]

> Do you mind if I keep shooting blindly? : )

Please, be my guest!

My main reason for starting the topic is to see if anyone saw something similar and figured out some way to narrow it down to a specific cause. Failing that, blind guesses are about the only thing I can count on at this point!

> Any round-robin databases on that web server? Monitoring tools can use those.

I use the sar (system activity report) utility to log system statistics. But that produces only about 700 KB of data per day. It also runs on my other systems, which don't exhibit the same problem with unexpectedly large backups.

There are no other databases I can think of other than ones that might be used as part of regular Linux system services. The same is true of logging. This system isn't creating tens of gigabytes of Apache logs per day.

> As for "very little writing" - overwriting the existing data with the same data also counts as "write".

I understand. I've examined all custom cron jobs, and there's nothing that creates or overwrites any significant amount of data with any frequency. The same is true of the Web site code. When static content is added, it isn't typically overwritten -- at least not in large volumes.

Thank you!

[aj_potc]

Just for confirmation:

Anything written into /tmp is excluded from the CBT monitoring, correct?

As a troubleshooting step, I'm checking all applications (especially Apache) for various temporary files and caches, and trying to ensure they are all written to /tmp and not somewhere else in the file system.

Thanks!

[PTide]

Anything written into /tmp is excluded from the CBT monitoring, correct?

Not really. For a volume-level backup veeamsnap tracks changes on every volume that is included in a backup scope. It is currently unable of excluding specific directories from tracking.

Thanks!

[aj_potc]

Well, that's not good news. So much for that trial...

My backups are set to "Entire computer," and the VBR UI says that "deleted, temporary, and page files are automatically excluded," so that's why I assumed tmpfs would be, too.

What about swap space? Support indicated to me that this shouldn't be backed up, but is it also tracked by CBT? I don't have any reason to believe my swap space is very active, but just want to check.

Thanks for the clarification.

[aj_potc]

I've done some experimenting on my questions, and it seems like swap space is indeed being backed up when Veeam is set to "Entire computer." It appears Veeam treats swap (which is a separate volume) as a regular data volume, and does not ignore it as the UI suggests it will.

Can anyone confirm this?

The followup questions to that are: how busy is my swap space, and could it be causing inflated backup sizes?
I know that the system has plenty of RAM, and swap usage is always a very small number, but that doesn't tell me how active it is. I don't believe it's very active, and doubt that it's causing CBT to record excessive changes, but I'm running out of things to investigate.

Do you think it's worthwhile to try a little experiment by setting up a new backup job in which I back up only the swap volume to see how large it is? Perhaps this would give me a way to rule it out?