Agent Filesystem Limit of 218TB

[JonJR]

We've just hit the limit of an agent backup with a Filesystem of 218.5TB and we discovered that the Veeam agent has a hard limit of 218 TB - the docs say "Total size of all file systems included in a file-level backup must not exceed 218 TB". We only actually want to backup 30TB or so of the data. We will see if we can shrink the filesystem to 217TB and get away with that.

However we were wondering if this is an arbitrary limit or an actual technical limitation. We anticipate having file systems greater than 218TB in the future that we would want to agent backup.

For reference, it's a RHEL host with BeeGFS file system, which has XFS as the underlying filesystem.

[HannesK]

Hello,
Yes, it is a technical limitation.

As an alternative, you could also consider NAS backup. NAS backup has no limitations on size and performance is much better than Veeam Agent for Linux file-based backup: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Best regards,
Hannes

[PTide]

Hi,

We only actually want to backup 30TB or so of the data. We will see if we can shrink the filesystem to 217TB and get away with that.

I am curious what is the application? Is that a swarm of small files, or rather a bunch of bigger files? How many clients do access them?

Thanks!

[JonJR]

Hello,
Yes, it is a technical limitation.

As an alternative, you could also consider NAS backup. NAS backup has no limitations on size and performance is much better than Veeam Agent for Linux file-based backup: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Best regards,
Hannes

Thanks, will investigate that

[JonJR]

I am curious what is the application? Is that a swarm of small files, or rather a bunch of bigger files? How many clients do access them?

Scientific data (mainly sequence data and mammogram x-rays) that we are analysing using GPUs. And yes, lots and lots of small files... At the top level, we have a "backed up" folder and a "scratch" folder, we didn't want to have totally separate file systems as data gets moved between the two folders regularly. BeeGFS does the tiering between an NVMe tier and a spinning rust tier without changing the file path. Unlike a regular SAN's tiering which is generally reactive, our users promote the data to the NVMe tier by running a BeeGFS command before they run their job in SLURM. Pretty cool stuff!

[JonJR]

As an alternative, you could also consider NAS backup. NAS backup has no limitations on size and performance is much better than Veeam Agent for Linux file-based backup: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Unfortunately what I hadn't realised is that NAS backup is licensed by the TB rather than the server, so it's about $5,000 to back up 50TB this way or if I could shrink my filesystem, it would be included in my B&R subscription.

[HannesK]

sure, it always depends on the needs and technical resources you have. If resizing a filesystem and the speed of VAL is fine, then it sounds reasonable of course.

I just gave you an alternative if technical limitations (speed, size) hit you in a way that require a better solution.