Backup taking up all disk space at repository, need help with new solution

[KaptenP]

Hello

We are taking backup with Veeam agent backup for one of our customers important, internal SQL backups. Due to special circumstances, the customer dumps their internal SQL backups (.BAK and log files) to the file cluster and then we take backup of the file cluster. They run full backups once a week and only transactions logs the rest of the week.

The daily backups work out fine and doesn’t grow nearly as much as the monthly backups. The monthly backups are always a full backup due to the Read and Transferred size is equally large as the customer dumps full backups each week to the file cluster. We have a 12 month retention policy for the monthly backups.

We were unaware of how much data it would turn out to be. Right now our repository is almost full and we are unable to expand the partition anymore. We have another 40 TB more free space to allocate in the backend but we are unable to expand the partition beyond 255 TB, the maximum size on Windows Server 2016 for NTFS partitions with a cluster size of 64 KB. We have searched if there was a way to circumvent this limit without having to set up a new repository server but to no avail.

Enable inline data deduplication is enabled, with compression level set to Optimal for the monthly backup job.

The average total size is about 36 TB and after an average dedupe of 2,0x and compression of only 1,0x the transferred size of the backup counts in at 22 TB monthly.

In a best-case scenario, we want to find a solution for this or at least a workaround so that we can find a more sustainable way to handle this amount of data without it taking up so much disk.
Right now, we are thinking about setting up a new Windows Server 2022 as repository server and map the backup here instead while also using some kind of dedup at the repository server instead of the solution we use today. Especially since we can’t take any more backups with only 14 TB free on disk.

We will gladly hear you out if you have any suggestions on how to best to approach this.
Beneath are what we have discussed and some questions that some of you may be able to answer:

- Are there any 3rd party deduplication software applications that you can recommend instead of the Windows and Veeam dedup feature?

- If understood correctly, Windows Server 2022 with the dedup role won’t help us since the backup files becomes so large and that there is a limit of only the first 4 TB that can be deduplicated. Is this correct or have we misunderstood it and the Windows dedup feature can be more efficient in deduplicating the data?

- Data deduplication with a Linux based file system

- If no other solution can be found to minimize the amount of data in the backend from this backup, our last resort will be to take full backups of this cluster to a scaleout repository instead and split it up between different partitions.

- Any other, better suggestion than those mentioned above?

And finally, we do now want to use ReFS for the new repository.



Thanks in advance!

[karsten123]

Repository with ReFS or XFS and calculate the numbers correct.
https://rps.dewin.me/rpc

[MarkBoothmaa]

+1 one for ReFS if going windows. Don't use the inbuilt deduplication as I found in the past the depup optimize job can cause havoc locking the backup files