feature request: split vbk

[veremin]

This request has been brought up before, but this feature is not planned for the foreseeable product versions. Further plans are yet to be determined. Thanks!

[t.maintz]

Hi,

I'm trying to backup 2 physical Exchange-Servers in a DAG. We have 20 Databases ( 1TB Database Size each) and both Exchangeserver have 10 Active and 10 Passive Copies.
The backup is performed to a Storage that currently allows Files up to 3TB. Veeam fails after the vbk files has reached 3TB for the first Host. Is it possible to split the vbk files?
Our Database and Log is split into 2 mount points: e.g. c:\Storage\DB01 and c:\Storage\Log01. I tried to create one Backup Job with the mount points of DB01 and Log01. The Backups works and runs without failures (logtruncating is displayed successful too), but neither the logtruncating nor the restore works (getting no items displayed in the Exchange-Browser).
Am I doing something wrong or is this not the way it was intended?

Thanks for any help!

[Mildur]

Hi Thomas

No, splitting VBKs in a backup job is not possible.
May I ask which storage only allows for 3TB files?
With our recommended storage approaches (windows repo with reFS, Linux repo with XFS) file size shouldn't be an issue.

Best,
Fabian

[t.maintz]

Hi Fabian,

we're using cephfs. The default is 1TB and we increased it to 3TB. I thought there was another option but then we think about increasing that further.
Just to understand: One backup job per database is not possible incl log truncating ?

Best regards,
Thomas

[Mildur]

Hi Thomas

Thanks for providing that information.
I checked the requirements internally.
You must protect all physical exchange nodes from this DAG with all disks. You cannot choose a single disk.
Our user guide explains how it must be configured for the two DAG types (regular/IP less). https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Please note, that backup storages should be able to store multiple full backups of your backup jobs.
Would it be possible to use your CephFS system as a Ceph S3 storage? Or is Ceph S3 not related to CephFS?
Maybe in V12 you can leverage the same storage and do direct 2 object storage backups to overcome your filesystem limitation.

Best,
Fabian

[mkaec]

How can I enable a hardened Linux repository to work as S3 storage? I've also been wanting Veeam to be able to do VBK splitting (for several years now).

[Gostev]

Hardened Linux repository does not provide an S3 interface.

However, XFS does not suffer from any sort of file size limits, so I'm not too sure why would anyone even want to introduce S3 protocol into the picture? Why add something that will definitely impact performance and scalability in a negative way AND bring the possibility of its own reliability bugs? We've been hunting one such random bug in one of the on-prem S3 storage offerings for over a month now so this is a very real pain point for me right now

[mkaec]

Because it's looking like S3 will be the only way to achieve "splitting" with Veeam. Even if XFS does not impose a size limit, there are other logistical problems that develop with excessively large files.

[HannesK]

about Ceph: it's one of the most complex systems I heard of so far. But it's used by one of our largest service providers. So it can work! From their presentation some weeks ago, they use RBD block devices with 900TB+ and they run XFS on top of it. That service provider put development resources in the Ceph project and they optimized things (I remember https://github.com/ceph/ceph/pull/44697 and https://tracker.ceph.com/issues/55324 from their presentation), so that S3 seems to become more and more usable on Ceph now / with V12. My take away from that presentation was, that one needs really good Ceph admins and in best case also developers, that can fix things in the code.

there are other logistical problems that develop with excessively large files.

which ones are that? Just in case, other new features might solve these as a side effect

[Gostev]

I'm curios about that statement too. As when talking about "logistical problems" specifically and assuming we're talking about backup data management, operating with billions of objects would seem much more challenging and error-prone comparing to with a few very large files.

[pstextractor]

Well for those wondering WHY people would want to do this, I have found an excellent reason that veamZIP should be able to split into chunks: onedrive... Onedrive only allows you to upload either 100gb or 250gb single files... i have a 1.6tb archive that i want to upload and it wont let me. Was hoping to archive old VMs to our basically unlimited cloud storage so that i wouldn't have to really deal with them anymore. This is long term storage of old machines i will likely never touch again.

But a feature that was in winrar 30 years ago apparently never made it into veeam.... The ability to split VBK files into chunks with veeamzip.

I guess i will have to install 7-zip and 7-zip the vbk file into 100gb chunks... lol

[Gostev]

Truth be told, this reason is not so excellent. Because we had this very feature in Veeam Agent for Windows (automatically chunk and upload backup files to OneDrive) for some years but discontinued it eventually due to performance and reliability issues with OneDrive with large data sets... it was a support nightmare.

[robnicholsonmalt]

My usage case was not so much cloud storage as migration.

[Gostev]

@robnicholsonmalt is this still actual though considering V12 support copying and moving backups natively? This was added specifically for migrations.

I mean, technically you can get "split VBK" for migration purposes by using some object storage (like free MinIO) as a target. Then export bucket content to a file system with some S3 client and you will get your desired millions of <1MB files. Then import this back into object storage in the target site and use Veeam to move backups to the ultimate target repository with VeeaMover.

[robnicholsonmalt]

I haven't had to do a migration in a while so will park it until that requirement pops up again. I tend to work with small companies backing up a legacy server and often the backup hardware isn't top-notch (yes I know!). They're often small enough to get away with the community edition which is absolutely superb so can't really complain

If the migration is hardened against drop outs and designed to work over less than ideal connections, then it'll be fine. I'm thinking along the lines of "Okay we need to transfer a single 5TB file. Let's do it in 1GB chunks, committing the writes each time, maybe check summing the last 1GB. If the task fails for whatever reason, one can restart the task and carry on". In migration situations, time isn't usually the most important constraint. Could take several days to copy that 5TB VBK as long as it's a consistent copy.

The original poster 8 years ago was more interested in leveraging Windows server DFSR duplication I guess or even good old rsync. For this, smaller chunks are more attractive.