feature request: split vbk

[jmmarton]

This is not currently possible, sorry. But a question, for your example 5TB backup, is that a single VM or multiple VMs? If it's multiple VMs creating a single large file then it sounds like you aren't using "per-VM backup files." This repository setting could help.

Joe

[Seve CH]

Dear All,

+1 to VM split by size (i.e. files up to 1TB big) and/or (at least) per VM disk.

Why?

Limits:

• I have just arrived from holidays and I've found out that my new year's full copy job failed. We use a bunch of 8TB USB3 disks in a scaleout repo to store an offsite 1st of Year full backup of some VMs. Even if I had several TBs free, one big server was unable to copy because it didn't fit (VBK file close to 7.5TB). Now I'm playing with high compression settings and we will see what happens (crossing fingers). I doubt I can (nor should) try to use spanned volumes on USB disks.
• W2016 dedup with files +1TB in size is a "non supported" scenario (or at least "not recommended", with in corporate, it's almost the same). We still use W2012R2 with VBR 9, but if we move to 9.5 it is very possible we will migrate also to 2016 to use ReFS for some jobs.

Performance:

• "A single backup file is written in a single thread into the underlying storage." Our storage is able to use parallelism and works at its best with long queues. So when the job starts with many VMs, it achieves 2-3GBytes/s sustained rate. But the last 2-3 VMs (bigger than 6TB) take hours to complete (130MB/s). Our backup storage has more than 200 disks waiting for something to do (IBM XIV arrays, 7200RPM disks) but with 1 thread we are only using one disk each time.
• Multicore. Compressing/deduping files by Veeam only uses one core per job. We planned our servers/proxies with fast cores (3.4Ghz), but it would be nice to use the other 11 ones
• Windows deduplication. Big files are slow to process, to garbage collect and also stay on the same volume (no multithreading).
• Copy/move backup files from one volume to another. Sometimes it is needed. Again, one file, one thread: impossible to leverage our storage performance on parallel loads.

Now that we are in January, it is too late to ask Santa for these features. That's why I'm asking Mr. Gostev instead

EDIT: Hmm... another nice feature would be a setting to "process big VMs first". Generally, but not always, it is better to start with big VMs as soon as possible because they will be the last to finish.

Happy new year to you all.

Regards.

[Martin Damgaard]

+1 to splitting large .VBK files option!
(I lost a lot of GFS archives because of Windows Server 2016 decuplication bug)

And i would also like so see an option to process big VMs before small ones, like Seve CH mentions - for all the same reasons...

- Martin

[mkaec]

On our main file server, I split out the data into 6 volumes with one of the criteria being importance. That would allow me to restore the most important volumes first in a DR situation, bring the VM up and then handle the less critical stuff while users are working.

[Delo123]

Agree, but splitting up in volumes makes windows dedup function within guests less effective...

[mkaec]

I've found that Veeam and NTFS dedup tend not to play well together in my environment. NTFS dedup garbage collection will occasionally cause Veeam to produce very large incrementals. I think that would be true of any block-based backup application. I did find that if your repository is also running NTFS dedup, the large incrementals will not be a problem. But, that's not what I'm using for a repository. So, I ended up turning off NTFS dedup in order to maintain decent retention.

[glamic26]

+1 for splitting large .VBK, and also .VRB and .VIB (as we have some large incrementals as well), at the data processing level for the same parallel processing reasons among others that Seve CH set out.

[DonZoomik]

Or as I suggested in another thread:
As we already have per-VM chains, how about further splitting to per-disk chains? For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitrary boundaries is too difficult, this could be much easier (extending existing architecture) and fulfil the requirements of at least some users.

[andyg]

+1 to splitting large .VBK files option!

[jeffmorlen]

I'm formally requesting a feature request to be added to the Veeam Backup & Replication suite of products.

The ability to break a back file (VBK) into many small files during the backup process.

This process has been available in other backup software for years (usually with default media size defaults; 4.7GB, etc).

This ability would assist in moving/replicating/copying data from your backup vault to a 3rd party resource or cloud provider (via BLOB storage).

Thanks, in advance.

[DGrinev]

Hi Jeff and welcome to the community!

You can use per-VM backup chains.
Please, review this thread for detailed information.

Thanks!

[crackocain]

+1

Per VM disk

Great and critical feature update i guess... VM disk sizes are constantly rising. Every customer has minimum one or two greater than 1 TB VM. Sometimes more.

[SyNtAxx]

+1 I have vms that have drives 8tb and larger.


-Nick

[TedLaurent]

I'm not sure if this has been suggested previously - if so please disregard.

When backing up to a scale out repository if backing up a drive that contains more data then the individual extents the backup fails.

This make sense but is not necessarily desirable. It would be better for the backup to be stripped across the extents so that the limitation would be the amount of aggregate storage available. For example, if I am trying to backup a data drive that has 1 TB of data and I have a scale out repository that has 2 repositories with 800 GB each, then backup ideally would not fail.

Ted

[foggy]

Hi Ted, I've merged your post into this thread, since your request basically means the ability to split backup files. Thanks.

[haslund]

I would also like the ability to chop up .vbk files (I use per vm chains) into small files. I have vms that are multiple tbs in size, and this causes issues for replication of those files. While the per vm chains greatly reduced the issue of not all files being replicate din a timely manner, I still have those massive vms still causing issues. It would be nice to split the .vbk files for those large vms into smaller more managable bites (pardon pun) in order to replicate more easily. Example, a setting in the backup job to split backup files larger than 1tb into some user configurable sub grouping, for example 250gb. This is a good idea.

-Nick

Can you elaborate a bit more on what you mean by replicating backup files? If you utilize a Veeam Backup Copy job to move backups off-site we will only move changed blocks, not actual files which might solve your issue if you are using something different than Veeam to replicate/copy backup files.

[final]

+1 for splitting VBKs. Windows Server 2016 deduplication only processes the first 4GB of a file (this is a fixed limit of Server 2016 deduplication according to Will Gries, the technology owner at MS). We have a single VM with a total size of 10TB and hence cannot really use deduplication on this.

[ERecker]

+1 for splitting VBKs

We also have a file server with individual volumes larger than 4TB. We use windows dedup and are about to upgrade archive servers to windows server 2016. It is already stated, that windows server 2016 stops dedup after processing the first 4TB of an individual file.
This Feature should be also usable within copy Jobs.

regards

[fika]

One more +1 for splitting VBKs !
It would really help syncing operations, some storage locations are also limited in file size and/or perform badly with huge large files.

There are many threads already voting for this :
veeam-agent-for-windows-f33/feature-req ... 35133.html
veeam-backup-replication-f2/feature-req ... 27833.html
veeam-backup-replication-f2/ability-to- ... 25340.html
veeam-backup-replication-f2/vbk-file-si ... t7895.html
veeam-backup-replication-f2/is-there-a- ... 37204.html
veeam-agent-for-windows-f33/split-vbk-f ... 25117.html
veeam-agent-for-windows-f33/any-way-to- ... 32186.html
...

I then think this feature should really be considered

Thank you to Veeam support !

[ayates]

Hi,

Is there any plans to have option to split VBK files?
Reason I ask is because of tape backups and parallel processing.

At the moment we can enable tape parallel processing and it will write our per VM backups to both tape drives at the same time.
However lets say you have lots of average size VMs and one really big VM that is a few TB in size and bigger than the combined total of the smaller VMs.
What happens then is for half the time the tape drives are working in parallel because the small VMs go to one drive and the big VM goes to the other.
But once the smaller VMs have completed you are then left with the big VM writing to one tape drive only and not making use of the second tape drive.
I assume the reason is because it's one big VBK file and it can only write that one file to one tape drive at a time.

If it did something similar to say CommVault where it splits the files into chunks then it could write different chunks to different tape drives at the same time and have both tape drives working in parallel nearly 100% of the time.

Cheers,

Anthony

[foggy]

Hi Anthony, thanks for your request. I'm merging your thread into another existing similar discussion.

[DonZoomik]

No news in upcoming 9.5 Update 3?

[veremin]

Unfortunately, Update 3 will not have this functionality implemented - priority has been given to other (more demanded) features. Thanks.

[Xamber]

Hi,

Spoke with some guy from the support on the phone and he told me that i should make a forum post regarding my request.

We would like to at least have the possibility to split a large backup job into smaller files meanwhile the backup job is running.

I see two possible ways in doing so, either split the files up per machine backed up. So that if you have 10 VM's they're split up by each separate VM.
Second idea would be to split the large files into smaller archive files. Just like some compression software like winrar, backupfile.r00, backupfile.r01 and so on.

The reason for asking this is, we've got about 30 machines setup for backing up each day, and take a full backup every two weeks. The total space allocation is about 26TB in total and in order to speed up the backup we're pushing all data over to a NAS that has an 1.2TB cache NVME drive, but when the full backup is running there is not enough space on the NVME drive and the backup will fail due to the one single large 26TB backup file.

I don't think that we're the only ones the would benefit from this feature.

Thanks for reading and i hope this is somewhat coherent.

[PTide]

Hi,

Thank you for the feature request. You're right, it's been already requested several times, so I'm merging your topic to the existing thread. Btw, have you tried per-VM backup files feature?

Thanks

[Delo123]

Nice request. We would also like to have split VBK's. Yes, we habe, Per-VM has no use with big VM's....

[acatic]

+1. Single large file server. Split by size or by vdisks.

[oyvindo]

Today, even the average notebook comes with an SSD of 512Gb or even 1 TB as standard. Desktop users probably have several drives totaling many Terrabytes
Backing up such huge disks as a single vbk file is risky, to say the least, especially if the objective is to make reliable backups, often over WLAN links and ultimately to internet cloud Storage over WAN.
Therefor, I would expect Veeam to offer simple and user-friendly solution that are able to break the vbk file into smaller manageable chunks that allows for successful retries in case of line drops or time-outs.

In my case, it takes me more that 24 hours to copy the vbk file – which for me renders Veeam useless as my cloud provider rejects a single session spanning more than 24 hours.

Viking

[madmadmaddog]

I’m wondering if anyone knows of a tool that would split a large 5 to 10TB vbk file into 20 or less pieces? Please not looking to change Veeam configuration, been there and done that. These files are already exist and I’m stuck with them for years, due a mandatory retention policy. I want to get them off my Data Domain and into Azure cold storage which has a limit of 4.7TB per file/blob.

All the utilities I’ve tried are a decade old and designed to split GB files to MB chunks.
I’ve tried and rejected:
7z – largest file piece is only 22GB
Gsplit – too old not designed for TB files
Hjsplit – too old not designed for TB files
Sfsplit – too old not designed for TB files
Looking at linux “split”, but again all the information I can find is large GB to to smaller GB at best.

Thanks,

[ferrus]

+1 to the tally for this request. Specifically because of this limitation I've just run into ...

Veeam Backup & Replication does not split one backup file across multiple extents.

https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Have a single VM which creates a weekly 11TB vbk file. It's quite difficult to juggle storage when each extent I add, has to have the full space free.