Comprehensive data protection for all workloads
jmmarton
Veeam Software
Posts: 1564
Liked: 194 times
Joined: Nov 17, 2015 2:38 am
Full Name: Joe Marton
Location: Chicago, IL
Contact:

Re: Can Veeam be configured to use maximum file size?

Post by jmmarton » Dec 23, 2016 3:13 pm

This is not currently possible, sorry. But a question, for your example 5TB backup, is that a single VM or multiple VMs? If it's multiple VMs creating a single large file then it sounds like you aren't using "per-VM backup files." This repository setting could help.

Joe

Seve CH
Influencer
Posts: 18
Liked: 9 times
Joined: May 09, 2016 2:34 pm
Full Name: JM Severino
Location: Switzerland
Contact:

Re: feature request: split vbk

Post by Seve CH » Jan 03, 2017 9:42 am 3 people like this post

Dear All,

+1 to VM split by size (i.e. files up to 1TB big) and/or (at least) per VM disk.

Why?

Limits:
  • I have just arrived from holidays and I've found out that my new year's full copy job failed. We use a bunch of 8TB USB3 disks in a scaleout repo to store an offsite 1st of Year full backup of some VMs. Even if I had several TBs free, one big server was unable to copy because it didn't fit (VBK file close to 7.5TB). Now I'm playing with high compression settings and we will see what happens (crossing fingers). I doubt I can (nor should) try to use spanned volumes on USB disks.
  • W2016 dedup with files +1TB in size is a "non supported" scenario (or at least "not recommended", with in corporate, it's almost the same). We still use W2012R2 with VBR 9, but if we move to 9.5 it is very possible we will migrate also to 2016 to use ReFS for some jobs.
Performance:
  • "A single backup file is written in a single thread into the underlying storage." Our storage is able to use parallelism and works at its best with long queues. So when the job starts with many VMs, it achieves 2-3GBytes/s sustained rate. But the last 2-3 VMs (bigger than 6TB) take hours to complete (130MB/s). Our backup storage has more than 200 disks waiting for something to do (IBM XIV arrays, 7200RPM disks) but with 1 thread we are only using one disk each time.
  • Multicore. Compressing/deduping files by Veeam only uses one core per job. We planned our servers/proxies with fast cores (3.4Ghz), but it would be nice to use the other 11 ones ;)
  • Windows deduplication. Big files are slow to process, to garbage collect and also stay on the same volume (no multithreading).
  • Copy/move backup files from one volume to another. Sometimes it is needed. Again, one file, one thread: impossible to leverage our storage performance on parallel loads.
Now that we are in January, it is too late to ask Santa for these features. That's why I'm asking Mr. Gostev instead :)

EDIT: Hmm... another nice feature would be a setting to "process big VMs first". Generally, but not always, it is better to start with big VMs as soon as possible because they will be the last to finish.

Happy new year to you all.

Regards.

Martin Damgaard
Influencer
Posts: 16
Liked: 7 times
Joined: Aug 31, 2015 6:31 am
Full Name: Martin Damgaard
Contact:

Re: feature request: split vbk

Post by Martin Damgaard » Jan 24, 2017 1:23 pm 1 person likes this post

+1 to splitting large .VBK files option!
(I lost a lot of GFS archives because of Windows Server 2016 decuplication bug)

And i would also like so see an option to process big VMs before small ones, like Seve CH mentions - for all the same reasons...

- Martin

mkaec
Expert
Posts: 318
Liked: 72 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: feature request: split vbk

Post by mkaec » Jan 24, 2017 4:57 pm

On our main file server, I split out the data into 6 volumes with one of the criteria being importance. That would allow me to restore the most important volumes first in a DR situation, bring the VM up and then handle the less critical stuff while users are working.

Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: feature request: split vbk

Post by Delo123 » Jan 30, 2017 9:22 am

Agree, but splitting up in volumes makes windows dedup function within guests less effective...

mkaec
Expert
Posts: 318
Liked: 72 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: feature request: split vbk

Post by mkaec » Jan 30, 2017 2:16 pm

I've found that Veeam and NTFS dedup tend not to play well together in my environment. NTFS dedup garbage collection will occasionally cause Veeam to produce very large incrementals. I think that would be true of any block-based backup application. I did find that if your repository is also running NTFS dedup, the large incrementals will not be a problem. But, that's not what I'm using for a repository. So, I ended up turning off NTFS dedup in order to maintain decent retention.

glamic26
Service Provider
Posts: 21
Liked: 9 times
Joined: Apr 21, 2015 12:10 pm
Contact:

Re: feature request: split vbk

Post by glamic26 » Jan 30, 2017 2:40 pm 2 people like this post

+1 for splitting large .VBK, and also .VRB and .VIB (as we have some large incrementals as well), at the data processing level for the same parallel processing reasons among others that Seve CH set out.

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: feature request: split vbk

Post by DonZoomik » Jan 30, 2017 3:00 pm 1 person likes this post

Or as I suggested in another thread:
As we already have per-VM chains, how about further splitting to per-disk chains? For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitrary boundaries is too difficult, this could be much easier (extending existing architecture) and fulfil the requirements of at least some users.

andyg
Service Provider
Posts: 58
Liked: 5 times
Joined: Apr 23, 2014 9:51 am
Full Name: Andy Goldschmidt
Contact:

Re: feature request: split vbk

Post by andyg » Feb 01, 2017 5:21 pm 2 people like this post

+1 to splitting large .VBK files option!
-= VMCE v9 certified =-

jeffmorlen
Lurker
Posts: 1
Liked: 2 times
Joined: Mar 27, 2017 8:17 pm
Full Name: Jeff Morlen
Contact:

[MERGED] FEATURE REQUEST: File segmentation.

Post by jeffmorlen » Mar 27, 2017 8:23 pm 2 people like this post

I'm formally requesting a feature request to be added to the Veeam Backup & Replication suite of products.

The ability to break a back file (VBK) into many small files during the backup process.

This process has been available in other backup software for years (usually with default media size defaults; 4.7GB, etc).

This ability would assist in moving/replicating/copying data from your backup vault to a 3rd party resource or cloud provider (via BLOB storage).

Thanks, in advance.

DGrinev
Veeam Software
Posts: 1639
Liked: 199 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: FEATURE REQUEST: File segmentation.

Post by DGrinev » Mar 28, 2017 11:00 am

Hi Jeff and welcome to the community!

You can use per-VM backup chains.
Please, review this thread for detailed information.

Thanks!

crackocain
Service Provider
Posts: 120
Liked: 9 times
Joined: Dec 14, 2015 8:20 pm
Full Name: Mehmet Istanbullu
Location: Turkey
Contact:

Re: feature request: split vbk

Post by crackocain » Mar 28, 2017 1:09 pm 2 people like this post

+1

Per VM disk

Great and critical feature update i guess... VM disk sizes are constantly rising. Every customer has minimum one or two greater than 1 TB VM. Sometimes more.
IBM, EMC, Huawei Storage Specialist - VMCEv9 - Dosbil - Turkey

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: feature request: split vbk

Post by SyNtAxx » Mar 28, 2017 2:35 pm 1 person likes this post

+1 I have vms that have drives 8tb and larger.


-Nick

TedLaurent
Service Provider
Posts: 3
Liked: never
Joined: Jan 18, 2017 11:48 pm
Full Name: Ted Laurent
Contact:

[MERGED] Scale Out Backup Repository feature request

Post by TedLaurent » May 26, 2017 1:16 pm

I'm not sure if this has been suggested previously - if so please disregard.

When backing up to a scale out repository if backing up a drive that contains more data then the individual extents the backup fails.

This make sense but is not necessarily desirable. It would be better for the backup to be stripped across the extents so that the limitation would be the amount of aggregate storage available. For example, if I am trying to backup a data drive that has 1 TB of data and I have a scale out repository that has 2 repositories with 800 GB each, then backup ideally would not fail. :D

Ted

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: feature request: split vbk

Post by foggy » May 26, 2017 5:51 pm

Hi Ted, I've merged your post into this thread, since your request basically means the ability to split backup files. Thanks.

haslund
Veeam Software
Posts: 521
Liked: 94 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: feature request: split vbk

Post by haslund » May 28, 2017 10:53 am

SyNtAxx wrote:I would also like the ability to chop up .vbk files (I use per vm chains) into small files. I have vms that are multiple tbs in size, and this causes issues for replication of those files. While the per vm chains greatly reduced the issue of not all files being replicate din a timely manner, I still have those massive vms still causing issues. It would be nice to split the .vbk files for those large vms into smaller more managable bites (pardon pun) in order to replicate more easily. Example, a setting in the backup job to split backup files larger than 1tb into some user configurable sub grouping, for example 250gb. This is a good idea.

-Nick
Can you elaborate a bit more on what you mean by replicating backup files? If you utilize a Veeam Backup Copy job to move backups off-site we will only move changed blocks, not actual files which might solve your issue if you are using something different than Veeam to replicate/copy backup files.
Rasmus Haslund
Principal Technologist, Global Education Services @ Veeam Software
Veeam Certified Architect #1 | Veeam Certified Trainer #4 [v7,v8,v9] | Veeam Certified Trainer Mentor #1
Twitter: @haslund
Blog: www.perfectcloud.org

final
Enthusiast
Posts: 32
Liked: 11 times
Joined: Aug 14, 2016 7:19 pm
Contact:

Re: feature request: split vbk

Post by final » Jun 27, 2017 9:56 am 1 person likes this post

+1 for splitting VBKs. Windows Server 2016 deduplication only processes the first 4GB of a file (this is a fixed limit of Server 2016 deduplication according to Will Gries, the technology owner at MS). We have a single VM with a total size of 10TB and hence cannot really use deduplication on this.

ERecker
Lurker
Posts: 1
Liked: 1 time
Joined: Aug 18, 2017 7:52 am
Full Name: Enno Recker
Contact:

Re: feature request: split vbk

Post by ERecker » Aug 18, 2017 8:16 am 1 person likes this post

+1 for splitting VBKs

We also have a file server with individual volumes larger than 4TB. We use windows dedup and are about to upgrade archive servers to windows server 2016. It is already stated, that windows server 2016 stops dedup after processing the first 4TB of an individual file.
This Feature should be also usable within copy Jobs.

regards

fika
Influencer
Posts: 15
Liked: 1 time
Joined: Sep 20, 2017 8:10 am
Full Name: fika
Contact:

Re: feature request: split vbk

Post by fika » Sep 20, 2017 8:15 am 1 person likes this post

One more +1 for splitting VBKs !
It would really help syncing operations, some storage locations are also limited in file size and/or perform badly with huge large files.

There are many threads already voting for this :
veeam-agent-for-windows-f33/feature-req ... 35133.html
veeam-backup-replication-f2/feature-req ... 27833.html
veeam-backup-replication-f2/ability-to- ... 25340.html
veeam-backup-replication-f2/vbk-file-si ... t7895.html
veeam-backup-replication-f2/is-there-a- ... 37204.html
veeam-agent-for-windows-f33/split-vbk-f ... 25117.html
veeam-agent-for-windows-f33/any-way-to- ... 32186.html
...

I then think this feature should really be considered :wink:

Thank you to Veeam support !

ayates
Lurker
Posts: 2
Liked: 1 time
Joined: Sep 29, 2017 11:05 am
Full Name: Anthony Yates
Contact:

[MERGED] Feature Request - Split VBK to improve tape backup

Post by ayates » Sep 29, 2017 11:23 am 1 person likes this post

Hi,

Is there any plans to have option to split VBK files?
Reason I ask is because of tape backups and parallel processing.

At the moment we can enable tape parallel processing and it will write our per VM backups to both tape drives at the same time.
However lets say you have lots of average size VMs and one really big VM that is a few TB in size and bigger than the combined total of the smaller VMs.
What happens then is for half the time the tape drives are working in parallel because the small VMs go to one drive and the big VM goes to the other.
But once the smaller VMs have completed you are then left with the big VM writing to one tape drive only and not making use of the second tape drive.
I assume the reason is because it's one big VBK file and it can only write that one file to one tape drive at a time.

If it did something similar to say CommVault where it splits the files into chunks then it could write different chunks to different tape drives at the same time and have both tape drives working in parallel nearly 100% of the time.

Cheers,

Anthony

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: feature request: split vbk

Post by foggy » Oct 01, 2017 9:58 pm 1 person likes this post

Hi Anthony, thanks for your request. I'm merging your thread into another existing similar discussion.

DonZoomik
Expert
Posts: 109
Liked: 26 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: feature request: split vbk

Post by DonZoomik » Oct 20, 2017 8:48 am 1 person likes this post

No news in upcoming 9.5 Update 3?

veremin
Product Manager
Posts: 16491
Liked: 1371 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: feature request: split vbk

Post by veremin » Oct 20, 2017 8:59 am 1 person likes this post

Unfortunately, Update 3 will not have this functionality implemented - priority has been given to other (more demanded) features. Thanks.

Xamber
Lurker
Posts: 1
Liked: never
Joined: Nov 10, 2017 7:34 am
Contact:

[MERGED] [Request] Split backup files to smaller archives

Post by Xamber » Nov 10, 2017 8:14 am

Hi,

Spoke with some guy from the support on the phone and he told me that i should make a forum post regarding my request.

We would like to at least have the possibility to split a large backup job into smaller files meanwhile the backup job is running.

I see two possible ways in doing so, either split the files up per machine backed up. So that if you have 10 VM's they're split up by each separate VM.
Second idea would be to split the large files into smaller archive files. Just like some compression software like winrar, backupfile.r00, backupfile.r01 and so on.

The reason for asking this is, we've got about 30 machines setup for backing up each day, and take a full backup every two weeks. The total space allocation is about 26TB in total and in order to speed up the backup we're pushing all data over to a NAS that has an 1.2TB cache NVME drive, but when the full backup is running there is not enough space on the NVME drive and the backup will fail due to the one single large 26TB backup file.

I don't think that we're the only ones the would benefit from this feature.

Thanks for reading and i hope this is somewhat coherent.

P.Tide
Product Manager
Posts: 5120
Liked: 443 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: feature request: split vbk

Post by P.Tide » Nov 10, 2017 8:31 am

Hi,

Thank you for the feature request. You're right, it's been already requested several times, so I'm merging your topic to the existing thread. Btw, have you tried per-VM backup files feature?

Thanks

Delo123
Expert
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: feature request: split vbk

Post by Delo123 » Nov 13, 2017 10:43 am

Nice request. We would also like to have split VBK's. Yes, we habe, Per-VM has no use with big VM's....

acatic
Influencer
Posts: 15
Liked: 1 time
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: feature request: split vbk

Post by acatic » Dec 22, 2017 2:04 pm

+1. Single large file server. Split by size or by vdisks.

oyvindo
Novice
Posts: 8
Liked: 1 time
Joined: Aug 11, 2016 10:15 pm
Full Name: Oyvind Overby
Contact:

[MERGED] [Feature Request]-Split vbk into manageable size ch

Post by oyvindo » May 10, 2018 3:57 pm

Today, even the average notebook comes with an SSD of 512Gb or even 1 TB as standard. Desktop users probably have several drives totaling many Terrabytes
Backing up such huge disks as a single vbk file is risky, to say the least, especially if the objective is to make reliable backups, often over WLAN links and ultimately to internet cloud Storage over WAN.
Therefor, I would expect Veeam to offer simple and user-friendly solution that are able to break the vbk file into smaller manageable chunks that allows for successful retries in case of line drops or time-outs.

In my case, it takes me more that 24 hours to copy the vbk file – which for me renders Veeam useless as my cloud provider rejects a single session spanning more than 24 hours.

Viking

madmadmaddog
Lurker
Posts: 2
Liked: never
Joined: Dec 13, 2017 10:10 pm
Full Name: Matt
Contact:

[MERGED] Splitting 5 to 10TB vbk files

Post by madmadmaddog » May 18, 2018 3:35 pm

I’m wondering if anyone knows of a tool that would split a large 5 to 10TB vbk file into 20 or less pieces? Please not looking to change Veeam configuration, been there and done that. These files are already exist and I’m stuck with them for years, due a mandatory retention policy. I want to get them off my Data Domain and into Azure cold storage which has a limit of 4.7TB per file/blob.

All the utilities I’ve tried are a decade old and designed to split GB files to MB chunks.
I’ve tried and rejected:
7z – largest file piece is only 22GB
Gsplit – too old not designed for TB files
Hjsplit – too old not designed for TB files
Sfsplit – too old not designed for TB files
Looking at linux “split”, but again all the information I can find is large GB to to smaller GB at best.

Thanks,

ferrus
Veeam ProPartner
Posts: 236
Liked: 31 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: feature request: split vbk

Post by ferrus » Aug 07, 2018 11:17 am

+1 to the tally for this request. Specifically because of this limitation I've just run into ...
Veeam Backup & Replication does not split one backup file across multiple extents.
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Have a single VM which creates a weekly 11TB vbk file. It's quite difficult to juggle storage when each extent I add, has to have the full space free.

Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests