feature request: split vbk

Availability for the Always-On Enterprise

Re: Can Veeam be configured to use maximum file size?

Veeam Logoby jmmarton » Fri Dec 23, 2016 3:13 pm

This is not currently possible, sorry. But a question, for your example 5TB backup, is that a single VM or multiple VMs? If it's multiple VMs creating a single large file then it sounds like you aren't using "per-VM backup files." This repository setting could help.

Joe
jmmarton
Veeam Software
 
Posts: 727
Liked: 86 times
Joined: Tue Nov 17, 2015 2:38 am
Location: Chicago, IL
Full Name: Joe Marton

Re: feature request: split vbk

Veeam Logoby Seve CH » Tue Jan 03, 2017 9:42 am 3 people like this post

Dear All,

+1 to VM split by size (i.e. files up to 1TB big) and/or (at least) per VM disk.

Why?

Limits:
  • I have just arrived from holidays and I've found out that my new year's full copy job failed. We use a bunch of 8TB USB3 disks in a scaleout repo to store an offsite 1st of Year full backup of some VMs. Even if I had several TBs free, one big server was unable to copy because it didn't fit (VBK file close to 7.5TB). Now I'm playing with high compression settings and we will see what happens (crossing fingers). I doubt I can (nor should) try to use spanned volumes on USB disks.
  • W2016 dedup with files +1TB in size is a "non supported" scenario (or at least "not recommended", with in corporate, it's almost the same). We still use W2012R2 with VBR 9, but if we move to 9.5 it is very possible we will migrate also to 2016 to use ReFS for some jobs.

Performance:
  • "A single backup file is written in a single thread into the underlying storage." Our storage is able to use parallelism and works at its best with long queues. So when the job starts with many VMs, it achieves 2-3GBytes/s sustained rate. But the last 2-3 VMs (bigger than 6TB) take hours to complete (130MB/s). Our backup storage has more than 200 disks waiting for something to do (IBM XIV arrays, 7200RPM disks) but with 1 thread we are only using one disk each time.
  • Multicore. Compressing/deduping files by Veeam only uses one core per job. We planned our servers/proxies with fast cores (3.4Ghz), but it would be nice to use the other 11 ones ;)
  • Windows deduplication. Big files are slow to process, to garbage collect and also stay on the same volume (no multithreading).
  • Copy/move backup files from one volume to another. Sometimes it is needed. Again, one file, one thread: impossible to leverage our storage performance on parallel loads.

Now that we are in January, it is too late to ask Santa for these features. That's why I'm asking Mr. Gostev instead :)

EDIT: Hmm... another nice feature would be a setting to "process big VMs first". Generally, but not always, it is better to start with big VMs as soon as possible because they will be the last to finish.

Happy new year to you all.

Regards.
Seve CH
Influencer
 
Posts: 10
Liked: 6 times
Joined: Mon May 09, 2016 2:34 pm
Location: Switzerland
Full Name: JM Severino

Re: feature request: split vbk

Veeam Logoby Martin Damgaard » Tue Jan 24, 2017 1:23 pm 1 person likes this post

+1 to splitting large .VBK files option!
(I lost a lot of GFS archives because of Windows Server 2016 decuplication bug)

And i would also like so see an option to process big VMs before small ones, like Seve CH mentions - for all the same reasons...

- Martin
Martin Damgaard
Influencer
 
Posts: 14
Liked: 7 times
Joined: Mon Aug 31, 2015 6:31 am
Full Name: Martin Damgaard

Re: feature request: split vbk

Veeam Logoby mkaec » Tue Jan 24, 2017 4:57 pm

On our main file server, I split out the data into 6 volumes with one of the criteria being importance. That would allow me to restore the most important volumes first in a DR situation, bring the VM up and then handle the less critical stuff while users are working.
mkaec
Expert
 
Posts: 172
Liked: 43 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: feature request: split vbk

Veeam Logoby Delo123 » Mon Jan 30, 2017 9:22 am

Agree, but splitting up in volumes makes windows dedup function within guests less effective...
Delo123
Expert
 
Posts: 330
Liked: 92 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: feature request: split vbk

Veeam Logoby mkaec » Mon Jan 30, 2017 2:16 pm

I've found that Veeam and NTFS dedup tend not to play well together in my environment. NTFS dedup garbage collection will occasionally cause Veeam to produce very large incrementals. I think that would be true of any block-based backup application. I did find that if your repository is also running NTFS dedup, the large incrementals will not be a problem. But, that's not what I'm using for a repository. So, I ended up turning off NTFS dedup in order to maintain decent retention.
mkaec
Expert
 
Posts: 172
Liked: 43 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: feature request: split vbk

Veeam Logoby glamic26 » Mon Jan 30, 2017 2:40 pm 2 people like this post

+1 for splitting large .VBK, and also .VRB and .VIB (as we have some large incrementals as well), at the data processing level for the same parallel processing reasons among others that Seve CH set out.
glamic26
Service Provider
 
Posts: 15
Liked: 9 times
Joined: Tue Apr 21, 2015 12:10 pm

Re: feature request: split vbk

Veeam Logoby DonZoomik » Mon Jan 30, 2017 3:00 pm 1 person likes this post

Or as I suggested in another thread:
As we already have per-VM chains, how about further splitting to per-disk chains? For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitrary boundaries is too difficult, this could be much easier (extending existing architecture) and fulfil the requirements of at least some users.
DonZoomik
Influencer
 
Posts: 16
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: feature request: split vbk

Veeam Logoby andyg » Wed Feb 01, 2017 5:21 pm 2 people like this post

+1 to splitting large .VBK files option!
-= VMCE v9 certified =-
andyg
Service Provider
 
Posts: 54
Liked: 4 times
Joined: Wed Apr 23, 2014 9:51 am
Full Name: Andy Goldschmidt

[MERGED] FEATURE REQUEST: File segmentation.

Veeam Logoby jeffmorlen » Mon Mar 27, 2017 8:23 pm 2 people like this post

I'm formally requesting a feature request to be added to the Veeam Backup & Replication suite of products.

The ability to break a back file (VBK) into many small files during the backup process.

This process has been available in other backup software for years (usually with default media size defaults; 4.7GB, etc).

This ability would assist in moving/replicating/copying data from your backup vault to a 3rd party resource or cloud provider (via BLOB storage).

Thanks, in advance.
jeffmorlen
Lurker
 
Posts: 1
Liked: 2 times
Joined: Mon Mar 27, 2017 8:17 pm
Full Name: Jeff Morlen

Re: FEATURE REQUEST: File segmentation.

Veeam Logoby DGrinev » Tue Mar 28, 2017 11:00 am

Hi Jeff and welcome to the community!

You can use per-VM backup chains.
Please, review this thread for detailed information.

Thanks!
DGrinev
Veeam Software
 
Posts: 233
Liked: 31 times
Joined: Thu Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev

Re: feature request: split vbk

Veeam Logoby crackocain » Tue Mar 28, 2017 1:09 pm 2 people like this post

+1

Per VM disk

Great and critical feature update i guess... VM disk sizes are constantly rising. Every customer has minimum one or two greater than 1 TB VM. Sometimes more.
EMC, IBM Storage Specialist - VMCEv9 - Dosbil - Turkey
crackocain
Veeam ProPartner
 
Posts: 47
Liked: 7 times
Joined: Mon Dec 14, 2015 8:20 pm
Location: Turkey
Full Name: Mehmet Istanbullu

Re: feature request: split vbk

Veeam Logoby SyNtAxx » Tue Mar 28, 2017 2:35 pm 1 person likes this post

+1 I have vms that have drives 8tb and larger.


-Nick
SyNtAxx
Expert
 
Posts: 125
Liked: 13 times
Joined: Fri Jan 02, 2015 7:12 pm

[MERGED] Scale Out Backup Repository feature request

Veeam Logoby TedLaurent » Fri May 26, 2017 1:16 pm

I'm not sure if this has been suggested previously - if so please disregard.

When backing up to a scale out repository if backing up a drive that contains more data then the individual extents the backup fails.

This make sense but is not necessarily desirable. It would be better for the backup to be stripped across the extents so that the limitation would be the amount of aggregate storage available. For example, if I am trying to backup a data drive that has 1 TB of data and I have a scale out repository that has 2 repositories with 800 GB each, then backup ideally would not fail. :D

Ted
TedLaurent
Novice
 
Posts: 3
Liked: never
Joined: Wed Jan 18, 2017 11:48 pm
Full Name: Ted Laurent

Re: feature request: split vbk

Veeam Logoby foggy » Fri May 26, 2017 5:51 pm

Hi Ted, I've merged your post into this thread, since your request basically means the ability to split backup files. Thanks.
foggy
Veeam Software
 
Posts: 14346
Liked: 1054 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Inferno131, jim3cantos and 40 guests