feature request: split vbk

Availability for the Always-On Enterprise

Re: feature request: split vbk

Veeam Logoby gosnells » Tue May 31, 2016 4:07 am

Hello,

+1 on this feature request
I too would be keen on being able to split large multi TB backup files. Several servers here with 2TB+ data drives.
gosnells
Novice
 
Posts: 8
Liked: 1 time
Joined: Mon Jul 23, 2012 4:28 am
Full Name: Mike Smith

Re: feature request: split vbk

Veeam Logoby hyvokar » Tue Sep 20, 2016 12:19 pm

hyvokar wrote:
I did a test using a 3 medium sized VMs. I created two identical backup copy jobs and targetted them to different repositories, one "normal" and one using per-vm backup files.
The results was a bit surprising. The one with per-vm backup files was *smaller* than one with single huge backup file. No idea how that's possible.

Another test in production environment. The compacted backup file size was 1.48TB. After enabling the per-vm backup files, the combined size of new backup files was 1,35TB. Makes you wonder if the compact really works.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
hyvokar
Service Provider
 
Posts: 267
Liked: 16 times
Joined: Fri Nov 21, 2014 10:05 pm

Re: feature request: split vbk

Veeam Logoby futureweb » Sat Nov 05, 2016 4:37 pm

+1 here too, this feature would be good, as Dedup of large files is not possible.
We used per VM Backup Chain, but still we do have one VM with about 2,7 TB Data, that can't ever go under 1 TB.
futureweb
Enthusiast
 
Posts: 55
Liked: 5 times
Joined: Thu Sep 03, 2015 12:15 am
Full Name: Patrick

Re: feature request: split vbk

Veeam Logoby Gostev » Sat Nov 05, 2016 8:25 pm

Patrick, why do you need to go under 1TB, what is the use case here?
Gostev
Veeam Software
 
Posts: 21621
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: feature request: split vbk

Veeam Logoby futureweb » Sat Nov 05, 2016 9:01 pm

Windows can't dedup files larger than 1 TB, also not with new Windows Server 2016.
futureweb
Enthusiast
 
Posts: 55
Liked: 5 times
Joined: Thu Sep 03, 2015 12:15 am
Full Name: Patrick

Re: feature request: split vbk

Veeam Logoby PTide » Sat Nov 05, 2016 9:28 pm

Hi,

Windows can't dedup files larger than 1 TB, also not with new Windows Server 2016.
As far as I know it's not a hard limit, is it?

Thanks
PTide
Veeam Software
 
Posts: 3248
Liked: 272 times
Joined: Tue May 19, 2015 1:46 pm

Re: feature request: split vbk

Veeam Logoby futureweb » Sat Nov 05, 2016 9:41 pm

I thought so too, but it deduped any backup file but didn't the one with 2,3 TB.

And the dedup job is stopped, until new backup files enter the right age. This one is already 6 days old.
But I didn't find any evidence from microsoft, stating that is true, only that files up to 1 TB now are highly performant.

At Windows 2016, I said, the dedup type is virtual backup server as it sounded right. Didn't find much info about that either,
so was just guessing.

Thanks
Patrick
futureweb
Enthusiast
 
Posts: 55
Liked: 5 times
Joined: Thu Sep 03, 2015 12:15 am
Full Name: Patrick

Re: feature request: split vbk

Veeam Logoby futureweb » Sat Nov 05, 2016 10:56 pm

Anyway, it's never good to use Dedup, if Microsoft states, it is "unsupported".
This is, why I also request that feature, as Veeam doesn't have Dedup on it's own.

Just added more RAM to the server, to see if it's dedupping it now, seems that
it didn't make any change, will go to MS Forum and see if I get any answer there why this file is skipped.

But also found an entry about Windows 2016 and troubles with Dedup of large files:
haven't tested it myself.

https://blog.zoomik.pri.ee/uncategorize ... uge-files/
futureweb
Enthusiast
 
Posts: 55
Liked: 5 times
Joined: Thu Sep 03, 2015 12:15 am
Full Name: Patrick

Re: feature request: split vbk

Veeam Logoby Gostev » Sun Nov 06, 2016 2:28 pm

Windows Server 2016 dedupe will dedupe files up to 4TB based on our testing, and skip bigger files.
Also, 1TB limit is for changing files only, not for read-only files such as Veeam backups.

I've actually discussed this directly with the dev team behind this tech some months ago...
This blog you shared uses some information from my earlier posts on this forum.
Gostev
Veeam Software
 
Posts: 21621
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: feature request: split vbk

Veeam Logoby futureweb » Sun Nov 06, 2016 2:46 pm

thanks for your info.
well lets see what happens with the next 2,4 TB Full Backup, if that's going to be deduped.
was done yesterday, so should see results in some days. anyway, this is off topic then if you say up to 4 TB should work.

Anyway, do you recommend already using server 2016 and dedup with Veeam?
futureweb
Enthusiast
 
Posts: 55
Liked: 5 times
Joined: Thu Sep 03, 2015 12:15 am
Full Name: Patrick

Re: feature request: split vbk

Veeam Logoby Gostev » Sun Nov 06, 2016 4:01 pm

Universally, I never recommend jumping any newly released technology at all :)
When I feel an urge to do this myself, I usually start small and see how it goes.
Gostev
Veeam Software
 
Posts: 21621
Liked: 2411 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: feature request: split vbk

Veeam Logoby futureweb » Sat Nov 12, 2016 9:27 pm

Just as update, we got the 2,4 TB File to dedup, I tried a restore afterwards, it had "read errors". Some files worked (Guest files Restore) others didn't.
So bad idea using Dedup with Windows 2016 as it seems broken, at least for us.

I tested a smaller VM under 500 GB which seemed to work fine.
Will now keep using Windows 2012R2 and not using dedup, even I don't know how big files would perform with 2012R2 - never tested it on my own.
Probably 9.5 with ReFS is the better idea, after some Windows 2016 and Veeam Updates ;)

Anyway wasoff topic but wanted to give feedback.

Probably splitting files will still come anytime in the future.

Thanks
Patrick
futureweb
Enthusiast
 
Posts: 55
Liked: 5 times
Joined: Thu Sep 03, 2015 12:15 am
Full Name: Patrick

Re: feature request: split vbk

Veeam Logoby Delo123 » Mon Nov 14, 2016 7:47 am

This discussion would be totally unnecessary if Veeam would support splitting backup files we would all be able to sleep better i assume :)
I have a slight understanding of Veeam saying having 5 files is more complicated as having 1 file but i feel it's more a bit of a joke.
Even if, one could create a very secret really important hidden registry switch! No pun intended btw!
Delo123
Expert
 
Posts: 355
Liked: 102 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: feature request: split vbk

Veeam Logoby DonZoomik » Fri Nov 25, 2016 2:54 pm 3 people like this post

I am the author of abovementioned blog post.

Current dedup implementations are... mixed. Basically you have to choose and accept the trade-offs.
WS2012R2 seems to have no hard file size limit, BUT:
* General throughput is slow (single-threaded)
* If data is already in chunk store and it processes a secondary copy (for example secondary full backup), throughput is better.
* 1TB+ file processing gets slower, although it will eventually complete.
WS2016 is much faster, BUT
* 4TB data limit means huge losses for large VMs
* Data corruption issue with 1TB+ files. I haven't poked the product team for a few weeks but a hotfix was promised waaay back.

For maximum total savings you'd go for 64TB volumes but on WS2012R2 it would probably never finish processing if new data keeps pouring in. My experiences shows that on WS2012R2 in realistic scenarios it'd probably take at least a week to process (assuming no new data was written). Actually I'd say that WS2012R2 is even better in these scenarios (you have huge files and goal is maximum data savings).
Either way 1TB is supported maximum. Though product team also confirmed that in Forward Incremental style scenario (write-once, never modify) NTFS dedup will work fine past supported limits.

I'd happily foreit Veeam dedup and compression if we'd get (for example) 1TB VBK extents. Per-VM chains is great but modern VM's do keep growing in size. I have 10TB+ VBKs and probalby soon I'll get some in ~20TB range - and that's already deduped data (NTFS dedup in VM). Some of them can be split up with some work but others will have stay as they are.
DonZoomik
Enthusiast
 
Posts: 25
Liked: 9 times
Joined: Fri Nov 25, 2016 1:56 pm

[MERGED] Can Veeam be configured to use maximum file size?

Veeam Logoby sambsamb » Fri Dec 23, 2016 2:16 pm 1 person likes this post

Can Veeam be configured to have a maximum backup file size for backup files like .vbr and other Veeam backup files - such as 512 MB or 256 MB?
So that a 5 TB backup job would result in 10*512MB backup files instead of a single 5TB file?

Thank you
Sam Boutros, Senior Consultant, Software Logic, KOP, PA
http://superwidgets.wordpress.com
Powershell: Learn it before it's an emergency
http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx
sambsamb
Influencer
 
Posts: 14
Liked: 3 times
Joined: Sat Sep 13, 2014 5:41 am
Location: KOP, PA
Full Name: Sam Boutros

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Julien3110 and 1 guest