Large VIB file on a small static server

Availability for the Always-On Enterprise

Re: Large VIB file on a small static server

Veeam Logoby Unison » Tue Apr 10, 2012 1:31 am

I have had some positive results since my last post on this issue above…

Based on some of the input in this topic I have gone back to my VM’s/Veeam setup and changed/tried a few things – my increments are now MUCH smaller now! What worked for me, may not work for you but if your seeing large increments – you could try what I did.


It seems that ‘reason 3’ in my last post above was the main cause for my issue with large increments. The issue is primarily due to the large block size of 1mb used by veeam when you specify the target as ‘local’ on your backup jobs – I could probably word that better so please do not interpret that as me ‘blaming’ veeam for this issue….I am not. I am basically saying, if veeam used a smaller block size…..this wouldn’t be much of an issue at all – if this is not too much of a ridiculous question, I wonder if someone can please provide a clear answer: why does veeam use such large block sizes (even 256k for WAN target is very large but is the smallest veeam can use)? Why not something much smaller still like 32 or 64k to get closer to agent based imaging increment sizes?

What IS working for me now though…
I trawled through the discs on all my VM’s and did a massive clean out – clearing old files, archiving others, deleting temp files etc etc…some of these servers are citrix servers so I got rid of temp internet folders and cookies where some of those folders had tens of thousands of files in them of no more than a few KB each!...after reclaiming some gigs I defragged all discs – everything looked a bit ‘cleaner’ from the defrag report.
I then did some backups with veeam – and my increments, were smaller! Clearing out the discs, removing unnecessary fragmented files and then doing a fresh defrag certainly appears to have helped. Increments have pretty much halved in size just by doing this – but obviously didn’t help much on servers that did not need a defrag and did not have much data ‘cleaned’ off.
Now to the block size issue – as in above posts, I have now also changed ALL my backup jobs and set their targets as ‘WAN Target’. This is so I can get the smallest possible block size with veeam of 256KB….now im getting super small increments again, pretty damn close to what I was getting with agent based imaging of the physical servers. With the tidy/defrag and the WAN Target setting, my increments have gone from being about 8-10gig on the worst end to now being around 2gig-500mb!! A massive improvement!

I will complete defrags on a more regular basis now (just before a new image base) and I will apply this WAN Target setting to all other VM’s as they come on board. There was talk of a performance issue or overhead from using the ‘WAN Target’ setting over the Local Target setting however I am seeing no such performance issues and I have been running like this for a week now. In most cases the jobs set as WAN target were finishing before/quicker than those jobs that were set as local target – Im not sure if this is weird, but it seems weird to me….i would expect the ‘wan target’ jobs to take longer – but they do not.

Appreciate all the input on this topic to date – thanks! Will keep an eye on this topic and eager to see if anyone can answer my queries above.
If you are having issues with your VM’s and Veeam creating large increments – hopefully you are able to use some of the information in this topic to get veeam and your vm’s producing acceptable increment sizes!
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby ChrisL » Tue Apr 10, 2012 12:11 pm

Thanks Vitaliy, I'll have a poke around and see what might be happening. The good news is that the server, being an old 2003 'box', is due to be decommissioned soonish and probably split up into a few smaller new 2008 VMs. Hopefully the new ones will play a bit nicer!
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

Re: Large VIB file on a small static server

Veeam Logoby dellock6 » Tue Apr 10, 2012 5:42 pm

Gav, reducing the block size Veeam is processing will surely improve deduplication ratio, but on the other side is going to increase the burden on Proxies CPUs: 256k is 4 times smaller than 1Mb, so you will have 4x the amount of blocks to compare in order to dedupe. Going even smaller on blocks will surely require much more cpu power, and probably memory/cache to store hashes for dedup.
Probably in the future, taking advantage of cpu constantly increasing their power, Veeam will be able to reduce this value.
My 2 cents.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 4989
Liked: 1307 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Large VIB file on a small static server

Veeam Logoby Unison » Wed Apr 11, 2012 2:40 am

Thanks for the response dellock6.
Basically the lowest block size is set by veeam at 256k (using WAN Target) because veeam thinks that any smaller block size would be too much for modern processors to handle and would overload veeam proxies during dedup/compression?
You might not exactly know why veeam have set the lowest block size at 256k but is that basically what your thinking is the reason for the limit?

During my tests, I didn’t see any performance hit on my veeam server using the different ‘block sizes’ of WAN (256k) and Local (1MB) targets – which is also the proxy. Its CPU is an i7 2700K 3.5ghz 8 cores – running W7 pro. Due to the fact I see no penalty in backup time using WAN or Local target (to my surprise I might add) – I now set all my veeam jobs to use WAN target so I get the smallest possible increment size.
It seems that with this processor (which is not the best, not server grade, not the fastest, not very expensive) we could all expect that smaller block sizes would not over load our veeam servers or blow out our backup windows….even when you say something scary like a 256k block size will take 4x more ‘effort’ to process compared to 1MB blocks…..in the real world, it doesn’t translate and doesn’t seem to impact processing time at all…..even with a desktop processor like this one I am using (obviously if your using a lower grade of processor than what I am, you might indeed see processing issues the smaller you set the block size).

Veeam let us choose from a block sizes from 256k to 1MB using the drop down target type in the job setup – would it be possible for veeam to add a few more items to this drop down list so we can drop the block size even further down past 256k – say adding a 128, 64 and 32k option? That may be massively over simplifying what would be required – but I wonder if you or one of the veeam team could comment on the possibility of allowing us to choose a smaller block size than 256k?
We could then test our veeam servers against the different block sizes and determine what gives us the best increment size without causing a processing overload on our veeam servers – this would allow us to choose a block size that takes more advantage of our individual processing capabilities and allow us to get maximum use out of our storage systems.
Whats the harm in letting us choose a block size lower than 256k? If it can be done, can it be added to a future patch/release?
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby tsightler » Wed Apr 11, 2012 3:25 am

I would have to say that yes, you are probably making it a little too simple.

Since you didn't really provide any details of the VMs you were testing against it's pretty much impossible to know for sure, however, my guess is that your VMs are generally small, and that your testing involved running a limited number of jobs at a time, with minimal restore points. Backing up a 2TB VM using 256KB blocks would naturally have a much larger impact than a 100GB VM as the 100GB VM would potentially need 400,000 hashes, while the 2TB VM would need 8 million. On the other hand, a 2TB VM with 1MB blocks would only need 2 million. All those hashes have to be stored in memory, and compared against as the job runs to determine deplicate blocks. As the VM size gets bigger and bigger, the more impact the smaller block size will have. Imagine the impact of 32K blocks on the 2TB VM? Suddenly it would potentially need 64 million hashes, which would certainly be impactful.

You then also need a storage technology that can store all those small blocks efficiently, track the blocks across incrementals and reverse incrementals, and do so efficiently for things like file level recoveries, instant restores, surebackup, etc. The smaller the block size the more likely that the file will become fragmented over time, thus adding another factor to performance.

That being said, if your VMs are fairly small, using WAN mode generally will not have a significant negative impact on performance, at least for backup, but it is a balancing act that has many more variables that your simple test addresses, specifically around restore times and capabilities. Even expensive hardware dedupe appliances have a very negative impact on features like Instant Restore and Surebackup due to the overhead of rehydrating data from their small block dedupe.

In other words, it's not just about about reducing the block size, but about doing so without impacting backup performance, restore performance, and scalability. Right now, 256K seems to be the lower bound that makes sense when balancing those requirements, but certainly I wouldn't rule out a smaller lower block size in the future.
tsightler
Veeam Software
 
Posts: 4737
Liked: 1728 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Large VIB file on a small static server

Veeam Logoby Unison » Wed Apr 11, 2012 4:24 am

Thanks Tom, lots of great information and detail in there – appreciate your insight.
We will have 9 VMs when I am finished and they mostly are pretty small….around 100-200gig each with just one being around 600gig….probably explains the reason why I don’t see a performance hit when I use the smaller or larger block size because we are not playing around with large enough numbers to reveal any of the saleability/performance/restore problems you mention.
I can see your point when you start to talk about TB’s….you clearly show where the problems will come in on the larger end of the scale – but on the lower end of the scale where a smaller number of VMs are involved and smaller VM sizes….even the 256k block size might be a bit much and a bit inflexible to the ‘needs’ of the smaller guys. We are an SMB Accounting firm with 2 offices and 60 staff….not terribly small and I am sure there are many businesses around this size or even smaller using Veeam. A smaller block size selection will help us, not the big guys ……but I absolutely agree that based on the potential problems as things scale up, a smaller block size is not practical.
1MB block is a good size for large systems to balance out the performance/risk issues…..however 1MB for smaller shops….even 256k for smaller shops really might not be low enough…..but as you said you wouldn’t rule out a smaller block size option in future, I too hope that comes about to recognise the differences in this section of the Veeam market and user base.

With the issues regarding recovery – I begin a new base and backup set every week, so that the set does not become too ‘complicated’. This also helps with fragmentation issues because last weeks images are cleared off the main drive and the new set is laid down basically on a clean drive each time and only has increments added to it for one week, minimising the fragmentation issue and taking complication out at restore time. Do you think that going to new sets more regularly will improve some of the potential recovery problems when using smaller block sizes because of the short set window, blocks not needing to be tracked very long etc etc?

Also with certain recovery options like instant restore, sure backup and file level recovery with smaller vm sizes and smaller block sizes – is it very common to see a recovery fail because the ‘system’ has trouble processing a larger number of blocks during the hydration process? I have not seen it ever happen – but then again, I have not been using veeam for long (note, the only dedup I am using is what veeam does – allowing only veeam dedup in the backup process removes a layer of complexity/potential-issue?).


As you describe, the risk and balancing act becomes more and more complicated as you scale up – like with most things. However hopefully veeam can provide lower block sizes for the smaller scale setups that would be impacted more by a larger block size as apposed to a smaller block size.

In smaller setups, a larger block size means larger increments and not much else – but smaller block size means smaller increments with no real processing performance impact, no real recovery impact, no real impact to backup window.

In a larger setup, a small block size means not much in the way of saving space (because you will have heaps anyway ), it means huge excess processing demands, extra requirements on storage, longer recovery times, flaky recoveries etc etc…lots of bad stuff.

Some scenarios the smaller block sizes would be useful, some it would not – but if possible, hopefully veeam in future offer some smaller selections. Im sure many people have been surprised by their new large increments after moving from physical to virtual servers….some might speak up, like here in this post, others might just accept it and not say anything…..possibly though, a lot of this ‘surprise/shock’ could be avoided with the use of a smaller block size for smaller VM’s (not that large block size is the only cause for large increments – but I think it would be a key cause most of the time).
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby dellock6 » Wed Apr 11, 2012 7:22 am

I think that the fact is, disks always tend to grow in size and performance at the same price, so it really easier to buy bigger disks and let the backup grow in size. Think about SATA disks, two years ago 1Tb was the biggest size, now we have already reached 3Tb. Using larger storage for backup gives you more space for retention, and can let Veeam run faster having the ability to use bigger block size.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 4989
Liked: 1307 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Large VIB file on a small static server

Veeam Logoby Unison » Wed Apr 11, 2012 10:47 pm

I agree that there is that argument too – ‘discs are so cheep…just buy more’. Without getting right into that and arguing against it, I think that really there is a possibility veeam could be ‘improved’ here by getting it to more efficiently using the space that it has available now (by allowing us to select a wider range of block sizes)……rather than just adding more drives as it sucks up huge amounts of space due to large block sizes.
Making it possible to select smaller block sizes (smaller than 256k) would allow many setups to overcome the huge increment size problem without impacting other areas (like cpu performance, backup speed, recovery time, recovery complexity etc etc).
Those with big systems obviously wouldn’t get much benefit from using the smaller block sizes….but those of us with a smaller number of servers and with servers of only a couple hundred gig each are going to see much benefit.
I was able to massively reduce my increment sizes across all servers and that was largely due to the fact I was able to drop down to a block size of 256k from 1mb without any performance or speed issues – my increments were touching 10gig each….NOW they are back down to pre veeam days of between 2gig and 500mb. If I am taking several snap shots a day…..those massive increments will soak up all my storage in less than a week…..where as with a more efficient (smaller block size), the same amount of storage will allow me to retain months of backups!
Hopefully veeam look at adding some smaller block size selections in a patch or future release – I think its pretty clear their product could do it but it will be up to us to select the appropriate block size to fit our performance, restore and storage capabilities.

Anton Gostev, you’re a veeam product manager and have contributed in this discussion – is there any possibility that smaller block sizes could be added to the selection list to help smaller shops with this large increment issue? It seems like a small change which could make a big difference to many in your SMB market.
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Gostev » Wed Apr 11, 2012 11:01 pm

No, smaller block sizes are not possible for a number of technical reasons, as well as backup performance considerations and unacceptable memory requirements for the data mover agent process. Tom has a good summary above.

Smaller increments for archival purposes, on the other hand, are achievable.
Gostev
Veeam Software
 
Posts: 21354
Liked: 2333 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Large VIB file on a small static server

Veeam Logoby Unison » Wed Apr 11, 2012 11:20 pm

Thanks Tom for all your input here - and thanks Gostev for providing that final answer on this possibility.

Anton, not sure what you mean here though "Smaller increments for archival purposes, on the other hand, are achievable." - can you pls explain what you are referring to? There are other ways besides what i have done to get smaller increments?
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Gostev » Wed Apr 11, 2012 11:32 pm

Sorry, I meant to say "possible in future". This will require some new functionality in our product... which we are considering to be high priority for some unrelated reasons anyway.
Gostev
Veeam Software
 
Posts: 21354
Liked: 2333 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Large VIB file on a small static server

Veeam Logoby Unison » Wed Apr 11, 2012 11:37 pm

great, thanks for clearing that up :)
Look forward to seeing the product evolve and backups becoming as small as possible.
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Thomy » Mon Apr 16, 2012 1:45 pm

Hello


I can confirm this behaviour, i have the same problem with Windows VMs (my Linux-based VMs are backuped fine with about ~5GB incremental for 70VMs daily) i have ~50GB for my 50 Windows VMs daily.
I am still searching for a solution (Veeam 6.0 Patch 3 on ESXi4.1U2 with vCenter Server 5.0).


Best regards

Thomas
Thomy
Lurker
 
Posts: 1
Liked: never
Joined: Mon Apr 16, 2012 1:41 pm
Full Name: Thomas Stather

Re: Large VIB file on a small static server

Veeam Logoby Unison » Tue Apr 17, 2012 12:52 am

Hey Thomy,
Your 50VMs are generating 50GIG worth of increments (across all of them) each day? How long have you been using Veeam? What were you using before veeam to backup and how big were the increments with it? If it was agent based physical imaging as discussed above - it may relate to the fact veeam uses a MUCH bigger block size than than agent based imaging.

Have you tried any of the suggestions in above posts to see if they help you?
# Seeing if your VM discs are badly defragged
# Not running defrags too regularly and only running a defrag after the last image of a set and just before the first image of a new set.
# Changing your Target Type to WAN Target in the backup jobs

Be interested to see what you find...
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby ChrisL » Sun Apr 22, 2012 5:04 pm

Hi all, little update for the knowledge base from here. We were finding we had large rollbacks on a relatively static file server but these have dropped off since running a defrag on the server. Previously we had rollbacks of about 60-70GB even during quiet times, now we have rollbacks of 20GB during busy times. On top of this, the backup time for the job (we have one job per VM) has dropped from 6+ hours to just over 2 hours - not bad for a 1.2TB VM!

For the record, I defragged both the system disk and the data disk, but didn't run a job between doing them, so I don't eally know which one made the biggest difference. It's possible that the large rollbacks were actually being caused by the system disk, which dropped from ~30% fragmented to 0% fragmented.

Either way, defrag has made a huge difference here and should be recommended for all.

The bad bit however is that we had to pretty much delete the backup chain and start a new one from scratch. The first job after the defrag obviously made a huge rollback, as you would expect, and this together with the large full VBK file pretty much filled the repository. Deleting everything and running a new full has obviously resolved this, but with a slightly scary few days while we were a little less protected than we would have liked - the new full took about 36 hours to complete so we were dependent on the tape backups if anything had have happened.

A thought for the Veeam team.. Maybe a mention in the installation guide to include the advice to run a defrag on each server before running the first full on it. Without fully understanding how Veeam processes the backups (which you may not during the initial install since you are new to the technology) it doesn't seem like an obvious thing to do, but it clearly helps hugely to start with a clean snapshot.
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], thosa, Yahoo [Bot] and 12 guests