Large VIB file on a small static server

Availability for the Always-On Enterprise

Re: Unusually Large Increments from Veeam???

Veeam Logoby J1mbo » Fri Mar 09, 2012 9:04 am

Anything that changes a block will create an increment load. Specify VM and host RAM to avoid paging. Do you have any defrag utility? Or some indexing function? Another potential culprit could be NTFS last accessed time if pre 2008 server (set HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem Key NtfsDisableLastAccessUpdate as DWORD to value 1 then reboot).
J1mbo
Expert
 
Posts: 253
Liked: 28 times
Joined: Tue May 03, 2011 12:51 pm
Full Name: James Pearce

Re: Large VIB file on a small static server

Veeam Logoby Unison » Mon Mar 12, 2012 3:30 am

Hi Jimbo
Disk defrag reports the drives are not badly fragmented and no there is no indexing happening on this server. All our servers are 2003 so i will try that key you mention. Can you pls give me any extra information on what exactly that key does or why it might help to resolve this large increment issue?

Also my backups are set to 'local target' - but seems as though another poster got good results from setting to LAN Target. I might try this but wanted to know if anyone is aware of a reason why you wouldnt want to set the jobs as LAN target even though the target is actually local to the veeam server? Could this impact speed or put any extra traffic on the network etc?

Thanks
Gav
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Gostev » Mon Mar 12, 2012 9:56 am

LAN and especially WAN target use smaller blocks to track the changes in the virtual disk, which reduces the incremental backup size (typically 2x with WAN target comparing to Local target). However, this may also reduce backup performance because of many more blocks to process (this is actually explained right there in UI in the option descriptions). This speed reduction will not be seen when the job is WAN link speed bound anyway, but you will definitely see the performance drops with local backups.
Gostev
Veeam Software
 
Posts: 21039
Liked: 2266 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Large VIB file on a small static server

Veeam Logoby Unison » Wed Mar 21, 2012 3:47 am

Hey guys,
I have tried setting the NtfsDisableLastAccessUpdate to 1 as per J1mbo's suggestion however that is still not helping to reduce the increment size on these servers. I am puzzled as to why a different imaging program that was backing these servers up for years only produced increments of a few hundred MB at most but veeam is popping out increments 6-7-8 gig in size.
Does anyone have any more ideas on how to pin point what is causing this? Really appreciate any help/advice.
Thanks
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Gostev » Wed Mar 21, 2012 10:30 am

Disk fragmentation, or workload specifics results in those few hundred MB making 6-8 GB of blocks all around the disk dirty?
Gostev
Veeam Software
 
Posts: 21039
Liked: 2266 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Large VIB file on a small static server

Veeam Logoby Unison » Thu Mar 22, 2012 1:47 am

Thanks Gostev,
Tho not exactly sure of your question - i just cant see why one backup program only sees (for years) a few hundred MB of changes per increment but Veeam is seeing Gig's and Gig's.

I might try disabling CBT on these jobs to see if that helps however i dont believe that will make any impact on this issue because CBT is just telling veeam what has changed since the last backup....which i think veeam will find the same changes when it does the backup without CBT.....OR is it possible that CBT is reporting back changes that veeam and other backup applications would just not consider as a 'change' or necessary to backup?
Will veeam backup every block that CBT says has changed or will veeam still read from the blocks presented by CBT and make a decision on whether or not to backup those blocks?
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby tsightler » Thu Mar 22, 2012 2:50 am

So was the other "image based" product backing up via the hypervisor, or was it using some type of in-guest agent? If it was an in-guest agent my guess is it was using a very small block size. Veeam uses large blocks (1MB by default for "local target"), that means that a disk with many very small changes spread across the disk will show a large change rate. You can try using "WAN target" which uses 256K blocks even if your disks are local as what this actually sets is the smallest block size that Veeam will process when a change to the disk occurs. This can have a huge savings on systems where there is a lot of very small changes, however, that's still typically much larger that agent based image products, which generally use something along the lines of the NTFS cluster size, commonly 16KB. This is pretty well covered in other threads.
tsightler
Veeam Software
 
Posts: 4659
Liked: 1680 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Large VIB file on a small static server

Veeam Logoby Unison » Thu Mar 22, 2012 3:34 am

Thanks tsighler
I have just done some testing on some VM's and turned off CBT on their jobs - i expected the increments to take a bit longer but no, they are taking ages. Some increments taking over 1hr when with CBT they took a few minutes. HOWEVER the increment size is extremely small - granted there was not much time for change during the increment tests but all increments were well and truly small, all less than half the size of an increment with CBT.....and one was down to 50MB!!!! perfect increment size for a 'low activity' VM!
So my question in the last post still stands - can anyone help? The question being:

Will veeam backup every block that CBT says has changed or will veeam still read from the blocks presented by CBT and make a decision on whether or not to backup those blocks?

It seems that with veeam doing the backup on its own without CBT, the increments are smaller (but take a lot longer because veeam has to process the whole vm again) - could CBT be more 'sensitive' than veeam and when CBT is enabled, veeam actually just backs up anything and everything CBT tells it to (by marking more blocks as changed compared to what veeam would consider changed)?


Also, yes the other imaging product was agent based - symantec system recovery - i understand what is said about different block size processing of the different products and could see how this would have an impact on some servers. It doesnt make sense on these servers though because some of them are doing very (very!) little.
BUT, what i will do now is set these backup jobs to have CBT turned back on (so i get the speed back!!!) AND i will change them to a WAN TARGET so we get the smallest possible block size...I will then report back what speed penalty there is (thats why i didnt change from local to wan already because of the performance hit) and if the increment sizes improve.

If anyone has more info on why veeam might be able to produce a smaller increment on its own without using CBT compared to when it us using CBT i would be really interested to know.

Thanks guys
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Unison » Thu Mar 22, 2012 5:43 am

POSSIBLE SOLUTION / CAUSE....

(Sry, long post but hopefully the pro's can shed some light :))

I think i found the answer to the above question (Does veeam just backup everything CBT says has changed?)....Short answer....YES.

Through some testing today, i have found something undesirable (or maybe realised it rather) - which is probably the cause of my larger than expected increments.

One of these servers suffering from large increments has one 40gig drive. 22gig of that is used - when veeam does a full backup of this server the VBK file is about 7gig - a veeam increment for this server with CBT is normally around 500MB.
Today i copied an 8gig file to this server, deleted it, copied it over again and deleted it again - after this the 8gig file was now gone from this server and its used space was again 22gig. I than ran another increment of this server. Some of you might at this point see where this is going....
With no new data added to this server, the increment should be extremely small....but in this case, because CBT is enabled on this server, this next increment was 9GIG (which is even bigger than this servers base image!)......obviously this is because when i copied the 8gig file to this server, all those blocks changed, then i deleted it, then i added the 8gig file again more blocks changed.....at the end of this process, 9gigs worth of blocks had changed so when the next increment was run CBT told veeam that alllllllll these blocks had changed. Simple. But - no data was on these blocks....they had just been changed.
This was what i was trying to find out and it seems this is the case - CBT tells veeam that a block has changed, veeam just listens and then backs up that block, veeam doesnt do any further analysis (suppose there is an obvious answer here as backing up block level veeam cant actually see what is in the block so doesnt know if it holds data or not, it just knows that its changed). I suspect this is why the increments are so large, when a file is added then deleted or just deleted, blocks are changed and then because they are changed they are backed up - even if those blocks actually contain no data any more - like in this test, 9 gigs worth of empty blocks were backed up even though there was no data in any of the blocks - basically creating a huge increment with absolutely no real data in it.

Is it possible......when CBT hands veeam a list of all the blocks that have changed, is there any way that veeam could then process each block and test/analyse if it actually contains 'real data' - so that the backup is fast because only changed blocks are being processed BUT blocks which contain no data are NOT backed up? (or is my comment above about not being able to analyse the block level the limiting factor?).

I also suspect that this is why the veeam increments are smaller in size when you turn OFF CBT - because veeam is then processing the vm like other imaging products (although through the HV), not just listening to CBT about what blocks have changed. So without CBT your increments only contain new data so are small, but with CBT your increments not only contain new data but also possibly empty blocks which will blow out the size of your increments.


From testing - without using CBT the increments take just as long as the full backup, obviously because veeam has to process the VM each increment. Does anyone know why, without CBT veeam takes so long to process the VM and create a new increment? With agent based imaging apps, the entire server is processed with each increment - why can it do it in minutes but veeam takes the same amount of time as its full backup? I would like to turn off CBT to reduce the increment sizes BUT i cant because without CBT the increments just take too long. Both agent based imaging and veeam (without CBT) process the entire server for an increment but the agent based is so much faster - why does veeam take so long when its doing the same thing?

Thanks in advance for taking the time to read/respond
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Re: Large VIB file on a small static server

Veeam Logoby Gostev » Thu Mar 22, 2012 9:54 am

The size of the produced increment is going to be exactly the same with and without CBT, because the only difference is how we identify changed blocks (by using CBT data, or by reading each and every block and comparing its state to the previous state). As you can imagine, both methods will produce exactly identical results - but one of them will be real slow.

Your testing shows difference in backup size because you are not doing it correctly. If you re-run the same job again soon, of course the incremental backup size produced will be smaller than with once-per-day run, because very little data is changed in this short time span. If you create 2 separate jobs for the same VM (with and without CBT) and run those on the same schedule, you will see that they size of backup files they produce is the same.

Your scenario with adding and deleting file is also not valid. As I already posted in another topic yesterday, in real world server workloads, data is never just removed, but is always replaced with more data (immediately, or eventually). When was the last time that you removed a hard drive from any of your computers, because you did not need the disk space that you needed before? We only keep adding.

The real reasons for increments being large for some server are well known, and explained by Tom just before your two post.
Gostev
Veeam Software
 
Posts: 21039
Liked: 2266 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Large VIB file on a small static server

Veeam Logoby tsightler » Thu Mar 22, 2012 2:58 pm

I believe you may not be understanding exactly how Veeam works as it works on a completely different principle from an agent based image backup solution. Agent based solutions effectively backup the "filesystem image", however, Veeam backs up the VM disk image, exactly as it exist, without regard to the filesystem (with one exception in V6 with regards to the Windows swapfile). If you create a 40GB virtual disk, add 20GB of data to it, and then delete it, then take a Veeam backup, the Veeam increment will 20GB (assuming no compression). The idea here is that, when you restore this VM, it will be an EXACT image of what that VM looked like when it was backed up, including any deleted data as, even if the data has been deleted, the blocks themselves did change. This actually offers some advantages. For example, I was once able to use a Veeam backup to recover a file that had been created and deleted on the same day by using an NTFS "undelete" program on a Veeam backup from a week earlier. This is something that would have been completely not possible with a traditional backup product. Also, a Veeam backup of a powered off VM is a forensically accurate image of the system, which can be very valuable for security related analysis.

Certainly, the disadvantage of this could be that a system which does have a lot of files being created and removed will show large increments as Veeam will literally backup every block that has changed on this disk. For example, a server which processes image files or archive files using a drop folder may see a very large increment. The only real workaround for this is to use some tool like "sdelete" that "zeros" the data whenever a file is deleted. Veeam will still see these as changed blocks, but since they will contain zeros it will not grow the incremental backup.
tsightler
Veeam Software
 
Posts: 4659
Liked: 1680 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Large VIB file on a small static server

Veeam Logoby Unison » Thu Mar 22, 2012 11:47 pm

tsightler - thank you! I really appreciate your response - thanks for delivering it with 'teaching/sharing' in mind rather than arrogance.
Your helping me to see this clearer. It makes sense why a veeam full backup is slower than the old agent based imaging we used because for the FULL veeam is looking at every block for the vm regardless of if that block contains data or not - i.e. even if your disk has 200gig of 'free space' the veeam backup process will still look at every single one of those 200gig blocks where as a 'file system' aware imaging app will recognise the space as empty....hence the longer times for veeam backups.
It doesnt look like i am going to be able to do anything about these large increments - on top of the fact a larger block size is being used (also tried WAN Target for the smaller block but that didnt reduce the size by much) i think that the large increment size issue is compounded by the fact that the increments will contain 'changed blocks' that actually contain no data. Thanks for pointing out the benefits of this - the fact you have an exact forensically viable disc copy AND you can recover deleted files from restore points because only the addressing is removed when a file is deleted, not the actual data. I would rather only have increments that contain new data than have these benefits but you cant have everything and i suppose i chalk this one up to new requirements/processes of our new VM/backup environment.
Because of dedup and better compression, i was actually expecting smaller backups for all my VMs compared to the old backup system on the physical systems - and this is true for the FULL backups but not with the increments. With the veeam Fulls they are a lot smaller than the fulls with the old software so i save heaps of space there but then when the increments kick in, 1-2 days into an imaging set i am already using more backup space than a full 7 day imaging set with the old solution because the increments are so much smaller with the old backup product/technology.

Also thanks Gostev. As mentioned in my post, my testing was fairly rough + quickly done and i realise it was not exactly scientific but it did seem to show me different results which is why i fired off the question to you guys who know a tremendous amount about how veeam and the backup process work. You did answer my question though, so thank you - confirming that veeam will recognise the exact same 'changed blocks' on its own as it would with just listening to CBT (essentially the increment sizes will be exactly the same size with or without CBT - so you have helped me remove suspicion from the CBT process). I didnt see your other post, sorry - but can you see how increments can be blown up as in my test simply by adding a file to a server then deleting it before an increment - you have deleted the file but because the blocks have changed, the increment will grow by the size of the file you added/deleted even though that file doesn't exist any more (i.e. if no compression etc is used). I didnt realise this so if someone else finds this post because they too are noticing huge increments - perhaps they will realise that even if a file is placed on a vm then deleted before a backup, that file will have caused changed blocks so those blocks will be backed up and add to the size of your increment. If you have a system where files are added/removed a lot, this will likely be your cause for large increments - would you consider this one of the 'real reasons' increments are huge?

So far i believe the reasons mentioned to be:
1- high activity discs (no help for you in this case)
2- Regularly defraging your disc (will cause huge amount of changed blocks so huge increments - dont defrag regularly in this case or defrag only right before your next full)
3- lots of small changes to lots of small files - i.e. resulting in full 1MB blocks being backed up for even a small 50kb file change (switching to WAN target may help you in this case)
4- NTSF change on last access reg key (I had this set but didnt seem to help at all so probably wasnt causing issues in my environment)
5- If files are added to a vm then removed - even those now 'empty' blocks will be backed up (even though we know they are not technically empty)

I suspect reason 5 and 3 are whats causing the large increments for me - even though switching to 256k blocks (down from 1mb blocks) via setting the WAN TARGET didnt really help. I cant do anything about 'empty' blocks being backed up and really a changed block is a changed block so it has to be backed up by veeam. Setting the backup job to WAN target didnt really produce any slower backups or any performance hits (not sure why that is? shouldn't i see performance problems going from local target to WAN target?) and didnt result is much smaller increments.

I think others who become new to veeam and discover larger than expected increments should be able to find all the reasons here and possible tweaks so thanks to all who have responded so far.
Unison
Enthusiast
 
Posts: 80
Liked: 16 times
Joined: Fri Feb 17, 2012 6:02 am
Full Name: Gav

Consistently large rollbacks from 2003 VM

Veeam Logoby ChrisL » Mon Apr 09, 2012 7:15 pm

[merged]

Hi all,

Interesting one for you, it's not specifically a problem with Veeam, that bit is working just fine, bit its sort of related.

Long story short, got a Server 2003 VM backing up in its own Veeam job set to user reverse incrementals and so creates a .vbk and corresponding .vrb's. The curious thing is that the vrb files are consistently quite large, regardless of how much the VM is use. It's a main file server, so I would expect some pretty heavy usage but since we are a school, you would expect the usage (and hence the rollback sizes) to drop right off during the holidays, as we are right now, but the rollbacks are still about the same size as normal.

Veeam reports the VM size as 900GB and the rollbacks are usually about 60-70GB, whether the system is used or not. I'm not seeing any errors in the reports, CBT is working fine as far as I can tell, and this is the only job of 22 jobs that is behaving in this way - its not actually failing, just taking a long time to complete.

So, im sure Veeam is working fine, it's presumably something to do with the VM, my question is what might it be doing that is causing the large amounts of data to change? There's no SQL on the server and as far as I can tell there's no background defragging going on. Alternatively, is the any way that I can see what is actually being backed up in the rollback file which might give a clue as to what the actual differences are?

Historically the VM was P2V'd from a physical box if that's relevant in any way..

Any thoughts welcomed..
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

Re: Large VIB file on a small static server

Veeam Logoby ChrisL » Mon Apr 09, 2012 9:10 pm

Thanks for merging, I knew I should have checked the forums first, d'oh!

I'll withdraw the last question then, about whether it's possible to look 'inside' to rollback, I realise now that that's a silly question and shouldn't have been asked and seems so obvious now taking about how Veeam actually works at a VMDK block level! The first bit remains though, I wonder what exactly the VM is actually doing during the day that can cause so many block changes that are then being recorded by VMWare and hance captured by Veeam..
ChrisL
Enthusiast
 
Posts: 41
Liked: 6 times
Joined: Mon Mar 21, 2011 12:04 pm
Full Name: Chris Leader

Re: Large VIB file on a small static server

Veeam Logoby Vitaliy S. » Mon Apr 09, 2012 9:56 pm

Hi Chris, actually it might be anything starting from last access time updates up to daily antivirus checks or file relocation. Please look through the first pages of this topic for more info. Hope this helps!
Vitaliy S.
Veeam Software
 
Posts: 19087
Liked: 1053 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], lharmer, Yahoo [Bot] and 58 guests