Why are the incremental backups so large and time consuming?

[Sk3tChY]

New to VEEAM. I have a single, physical W2K12R2 server running a few things like ADDS, DNS, DHCP, File Services, Exchange, SQL. I'd like to have the server backed up daily with the ability to perform both bare-metal and file level restores from any previous point in time - i.e. retain a lot of previous versions. Since my setup and initial requirements seem fairly simple, I've tried starting out with VEEAM Agent for Windows Free Edition.

Server storage currently looks like this:

C: = 3.63TB Total, 3.16TB Free, 486GB Used
D: = 223GB Total, 124GB Free, 99.2GB Used
Total = 585.2GB Used

Backup Location:

F: = 1TB USB External HDD, dedicated to backups, 931GB available

----------

Configuration of VEEAM Agent for Windows Free Edition is as follows:

Source -------------------------------------------
Backup mode: [ entire computer]
The entire computer will be backed up

Destination --------------------------------------
Local folder: F:\VeeamBackup\
Keep restore points for the last 14 days
Compression level: Optimal (recommended)
Storage optimization: Local target
Storage encryption: disabled

Schedule -----------------------------------------
Schedule: workstation
Daily at [12:30 AM]
If computer is shutdown at this time: [skip backup]
Once backup is taken, computer should: [keep running]
Backup no more often than every [2 Hours]

I haven't configured periodic active full backups because I figured this would consume more space, but please correct me if this is wrong.

----------

Initially I had the restore points set to 365, but after running this backup schedule for just under 2 weeks I started receiving warnings about the backup location getting low on free disk space.

This was a little perplexing when the server is only consuming 585.2GB and essentially has 931GB of space to backup to.

Upon further investigation, I noticed that the incremental backups appear to be taking several hours and consuming significantly more space than I'd have expected, with the majority of the 13 .vib files ranging from 43GB-74GB!

My fundamental understanding with modern backup technology is that an initial full backup is taken, which is essentially a copy of all of the data and thus the largest and most time consuming to perform, but this is then followed by incremental backups which basically just track and store changes that are made from the original full backup file - allowing for significantly quicker and smaller subsequent backups.

So in my case I would have expected the initial full backup (.vbk) to have taken the longest amount of time and been somewhere in the range of 585.2GB, followed by daily incremental backups (.vib) that would be no more than about 1GB, as I'm sure there's no more than a 1GB worth of data added/modified each day on the server.

Instead, the initial full backup looks to only be about half the size of the server at 286.6GB and as previously mentioned, the incremental backups are all relatively huge, mostly ranging from 43-74GB but with a few at 10-21GB.

I've gone over the entire interface and aside from having the option to set periodic active full backups and set storage compression level, optimization and encryption - it doesn't look like there's anything else I can change to make things any different.

----------

Which brings me to my questions:

1. Is there something I'm missing here? Why are the incrementals so huge? I would say about 200 emails are received each day, 200 records added/modified on the SQL DB and perhaps 50-100 documents added/modified - if that. I certainly don't think there's any more than a GB worth of data being added/modified on a daily basis.

2. Any reason why the initial full backup is so small? It's about half the size of the actual amount of data consumed on the server, is it just really compressed?

Any input much appreciated, I'd like to avoid going out and purchasing a larger external drive if I can make do with the existing one. Even then though, at this rate I'd only ever be able to retain about 2 months worth of previous versions.

[nielsengelen]

We have built-in compression and dedupe and in most cases a full backup is around ~50% of the used size so this seems to be correct. An incremental is usually around 10% as this is a fairly common change rate for an infrastructure. If you want to store your backup for 365 days this would mean between 15-20TB on the data you've given.

We don't work on a file based backup but block based. Looking at the services running on the server it looks normal in regards to the size. Even if you only modify X amount of files it's more in the background.

It may also depend on which software versions you are running (Exchange, SQL) to explain the growth of the incrementals.

[Sk3tChY]

Thanks for the reply, I appreciate it.

Would data dedupe be doing anything in my scenario where I'm literally just backing up a single physical server? I'm guessing if it finds multiple copies of the same chunk of data in different locations on the server it could save space there?

Having the initial backup at around 50% is impressive, it's also something I can wrap my head around with all the fancy wizardry which likely goes on behind the scenes.

What I'm still struggling to understand is how/why the incrementals would still be so big and time consuming? Is my fundamental understanding of an incremental backup incorrect or something? (i.e. It just backs up the changed blocks of a piece of data, rather than the entire piece of data.)

I don't understand how the initial full backup can be just 286GB, but then have some incrementals which are a quarter of the size at like 72GB - how could there have been 72GB worth of changes in one day?

I don't mean to mention other products, but in other environments where I use a Synology NAS as the File Server with its built-in Hyper Backup, I can retain an almost indefinite amount of previous versions and the incremental backups don't take up much space or time at all.

As an example: One of these NAS devices has Volume 1 which contains all of the data and is a comparable 1.34TB total with 595GB used. This uses the built-in Hyper Backup application to backup to Volume 2 daily, which is on the same set of disks and dedicated to backups only. Volume 2 is only 359.18GB, even though it has backed up EVERYTHING on Volume 1 and retained 364 previous versions of each file. Each incremental backup that runs every night takes less than 10 minutes to run also.

My interpretation of this, is that the initial backup also uses some compression/dedupe wizardry to make itself smaller and then the daily incremental backups are essentially just backing up a few new email/document files each night as they're added to the server, so they're all very small and run in minutes.

Obviously it's not the same scenario, they're not bare-metal backups and the NAS isn't running Exchange/SQL, but yeah - I don't understand why incrementals in that scenario seen so efficient, but in this scenario they're so large? I don't mean to sound negative towards VEEAM, I'm sure there's a good reason for this, but I'm just trying to understand what's going on here.

[ajafar]

I had a similar problem. Server 2012 R2 Essentials using around 200GB of storage. 1st Full Job used around 180GB. Each incremental was around 20-30GB daily which was not normal in my opinion. The second I uninstalled the CBT driver, incrementals are now around 200-400mb/day.

[Sk3tChY]

Thank you for the recommendation, appreciate it.

I definitely agree with you - this doesn't seem normal to me. As mentioned in my previous post, the Synology setup/solution I also use feels more like a typical "incremental" backup system to me - the first backup is time consuming and more-or-less just copies everything and then each subsequent backup appears to just backup changes, making it very small and quick to run.

The only major difference I can see between the two setups, is in this instance we're wanting to do bare-metal backups - but surely this shouldn't result in the incrementals been tens-of-gigabytes larger in size.

Doing a little research it looks like this CBT Driver is supposed to increase the speed and efficiency of incremental backups by tracking changed blocks - it seems like removing it would be counter-productive? Have you tried recovering from any of these smaller backups? (Perhaps the sizes have reduced significantly because without the CBT it doesn't consider there to be any changes and it's actually not backing up anything?)

More than happy to give this a shot and see what happens - we're still fiddling and trying to settle on a backup solution.

----------

I ended up purchasing a 5TB external drive to dedicate to these backups, so setup an entirely new backup job/schedule within VEEAM, which has been running for almost two weeks now and it's pretty much exactly the same story.

The first full backup was about 291GB (there's about 390GB of total data) and every single incremental since then has been:

1st - 38GB
2nd - 39GB
3rd - 50GB
4th - 61GB
5th - 46GB
6th - 43GB
7th - 48GB
8th - 46GB
9th - 62GB
10th - 48GB
11th - 66GB
12th - 45GB

Actually clicking on one of the jobs to get more information, I've noticed the following trends:

- It tends to be reading about 80-100GB from the C: each night. And it seems to read it very, very slowly at just 17-27MB/s.
- It tends to be reading about 75-80GB from the D: each night. And it seems to read this much, much faster at about 400-500MB/s.

The D: is a 4 x SSD RAID10 array, so it makes sense for that to read significantly quicker than the C: but the C: is still a 3 x 7200RPM RAID5 array and should definitely be reading faster than 17-27MB/s.

I've also noticed [CBT] at the end of some of these items - which I'm guessing means the CBT Driver was used for these items?

[Sk3tChY]

By the way - I've looked up how to install/uninstall the CBT Driver on Veeam Agent for Microsoft Windows and apparently I should be able to do this from the Control Panel > Settings, but I don't see anything related to the CBT Driver here or anywhere within the Control Panel / Console. I've also tried looking for Veeam CBT Driver in Control Panel > Programs and Features and it isn't listed anywhere.

According to this - https://helpcenter.veeam.com/docs/agent ... tml?ver=21 - I should be able to install/uninstall the CBT driver like this.

Could this perhaps be a reason for my issue? Is there any reason why I can't see the CBT Driver anywhere? Is it because I'm using the free version?

[Rudolf]

Same issue here.
Would be great if we could see some info on what was causing the large backup.
Maybe some reverse block allocation tool that can be run on the client and do lookup backed up blocks against current file table - Not sure how possible or difficult that would be.

RE the CBT driver: Only available in Server OS's:
https://helpcenter.veeam.com/docs/agent ... tml?ver=21

You can install the Veeam CBT driver at any time you need. This operation is available if you use the Server edition of Veeam Agent for Microsoft Windows on a computer that runs a Microsoft Windows Server OS.

[ajafar]

Sorry for the delayed response. In the Server Agent, it's inside the agent control panel and not the windows control panel.

https://helpcenter.veeam.com/docs/agent ... tml?ver=21

Follow the top half of the instructions, not the bottom half.