Incremental backups are big

[robbys]

Hello,

I have a backup job of about 3.5Tb. I'm seeing a vib file each day of around 300Gb's. Is this normal?
ex. The job of yesterday had 302,7Gb read and 268,9Gb Transferred. Job's are almost not compresses (dedup friendly) and not dedupped by Veeam, so I get that I lose a lot on that side, but I'm actually not worried about the 268,9Gb number, but more of the 302,7Gb. Is it normal that there is 300Gb's of changes in 24 hours??? (the answer is no, because I do not see this behaviour with our other clients). But how can I troubleshoot this? how can I easely see what is changed in a day?

The disk that has the most data changed is the file server. This is a Windows 2012 R2 machine and it has Windows Deduplication enabled on it's 1Tb hard disk. Can deduplication inside a VM be held responsible for these large changes each day, or do I need to look at files/automatic backup processes of software,...

[Shestakov]

Hello Robby,

Could you tell more about those VMs in the backup job, are there Exchange, SQL servers?
For instance, Exchange servers create large amount of transaction logs every day, thus creating a lot of changes in the virtual disk that we need to process on the image level, putting all those changed blocks into the incremental backup file.

Antivirus scanning and defragmentation process between backup job`s runs also can cause this behavior. Since Veeam B&R is an image-based backup solution, it works with blocks, so a change of a file`s timestamp causes copying of the whole block.

Thanks.

[robbys]

Hello Shestakov,

it are 50 machines, but only our fileserver looks suspicious for me. 110Gb changed, and this day after day.
As I read your reaction right, this can be caused bij Windows 2012 R2 Datadeduplication inside a VM machine?

[Shestakov]

Correct, Windows data deduplication is probably the main root of the issue.

[centneroff]

Hi, All!

We have: Two VMs with guest OS Windows Server 2012 R2. The VMs are on the different ESXi hosts, ESXi version is 5.5 U2. Each VM has 1,5 TB thin provision virtual hard disk. Data Deduplication is enabled on each volume. This servers are in the DFS namespace and are in the DFS replication group. Also we use Shadow Copies on each partition with 100 GB limit. We use VEEAM B&R for each VM. Backup mode for Backup Jobs - Incremental.

Problem: Two weeks ago the size of increment files became very big. On the average the size is about 200 GB. Before it the size was about 50 GB.

Question: What happened We have open case about this problem, but we don't have solution. May be here we will find a member with the same problem?

[Shestakov]

Hi Dmitriy,
Data Deduplication is most likely the reason of incremental backups size growth.
Please review the post about best practices of dealing with MS Server 2012 deduplication. Ask additional questions if you have any.
Another possible reasons, as you can see above are: antivirus scanning, defragmentation process, work of Exchange, SQL servers inside the VMs.
Thanks.

[centneroff]

After increment files had become very big, we changed deduplication schedule. Now deduplication perform it's operations only on sunday and saturday.
We have no Exchange or SQL servers inside this VMs.
So, only one reason - antivirus. We removed antivirus(ESET Endpoint Antivirus) yesterday evening. Wating for tomorrows results...

[Shestakov]

Sounds good! Let`s wait for several days to see how significantly it will help.
Thanks!

[centneroff]

The Increment file of the first VM is 72,3 GB today, But the increment file of the second VM is 256 GB.

But We thanged compression to none on the second backup job...

Is it possible that the difference between increment files of the identical servers is so big in our configuration?

[Shestakov]

It`s possible to get the provided difference, especially for the first run after the configuration changes are done. Try to wait for 1-2 more runs and see how it works.
In general it`s recommended to set None (zero) compression level only if you use storage devices with hardware compression and deduplication tools for created backups. But since you switch it off, I would recommend to enable compression and deduplication options in the backup job. Thanks.

[centneroff]

It is unreal...

The Increment file of the first VM is 92,9 GB today, But the increment file of the second VM is 533 GB!!!

Is it possible? We,ve returned back the compression on the second backup job to hight.

Configurations of the VMs are absolutly Identical.

[Shestakov]

Once it happened, it is possible.
Again, to get smaller incremental backup files, I would suggest to turn deduplication on for your backup jobs(if it`s off now).
Then try to make active full and see what results you will have for further incremental jobs.
What is the size of full backups for those vms and what backup method do you use by the way?
Thanks.

[centneroff]

Deduplication for our backup jobs is turned on.
The size of full backup is 1,49 TB, backup method is incremental

[Shestakov]

Dmitiy,
Make a Full backup and new increments should be smaller, since after job configuration changes, increments of the same chain can have different sizes.
As to the fact that same VMs have different incremental sizes, it means that one vm`s data blocks change more often. Despite above-mentioned reasons can be just a greater activity.

The size of full backup is 1,49 TB, backup method is incremental

Is it the same for both VMs?
Thanks

[centneroff]

Yes, the settings is the same for both VMs.

[Shestakov]

Thanks Dmitriy,
But the question was about full backup size. Is the size of full backups same for those VMs?

[centneroff]

Yes, the full backups sizes are almost the same. 1,49TB and 1,47TB.

[Shestakov]

Thanks for the reply,
The significant difference between incremental backup sizes can be caused by a number of reasons: Above-mentioned Deduplication, Antivirus, Defragmentation, Exchange, SQL or just blocks of the 1st VM chane more intensively than of 2nd because of inner applications activity.
Let`s see how that will change after you shut down antivirus scanning and make active full backup.
Thanks!

[ThaVWMan]

I have one VM that I cannot seem to nail down what is causing huge amounts of changed blocks on a daily basis. The guest is running 2008 R2, and is also a domain controller. It has to virtual disks, and the c: disk is the one that has around 24GB of changes backed up each job run. There are no file shares off of this drive aside the ones for AD. Here is what I have done to narrow down the issue thus far.

1. Turned off AV, no change.
2. Verified there are no scheduled tasks running.
3. Moved print spool folder to different drive (grasping at straws here).
4. Starting running backup job twice daily, would have thought my vib files would have been about half the size, but they stayed consistent.
5. Windows reports drive is about 50% fragmented. I have gone through other threads on here, and it seems to be that this could be a source but also may not matter. Some things say that this causes small file changes to create large amounts of changed blocks, others say to not worry about defragging a disk any more.

I have not tried turning off CBT, as I can guess it would make the files much smaller, it is not really a valid solution. I have other servers that i back up such as exchange, database servers, and other domain controllers that do not have any where near the size of changed blocks that this one does. I am at a loss of what else to look at on this system. Any ideas would be great!

[Shestakov]

Hello Nick and welcome to the community!
Indeed small change causes the whole data block to be copied. Antivirus is only one of the possible reasons.
Please read the thread above and check if you have the same issue.

[ThaVWMan]

I believe I did find this thread as well before posting, but still have not been able to nail down what is causing the changes. I have not found any of the items in this current thread to be the culprit for the large backup files thus far. As far as I can tell, this disk should only have minor changes daily caused by regular AD activity. There are no other apps running on this system, and the file shares are on a different disk. The C: drive contains windows system files and the AD sysvol, that is it as far as I can tell. No other drive in my environment has any where near this level of changed blocks on a daily basis, including my SQL and Exchange systems. I dont get it, and this is becoming a huge issue as it is making daily offsite replication difficult. Can CBT be turned off for just one VM, or even just one disk in a VM, within a job that has other VM's in it? Or would I have to create a special non-cbt job just for this one system?

[veremin]

If you turn off CBT, our own proprietary mechanism will be used to determine changed blocks. Most likely, it will find the same amount of changed data, and increment similar in size will be created.

You can try to reset CBT, if you think it misbehaves. Otherwise, open a ticket with out support team. Through deeper investigation the root cause will be found.

Thanks.

[ThaVWMan]

How do you reset CBT, and by doing that, it will cause a full backup to be taken, correct?

[Shestakov]

Nick CBT reset instruction is here. It will not cause full backup, but the increment size will equal a size of full backup. You can perform full backup manually if you wish.

[Vsevolod Zubarev]

Resetting CBT isn't going to help since identical blocks are still to be filtered out by our own deduplication.

Common reasons for large incrementals:
- Windows Deduplication (not applicable in your case since it's a system volume)
- Periodically scheduled in-guest shadow copies.
- Potentially any other scheduled activities that you might've set up.

[Matts N]

Hello ThaVMMan,
A quick and maybe stupid question: is it possible that you by accident have enabled backing up the swap file? Normally this is excluded, but it could explain the large changes on disk.

// Matts