Large VIB file on a small static server

[Yuki]

this is driving us crazy.

On our 10TB server, last friday's backup is 1.6TB again! Even though only about 150GB of files changed friday AND saturday. How can we find out what is changing? We know users didn't type up 1.6TB worth of new documents, and our space usage did not increase 1.6tb.

What's going on? How can we figure out what is happening?

Win 2012 Standard
File server
No Anti-virus
No defrag

[Gostev]

Is it possible that the file server's file system is fragmented? In this case, changing 1MB file can easily results in 10 1MB blocks changed (10x of the actual file size).

[tsightler]

Aren't you running dedupe? When are you running chunk store scrubbing and garbage collection? Those processes will make a large number of changes across the entire disks. I think the default is Saturday morning (or Friday night, based on how you look at it) at 2:45AM for Scrubbing, and 3:45AM for Garbage Collection. You can use "Get-DedupSchedule" to see the schedule. That's really the most likely culprit and is simply a side effect of running dedupe. What is the size of a more typical night?

[Yuki]

A typical day would produce between 100-300GB.

It could be fragmentation mixed with dedupe, but at this point we are moving away from veeam as vmware level backups are not a good solution for this system. Not an issue with Veeam directly, just a bad match of existing needs and technologies involved.

[pkelly_sts]

I have a file server that only generates around 1gb new data a day maximum but which is generating 36Gb+ incremental backup sizes and I can't figure out why.

There's no A/V installed on this server so it isn't an AV scan modifying last accessed timestamps
There's no defrag job scheduled/running

Because it's veeam that's effectively highlighting to me that /something/ is changing somewhere, thus causing a lot of changed blocks, it's a little difficult to search on in the big-wide-t'internet.

Does anyone have any other clues they could point me towards?

Paul

[foggy]

Paul, this topic should give you some hints to check. Though, feel free to ask for additional clarification, if required after reviewing it. Thanks.

[eric]

Hello All,

Running VBR 7.0 R2 as a test.

I just backed up a domain controller, which has very low volume, subsequent incremental backups are around 600MB.

That seems too large of a change in just a few hours.

Any thoughts?

~eric

[veremin]

Hi, Eric,

Your post has been merged into existing discussion regarding similar issue; so, please, take a look at the answers provided above. In general, VMs running special applications such as DC, SQL, Exchange are notoriously known for producing large incremental files. Another possible reasons include regular anti-virus scans or de-fragmentation activity.

Thanks.

[Gostev]

Antivirus scans are read only, so I don't see how they can introduce disk changes?

[veremin]

As far as I know, some anti-virus program might change archive bit of a file, or file attributes, in general. This results in increased amount of changes. Some users have also claimed previously that disabling antivirus activity resulted in decreased incremental files.

[dellock6]

This sounds really strange to me too. Antivirus are supposed to be a problem for IOPS, but not for changing files. In this way, an antivirus can also cause problems to replication or backup software installed at the guest OS level, and is a really bad behaviour...

Luca.

[pkelly_sts]

Just as an addendum (apologies if already covered in this thread, I've not read through it all) I was recently looking into a similar issue on one of our servers and I think I've found the culprit to be DFS replication. Even though we have a 4Gb DFS cache thingy configured (can't remember proper name right now) I was watching the cache whilst trouble-shooting & noticed it create a 15Gb temp file there which it repeated a few times.

I've not /confirmed/ this to be the cause but I strongly suspect it to be a big contributor in my case...

Paul

[speedline]

I have a job that backs up 7 virtual machines from ESX. It takes around 12 hours to replicate them off site with "optimal" compression. So far it's 12% through the second to last virtual machine and it says that it's 65% complete and has already moved 35.6 GB of data. The servers it's backing up don't change a heck of a lot, or at least I don't think they do. I've been trying to find way of determining what's changed. Initially I had thought I'd solved the problem with all my SQL servers making backups on themselves (the setup before we had Veeam) but apparently there's still a lot of block change. Any way to track and record what's changing, or which server is causing the most data change?

[Gostev]

One thing you can do is disable compression, create a test job that uses incremental backup mode, and try to make sense out of the raw content of VIB file beyond the first 8MB. May be you will be able to recognize the content as belonging to some application.

[Eagleman]

I;ve been using Veeam for over a month and i really like it, i was using ghettovcb before and this felt like a huge step in time.

I do however have one problem.
Whenever i backup my "production machines" i also backup the machines that has Veeam Installed (Windows Server 2012)

Here is a picture of the backup results:
[EDIT] We had to delete a reference to the picture, because Google Chrome started blocking this page due to the image sharing provider chosen by the poster. Sorry about that.

As you can see the Windows Server is eating up 75% of the total backup.
This Windows Server is only using these roles: Ative Directory and DNS.
I think Veeam is somehow changing a lot of data during backups and then the incremental backups will be very large only because of Veeam.

Can i somehow stop these incremental files being so large?

[Vitaliy S.]

Veeam does not modify any blocks, so it cannot influence the size of the incremental files of the VMs you protect. Please review this topic for possible reasons of this high change rate. Thanks!

[blodsbror]

Hi,

I have a job with 12 Windows 2003 servers (IIS webservers) that each have a 250GB 'data' disk comprosed of website data. For each incremental backup with CBT enabled, it reads about 1TB of data and generates around 400GB of backup data on average. This number seems really high, as there absolutely wouldn't be that amount of changed data on these servers each day. Even when I run an incremental straight after the last, approximately the same amount is backed up each time - when really the number should be very low (such as 1-10GB).

What could be causing this large incremental backup size ? I know there isn't that amount of changes in the data - so is there any tips to improving this ?. At this rate, I'll need a huge amount of storage to have just 2 weeks retention.

many thanks!

[foggy]

Evan, please review this thread, probably will give you some hints to justify the amount of changes.

[metrobg]

We are using VMware ESXI 5.1u1 as part of a disaster recovery system for a client. We are currently using VEEAM Incrementals to backup the data to a local data store.

VEEAM is backing up 30 - 40 GB a day, which I believe is based on VMware CBT, on one of the VMs. This VM is a file & Exchange server.

What would be useful that we have not found yet is a program that will identify which programs on the Host OS (Windows 2008) that are creating all of the changed blocks. Does anyone know of such a program?


We have made a copy of the entire environment in our lab and with no users accessing the file/exchange server we get similar amounts of data that needs to be backed up.

Any Help is appreciated.

[foggy]

Gilbert, please review this existing thread for some hint on what could cause such amount of changed blocks. Also, since Veeam B&R is a block-level solution, there's no ability to correlate those blocks to the applications inside the guest OS.

[Gostev]

Exchange and SQL servers generate tons of transaction logs, which is the main reason for large incremental backup size. Plus, both make many small and quite random updates of their respective database files, which also contributes to the amount of changed blocks.

[ColtsFanMN]

I'm having an issue with my backup file sizes.. it's not an error yet, but at this rate, I will be out of disk space soon.

I am running B&R 6.5 right now, full backup each day, in a VMware environment to removable media.

My backup sizes are 'out of range' for the actual size of the data on the drives, even without deduplication (which I am using).

All the servers are on 'thick provisioned' drives in VMware. There are no 'orphaned' snapshot files present.

For instance, this are what I am seeing:

Server 1:
Veeam shows:
Drive size: 225GB
Read size: 231GB
Transferred: 126GB
Server shows:
Drive size: 225GB
Used size: 76GB

So VEEAM is showing transferring 50GB more data than is actually present on the drive, even without deduplication.

Server 2:
Veeam shows:
Drive size: 525GB
Read size: 523GB
Transferred: 326GB
Server shows:
Drive size: 525GB
Used size: 290GB

So VEEAM is showing transferring 35GB more data than is actually present on the drive, even without deduplication.

I have 13 VMs backing up, all with similar numbers.. How is VEEAM backing up more than is actually present on the servers? Also, I will note that I deleted 30GB of old files off the first server I listed, this made NO change in my backup file size the next day (or even weeks later). What is going on?

Please help!

[Vitaliy S.]

Hello,

Please take a look at the third page of this topic, where Tom explained the behavior you most likely experience.

Thank you!

[ColtsFanMN]

Hello,

Please take a look at the third page of this topic, where Tom explained the behavior you most likely experience.

Thank you!

NO THIS DOES NOT APPLY. PLEASE READ MY POST AND RETURN IT AS A SEPERATE POST!

THANKS.

[Vitaliy S.]

Also, I will note that I deleted 30GB of old files off the first server I listed, this made NO change in my backup file size the next day (or even weeks later).

The reason why your post has been merged to this thread can be found in this quote:

If you create a 40GB virtual disk, add 20GB of data to it, and then delete it, then take a Veeam backup, the Veeam increment will 20GB (assuming no compression)..... For example, I was once able to use a Veeam backup to recover a file that had been created and deleted on the same day by using an NTFS "undelete" program on a Veeam backup from a week earlier. This is something that would have been completely not possible with a traditional backup product. Also, a Veeam backup of a powered off VM is a forensically accurate image of the system, which can be very valuable for security related analysis.

Please be aware that with NTFS, when you write data to the disk and then delete it, the data doesn't actually get removed from the disk, so virtual disk blocks still contain the same dirty data blocks. If you want to make your full backups smaller, then you should sdelete your virtual disks. Please search these forums for existing topics describing how to use this utility.

NO THIS DOES NOT APPLY. PLEASE READ MY POST AND RETURN IT AS A SEPERATE POST!

It seems like the explanation above about "dirty" blocks on NTFS does apply to your case. Thanks!

[out2getyou]

Hi All,

I currently have two mailbox servers in a DAG. I created 1 job that contains both passive and active mailbox servers. The initial full backup (which contains both servers) was about 1.31 TB. Every increment after that (1 daily incremental) for 1 week varied between 70-100 GB (both servers). So the obvious question is, why are the incremental backups so big? Surely we cannot be generating 30-50 GBs of email per day? we only have 200+ staff . Thats close to 5-7 GB per person!

Is there a way to make these incremental smaller? its eating up my storage

Thanks guys.

[Dima P.]

Hello,
I’ve merged your post to the existing discussions. If you suspect increment growing unpredictably large, please review this thread for possible solution. Thank you.

[deduplicat3d]

I have an open case 00746249

They are insisting that I am changing files and that is why I have large incrementals. I had the problem saturday morning and tuesday morning, these are forever incremental backups. Most VM's appears not to have this problem but 2, that are on separate hosts separate domains have the issue. Some jobs are not impacted at all. I have not done any defrag, disk check, virus scanner hasn't changed settings. I've browsed throughout the servers and almost all of the files have old modified and accessed times (it's a file server, people are usually only changing and accessing current documents).

I can't have this happen again as it will fill up the disk on the backup server. Is there a way to cross reference the blocks that have changed with files or to do some sort of analysis. I uploaded logs to support and a bunch of screen shots.

[foggy]

As it is mentioned above, there's no ability to correlate the changed blocks with files. What you could do to verify what's changed, is fire up two restore points using Instant Recovery and compare files between them. There's also a couple of other guesses in the referred thread, might give you some thoughts.

[tomhkr]

I have a virtual Windows 2012 R2 server acting as a file server for users home directories.

Yesterday I looked at the backup sizes this server generates and they are questionable large.
The volume being backed up is sized at 2.3 TB and has about 3,600,000 files on it.
It is a NTFS volume with an allocation unit size of 4096 bytes. Residing on an VMFS-volume with block size 1 MB.

Yesterdays incremental backup of the volume, using CBT, was 23.9 GB.

Looking at all files that were either modified or changed yesterday, they are only about 2 GB in total.
Total of about 3500 files were created or modified, of these about 3100 were smaller than 64 KB and 2500 smaller than 4 KB.
I understand that CBT takes whole changed blocks but this is ridiculous. Even with an assumed CBT block size of 1024 KB; 3500 files would be 3.5 GB, not 23.9 GB.

How can I troubleshoot and find out what is causing this massive leverage in backup size?