From the logfile Agent.xx.Source.yy.vmdk.log I can see:
[26.05.2015 18:02:52] < 9904> dsk| Initializing swap CTK filter for disk 'VDDK:[datastore1] VM/VM01.vmdk'. Std. block size: '524288'.
...
[26.05.2015 18:12:26] < 17308> dsk| vSphere CTK cursor reached EOF. [48848] blocks were reported as updated.
So, 48848 CBT blocks of 524288 bytes size indeed makes up 23.9 GB of data, it appears correct.
Since actual modified files at the end of days was about 2 GB, how come CBT detects so many changed blocks?
Realizing this is not a Veeam question but someone in here might have a clue on why this occurs and how to minimize this.
I have two main suspects:
Microsoft Word periodically autosaving, meaning it creates a new file on a new block and deletes the old one.
Group Policies pushing out Internet Explorer favorites with the Replace option. 10 shortcuts and 100 clients updating every 90 minutes in an average of 8 hours workday would alone give 5000 updates.
Hi,
We have a large fileserver (3TB) with a Backup and separate Replica each night. (I learned at VeeamON that I can do the replica from the backup files, and will be changing this soon!) The backup is reverse incremental.
Over the w/e both replica and backup are copying >500GB, and in an attempt to find out why, we have searched the drives for all files with modification date in the past three days, and there are only about 20GB of them.
Of course we are searching at the Windows level, not the VMware block level. CBT is enabled and Hotadd is being used.
Can anyone suggest other reasons why both the replica and the backup are incrementally copying this much data - it has really hit our backup window which still has several hours to run this morning, when it normally finishes about 4am (it's now about 9am UK time).
It's a puzzle rather than a serious problem, assuming it goes back to normal tomorrow.
thanks, Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
Repository is Windows, and so is the Guest OS. Both 2012 R2 and the Guest has dedupe set, but the repo does not (actually tried it once, but I think the 3TB files are too big for Windows dedeup?)
Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
Such things as guest-OS-level deduplication, antivirus software, defragmentation and other things that can change blocks might be a cause of large incrementals. Please check if any of the mentioned is applicable to your environment.
Ah - we did use DFS when we migrated the fileserver from an old VM (2008) to a new 2012 VM, and although we have switched off DFS pairing (as the old pair is dead), DFS service might still be running.
Have you experience that DFS might cause this?
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
I think we'll just put it down to 'one of those things'. It's gone back to normal over the past few days, and hasn't done it before. Thanks for your suggestions.
Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
So I have a decent small environment, pretty well built and managed and I'm always trying to make it better. One thing I noticed odd in my backups is just how big the vibs were for a couple of my VM's.
In particular lets say a SharePoint Web Front End, usually very little changes on this server, also since the content for the server are all stored and backed up on a separate SQL server, so it's not that.
I backup this server only once a week since normally no changes are made to the layout of said content, however even in a span of a week with little AFAIK changes, the Data size is usually 25 - 50 Gigs with a Backup size with a ratio of roughly 1.9x. What could cause this much backup size with CBT enabled? Logs? I would understand windows updates would cause plenty of changed blocks, but that's only once a month.
2) No AV (I have on most of my servers but removed from this one for some reason recently, so not that)
3) There are SharePoint Crawls taking place every night for indexing purposes. Again I figured most of this stuff/changes happen on the Database side, which is a different server. Unless this does similar to AV and changes Access times on a bunch of local files.
They are fraged, I was going to clean that up when I figured out all the fun stuff about CBT and changed blocks and VM hole punching. Would really fragmented discs cause this? I figured it would cause this after I clean all the fragmentation up.
Fragmentation affects the amount of data to back up, due to the reason I've mentioned above - if the small changes are scattered along multiple blocks instead of being written into a single one, then more data needs to be read and transferred.
Perfect thanks for all the info guys... defragging was in my "things to do" I'll report back what I find after its all done, I'm expecting a big VIB after all the defrag, and then hopefully smaller ones from then.
summary, what to do to make vib smaller as possible?
1. turn of last access time help?
2. turn off defragmenter help?
3. disable 8.3 name creation help?
4. ...
This isn't an error or technical issue, more just general question.
We have Windows server 2012 r2 dedupe enabled on all of our file servers. We have Veeam setup using reverse incremental, keeping 14 restore points, running at 7pm nightly.
This has been working well for us for a few years now.
Every once in a while, we get a huge reverse incremental, and we're wondering if the Windows dedupe might be the cause.
For example, the file server that has this issue has 3 drives. 55GB OS drive, 1.8TB shared drive, and 2.5TB shared drive. The main backup file is right around 4TB for this server. Nightly restore points are usually in the 3-60GB range.
This past saturday night, the restore point is 861GB and took 33 hours to complete (normally under 2). We've looked into the antivirus on the server, but it didn't run any scans over the 48 hours leading upto this. We've looked at new/modified files on the server, but can only account for around 30GB of changes. We've had this happen a few times in the past, also seemingly on weekends when almost no one is working. It's only once every few months usually though, not a weekly thing.
Windows dedupe background optimization is set to constantly run in the background every day, all day. Throughput optimization runs nightly from 1am-5am. On Saturday and Sunday, it runs a garbage collection and a scrubbing jobs around 3am.
Looking at those dedupe logs, the garbage collection and scrubbing jobs only took a few minutes to run, so I can't see how they'd be creating so many changed blocks. The timing of it though is suspicious, since the garbage and scrubbing run on weekends, and these large files seem to happen on weekends.
I saw another thread speculating defragging might be causing it, or antivirus, or dedupe. I also saw an old best practices thread that said to use forward incrementals with Veeam if you have dedupe, but this seemed to be talking about deduping the Veeam backup files.
Does anyone else with a similar setup experience this? It's not a problem, but we'd love to know what's causing it.
The issue is likely being caused by the periodic garbage collection. Using forward incremental and setting the full to happen the next backup interval after the garbage collection is scheduled is the only way I have found to get around the large increment issue.
Setting your Veeam job to ignore deleted blocks helps quite a bit, but the chunklets that get cleaned up by windows dedupe still end up creating a lot of changed blocks for Veeam since they don't excatly align with the Veeam blocks so you get enough overlap that creates the need to back up blocks that have had some of their data change.
we have incremental backups that are between 200 and 500 gb per day. I cant believe that users change/create 500 gb of data per day, considered that we only backup changed blocks. Is there any chance to see the details/contents of the incremental backups?
Incremental file size depends on different processes such as file relocation or defragmentation causes block modifications on the file system.
Please review this thread for more information.
I have the issue that the incremental backup files of one of ours servers is extremely large.
It's a rather small server and the full backup size is about 40 GB. The incremental backup files are usually about 10 GB (going up to 20 GB on some days), which is a very large percentage.
For the other servers, the incremental backup file size is about 5% of the full backup, but depending on the servers also lower or higher on some days, rarely over 10%
The server with the large backup files doesn't have much activity, It is only used as a file server. As we made backups with a different system so far, the file size was never that large.
One aspect that may be important is that this is the only server having a VHD dynamic disk configuration. Could that be the reason for the problem?
There are many reasons for an incremental file size changing such as dirty blocks, file relocation, fragmentation and other things that change data blocks.
Please review this discussion for additional information. Thanks!
we backing up our Fileserver which is a 2012 R2 VM on vSphere 6.5.
The whole partition is 2 TB.
The Chunk Store under "System Volume Information" has a size of 1,4TB.
Normally I would expect 20 GB of changed/new files on the server per day but if Dedup is running the amount of new data backups per day is 400+GB. Even when I just let run Dedup on the weekend it will be still in the week 200GB... and after the Full Dedup on the weekend 800+GB are transferred by Veeam.
When I exclude the Chunk Store Folder and set the retention policy of Dedup to one day the amount of to backup data seems normal, but with that deduped files older than one day are not possible to restore.
Does someone has an idea why is there so much change within Chunk Store that lets Veeam Backup so much? What's your experience with Dedup running inside a guest?
I'm thinking about moving the fileserver to a new 2016 and unoptimize the data and then again optimize them.
Yes, that's the nature of deduping storage. Forward incremental and good timing is certainly the way to go....