Comprehensive data protection for all workloads
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

Re: Troubleshooting backup size

Post by tomhkr »

From the logfile Agent.xx.Source.yy.vmdk.log I can see:
[26.05.2015 18:02:52] < 9904> dsk| Initializing swap CTK filter for disk 'VDDK:[datastore1] VM/VM01.vmdk'. Std. block size: '524288'.
...
[26.05.2015 18:12:26] < 17308> dsk| vSphere CTK cursor reached EOF. [48848] blocks were reported as updated.
So, 48848 CBT blocks of 524288 bytes size indeed makes up 23.9 GB of data, it appears correct.

Since actual modified files at the end of days was about 2 GB, how come CBT detects so many changed blocks?
Realizing this is not a Veeam question but someone in here might have a clue on why this occurs and how to minimize this.

I have two main suspects:
Microsoft Word periodically autosaving, meaning it creates a new file on a new block and deletes the old one.
Group Policies pushing out Internet Explorer favorites with the Replace option. 10 shortcuts and 100 clients updating every 90 minutes in an average of 8 hours workday would alone give 5000 updates.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Tom, please review the thread above for an answer. Thanks.
readie
Expert
Posts: 158
Liked: 30 times
Joined: Dec 05, 2010 9:29 am
Full Name: Bob Eadie
Contact:

[MERGED] Large Incremental Backup/Replica

Post by readie »

Hi,
We have a large fileserver (3TB) with a Backup and separate Replica each night. (I learned at VeeamON that I can do the replica from the backup files, and will be changing this soon!) The backup is reverse incremental.
Over the w/e both replica and backup are copying >500GB, and in an attempt to find out why, we have searched the drives for all files with modification date in the past three days, and there are only about 20GB of them.
Of course we are searching at the Windows level, not the VMware block level. CBT is enabled and Hotadd is being used.
Can anyone suggest other reasons why both the replica and the backup are incrementally copying this much data - it has really hit our backup window which still has several hours to run this morning, when it normally finishes about 4am (it's now about 9am UK time).
It's a puzzle rather than a serious problem, assuming it goes back to normal tomorrow.
thanks, Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Large Incremental Backup/Replica

Post by PTide »

Hi,

What is your target repo storage and what is your Guest OS? Is there any kind of deduplication enabled in OS/Repository?

Thank you.
readie
Expert
Posts: 158
Liked: 30 times
Joined: Dec 05, 2010 9:29 am
Full Name: Bob Eadie
Contact:

Re: Large Incremental Backup/Replica

Post by readie »

Repository is Windows, and so is the Guest OS. Both 2012 R2 and the Guest has dedupe set, but the repo does not (actually tried it once, but I think the 3TB files are too big for Windows dedeup?)
Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Large Incremental Backup/Replica

Post by PTide »

Such things as guest-OS-level deduplication, antivirus software, defragmentation and other things that can change blocks might be a cause of large incrementals. Please check if any of the mentioned is applicable to your environment.

Also please take a look at this post.

Thank you.
TrueAlex
Novice
Posts: 3
Liked: never
Joined: Nov 09, 2015 2:06 pm
Contact:

Re: Large Incremental Backup/Replica

Post by TrueAlex »

Are you using DFS inside guest?
readie
Expert
Posts: 158
Liked: 30 times
Joined: Dec 05, 2010 9:29 am
Full Name: Bob Eadie
Contact:

Re: Large Incremental Backup/Replica

Post by readie »

Ah - we did use DFS when we migrated the fileserver from an old VM (2008) to a new 2012 VM, and although we have switched off DFS pairing (as the old pair is dead), DFS service might still be running.
Have you experience that DFS might cause this?
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
TrueAlex
Novice
Posts: 3
Liked: never
Joined: Nov 09, 2015 2:06 pm
Contact:

Re: Large Incremental Backup/Replica

Post by TrueAlex »

DFS have some cache folders thats cause to make backup bigger, but if you dont use DFS now, probably its not your case.
readie
Expert
Posts: 158
Liked: 30 times
Joined: Dec 05, 2010 9:29 am
Full Name: Bob Eadie
Contact:

Re: Large Incremental Backup/Replica

Post by readie »

I think we'll just put it down to 'one of those things'. It's gone back to normal over the past few days, and hasn't done it before. Thanks for your suggestions.
Bob
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

[MERGED] Whats causing so much change

Post by Zew »

Hey all,

So I have a decent small environment, pretty well built and managed and I'm always trying to make it better. One thing I noticed odd in my backups is just how big the vibs were for a couple of my VM's.

In particular lets say a SharePoint Web Front End, usually very little changes on this server, also since the content for the server are all stored and backed up on a separate SQL server, so it's not that.

I backup this server only once a week since normally no changes are made to the layout of said content, however even in a span of a week with little AFAIK changes, the Data size is usually 25 - 50 Gigs with a Backup size with a ratio of roughly 1.9x. What could cause this much backup size with CBT enabled? Logs? I would understand windows updates would cause plenty of changed blocks, but that's only once a month.

Thoughts?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Multiple hints can be found in the thread above, as well as in this related thread.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Large VIB file on a small static server

Post by Zew »

Thanks for the merge, Sadly..

1) 2008 R2 (so no garbage collection.)

2) No AV (I have on most of my servers but removed from this one for some reason recently, so not that)

3) There are SharePoint Crawls taking place every night for indexing purposes. Again I figured most of this stuff/changes happen on the Database side, which is a different server. Unless this does similar to AV and changes Access times on a bunch of local files.

Sure is interesting.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Keep in mind that even a 1 bit change would require copying the entire 1 MB (by default) block.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Large VIB file on a small static server

Post by Gostev »

Could it be that those servers are simply very fragmented?
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Large VIB file on a small static server

Post by Zew »

They are fraged, I was going to clean that up when I figured out all the fun stuff about CBT and changed blocks and VM hole punching. Would really fragmented discs cause this? I figured it would cause this after I clean all the fragmentation up.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Fragmentation affects the amount of data to back up, due to the reason I've mentioned above - if the small changes are scattered along multiple blocks instead of being written into a single one, then more data needs to be read and transferred.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: Large VIB file on a small static server

Post by Zew »

Perfect thanks for all the info guys... defragging was in my "things to do" I'll report back what I find after its all done, I'm expecting a big VIB after all the defrag, and then hopefully smaller ones from then.
folerx
Expert
Posts: 113
Liked: 8 times
Joined: Jun 22, 2016 9:47 pm
Full Name: Daniel Kaiser
Contact:

Re: Large VIB file on a small static server

Post by folerx »

summary, what to do to make vib smaller as possible?
1. turn of last access time help?
2. turn off defragmenter help?
3. disable 8.3 name creation help?
4. ...
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Disabling/reducing any activity resulting in data change on the source VM will help to decrease the size of the increment.
GreenEnvy
Enthusiast
Posts: 25
Liked: 5 times
Joined: Jul 31, 2012 3:45 am
Full Name: Lee
Contact:

[MERGED] Windows server Dedupe and Veeam

Post by GreenEnvy »

This isn't an error or technical issue, more just general question.

We have Windows server 2012 r2 dedupe enabled on all of our file servers. We have Veeam setup using reverse incremental, keeping 14 restore points, running at 7pm nightly.
This has been working well for us for a few years now.

Every once in a while, we get a huge reverse incremental, and we're wondering if the Windows dedupe might be the cause.

For example, the file server that has this issue has 3 drives. 55GB OS drive, 1.8TB shared drive, and 2.5TB shared drive. The main backup file is right around 4TB for this server. Nightly restore points are usually in the 3-60GB range.
This past saturday night, the restore point is 861GB and took 33 hours to complete (normally under 2). We've looked into the antivirus on the server, but it didn't run any scans over the 48 hours leading upto this. We've looked at new/modified files on the server, but can only account for around 30GB of changes. We've had this happen a few times in the past, also seemingly on weekends when almost no one is working. It's only once every few months usually though, not a weekly thing.

Windows dedupe background optimization is set to constantly run in the background every day, all day. Throughput optimization runs nightly from 1am-5am. On Saturday and Sunday, it runs a garbage collection and a scrubbing jobs around 3am.
Looking at those dedupe logs, the garbage collection and scrubbing jobs only took a few minutes to run, so I can't see how they'd be creating so many changed blocks. The timing of it though is suspicious, since the garbage and scrubbing run on weekends, and these large files seem to happen on weekends.

I saw another thread speculating defragging might be causing it, or antivirus, or dedupe. I also saw an old best practices thread that said to use forward incrementals with Veeam if you have dedupe, but this seemed to be talking about deduping the Veeam backup files.

Does anyone else with a similar setup experience this? It's not a problem, but we'd love to know what's causing it.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Most likely Windows Deduplication activity (garbage collection/scrubbing tasks in particular) are indeed the culprits of the observed behavior.
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Windows server Dedupe and Veeam

Post by skrause » 1 person likes this post

The issue is likely being caused by the periodic garbage collection. Using forward incremental and setting the full to happen the next backup interval after the garbage collection is scheduled is the only way I have found to get around the large increment issue.

Setting your Veeam job to ignore deleted blocks helps quite a bit, but the chunklets that get cleaned up by windows dedupe still end up creating a lot of changed blocks for Veeam since they don't excatly align with the Veeam blocks so you get enough overlap that creates the need to back up blocks that have had some of their data change.
Steve Krause
Veeam Certified Architect
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Windows server Dedupe and Veeam

Post by Delo123 »

Well said Skrause :) Yes, that's the nature of deduping storage. Forward incremental and good timing is certainly the way to go....
tym
Enthusiast
Posts: 83
Liked: 9 times
Joined: Mar 26, 2015 9:30 pm
Full Name: Tim Diekhans
Contact:

[MERGED] why are incremental backups so big?

Post by tym »

Hi,

we have incremental backups that are between 200 and 500 gb per day. I cant believe that users change/create 500 gb of data per day, considered that we only backup changed blocks. Is there any chance to see the details/contents of the incremental backups?

Best,
Tim
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: why are incremental backups so big?

Post by DGrinev »

Hi Tim,

Incremental file size depends on different processes such as file relocation or defragmentation causes block modifications on the file system.
Please review this thread for more information.

Thanks!
StefanVM
Novice
Posts: 8
Liked: never
Joined: May 05, 2017 7:58 am
Full Name: Stefan B
Contact:

[MERGED] incremental backup files are huge

Post by StefanVM »

Hi dear Veeam community,

I have the issue that the incremental backup files of one of ours servers is extremely large.
It's a rather small server and the full backup size is about 40 GB. The incremental backup files are usually about 10 GB (going up to 20 GB on some days), which is a very large percentage.
For the other servers, the incremental backup file size is about 5% of the full backup, but depending on the servers also lower or higher on some days, rarely over 10%

The server with the large backup files doesn't have much activity, It is only used as a file server. As we made backups with a different system so far, the file size was never that large.
One aspect that may be important is that this is the only server having a VHD dynamic disk configuration. Could that be the reason for the problem?


- Stefan
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: incremental backup files are huge

Post by DGrinev »

Hi Stefan,

There are many reasons for an incremental file size changing such as dirty blocks, file relocation, fragmentation and other things that change data blocks.
Please review this discussion for additional information. Thanks!
Toschu
Lurker
Posts: 1
Liked: never
Joined: Aug 15, 2017 11:23 am
Contact:

[MERGED] Windows Guest Deduplication - Big Chunk Store Backu

Post by Toschu »

Hi Together,

we backing up our Fileserver which is a 2012 R2 VM on vSphere 6.5.

The whole partition is 2 TB.
The Chunk Store under "System Volume Information" has a size of 1,4TB.

Normally I would expect 20 GB of changed/new files on the server per day but if Dedup is running the amount of new data backups per day is 400+GB. Even when I just let run Dedup on the weekend it will be still in the week 200GB... and after the Full Dedup on the weekend 800+GB are transferred by Veeam.

When I exclude the Chunk Store Folder and set the retention policy of Dedup to one day the amount of to backup data seems normal, but with that deduped files older than one day are not possible to restore. ;-)

Does someone has an idea why is there so much change within Chunk Store that lets Veeam Backup so much? What's your experience with Dedup running inside a guest?
I'm thinking about moving the fileserver to a new 2016 and unoptimize the data and then again optimize them.

Thanks in advance,
Tobi
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Large VIB file on a small static server

Post by PTide »

Hi,

Generally speaking, dedupe activity changes blocks on a VMs harddrive thus causing incrementals to grow. Please review this thread for more insights.

Thank you.
Post Reply

Who is online

Users browsing this forum: a.sim and 154 guests