Comprehensive data protection for all workloads
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Server 2012 File Server random large backups on certain disk

Post by agossett »

Let me Start by saying this is probably one of the strangest IT problems I have ever had. I have been working on this issue with Veeam Support case 01093141.

We have a Server 2012 R2 file server that is roughly 5TB worth of deduped data crossed 9 different disk, we are running the latest version of Veeam B&R and this is a reverse incremental job. This all runs on a fully updated VMware environment.

My issue is this, after working on this case for about 4 months with Veeam support we have found that 2 of the 9 disk have a much larger then usual backup on the first Friday of every month, about 4-5 times larger then normal, with the latest one happening this past Friday New Years day. As this is a K-12 school I can pretty much guarantee that no one was here and accessing the file server. Now a 3rd disk has the same issue but it happens on the 2nd Friday of every month. We are not able to find anything that could be making this large of a data change and that would also be scheduled to run not only on those specific days but on the specific disk as well. We have checked anti virus software and it is an on access scanning based utility so it scans every new file that comes in as well as running a scan every night at midnight so we have ruled this out as a possible cause. We did have shadow copies set up on all of the disk, except for the C: drive, the tech thought that might be the issue so we disabled it of the first 2 drives, that did not change anything.

Has anyone seen anything like this before? The last thing that tech can think of is to find software that will allow me to view CBT data on the days that we are pretty confident at this point this is going to happen but I have not been able to find anything that allows you to do that and VMware does not have any such tool or even know of a 3rd party tool to do it.

Thank you,
Adam
tdewin
Veeam Software
Posts: 1775
Liked: 646 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Server 2012 File Server random large backups on certain

Post by tdewin »

Is the deduplication engine running on these inguest disks? I've met customers whose backups explode after enabling windows inguest deduplication.
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Server 2012 File Server random large backups on certain

Post by skrause »

Do you have a full garbage collection scheduled to run once a month? I have found that when I do a Start-DedupeJob GarbageCollection [VOLUME] -Full , it will "change" a large number of blocks as far as CBT is concerned.
Steve Krause
Veeam Certified Architect
widmerkarl
Expert
Posts: 122
Liked: 29 times
Joined: Jan 06, 2015 10:03 am
Full Name: Karl Widmer
Location: Switzerland
Contact:

Re: Server 2012 File Server random large backups on certain

Post by widmerkarl »

Hi Adam,

I stumbled across this thread. First i wasn't sure if this could be an issue. But the more i think about, the more i think this could be something.

I have something in mind about free space reclamation on virtual servers. If there were deleted many files (big files or thousands of small files for example) the free space is not really free on the LUN.

https://kallesplayground.wordpress.com/ ... ion-part1/

Perhaps it helps to clean up here.
Best regards,
Karl

-----------------------
vExpert 2017-2024
VMware VCP-DCV 2023 / VCA6-DCV / VCA5-DCV / VCA5-Cloud
Former Veeam Vanguard / VMCE v9 / VMTSP v9 / VMSP v9
Personal blog: https://www.driftar.ch
Twitter: @widmerkarl
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

Timothy,

The dedup has been running since the beginning of August and this problem started in October. Do you think this could still be a side effect of that? When we first started enabled dedup we did see pretty large backups but we have got to be passed that point by now, right?

Steve,

I will have to look into that, I have not dealt with any of the dedup here so this vary well could be the cause. Just one question about this, shouldn't it be happening on all of the disk that dedup is enabled on?

Karl,

I will take a look at that.

Thank you all!
Adam
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

skrause wrote:Do you have a full garbage collection scheduled to run once a month? I have found that when I do a Start-DedupeJob GarbageCollection [VOLUME] -Full , it will "change" a large number of blocks as far as CBT is concerned.
I just checked and we have a garbage collection set to run weekly on Saturdays at 2 am. unfortunately I don't think this is what our issue is
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Server 2012 File Server random large backups on certain

Post by skrause »

I have found that if the volumes that you are deduplicating with Windows Dedupe have fairly high change rates when the dedupe runs it will actually make the number of changed blocks that CBT sees increase quite a bit (we have a DFS group we use for all of our SQL backups which I see this issue on, since we get a lower than expected dedupe rate I am going to be disabling deduplication on those volumes soon).

Are there a lot of active writes/deletes on the volume in question? Is there maybe a batch process that dumps a lot of files at the end of the month to those volumes?
Steve Krause
Veeam Certified Architect
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

The dedup is set up to dedup files that are older then 3 days so it should be make small changes probably every day or so I would imagine?

There shouldn't be hardly any writes/deletes to at least one of the first 2 disk specially this past Friday because that disk is for the students home drives and they where gone for the past 2.5 weeks.

If there is a batch process i have no idea where it is or how it would have been set up as all of the drives in question cant not be accessed by the same person and even if they did I do not know where they would get enough data to cause such a change.

Our backups are generally 10-20gb on average for each disk during the week. But when that first Friday comes the backup size shoots up to 300+ GB.

Is there a script that I could run on Saturday that would show me all of the new, changed and delete files on the disk in question?
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Did you check if the default defrag job is disabled on the deduped volumes? I believe it's right under the dedup tasks.... If so you could also disable dedup tasks for an amount of time to see if this has any effect... Ps. How do you mean deduped accross 9 disks? Hoe is it setup? Did you create a storage pool, dynamic disk or multiple volumes? Ps even if you schedule your garbage collection & purge jobs at a fixed schedule this doesn't mean they will actually run at that time since all dedup jobs are queued, so if you have 7 optimize tasks scheduled during a week it could be mondays job is still running wednesday if you allow it to... You can check active dedup jobs in an elevated powershell get-dedupjobs , also be sure to check get-dedupstatus to see how many files are actually optimized etc...
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

The Defrag has been disabled and it has never ran. We have 9 separate disks setup each with there own volume, we have dedup set up on each of the volumes except for the C: drive. At the current time there are no active dedup jobs and based on how quickly dedup ran when it was enabled (we did not have it enabled on any of our drives, some of which where over 1TB) I would be shocked if the jobs are actually exactly a month to run through.

case ID 01093141
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

You don't happen to have anothet legacy backup product running like windows backup? Or something which resets archive attribute or something? Otherwise it would probably useful to install some file monitoring utility!? But that also wouldn't help you if it's something underlying in the filesystem....

Ps why do you have the 9 disks all seperated? Why no raid / storage space or something? Having 9 seperate disks doesn't give you any redundancy nor can you utilize the combined speed of all of them (striping), there is nothing wrong with having multiple volumes however :)
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Maybe a stupid question, but have you tried "normal" forward incrementals? Maybe it's worth switching to test (at least if there is no specific reason to do reverse...)
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

Nothing, Veeam is the only backup product that has ever been used on this server. There is very very little software on this server so I don't have any idea what could be possibly doing that. Do you have any suggestions on file monitoring software?

I find it pretty odd as well, it was setup this way before I started here and I figure if its not broke don't fix it, right? We do not have raid or storage spaces setup because its a vm that resides on a raided storage system.

Guido,

I wish I could have forward incrementals but unfortunately we do not have enough storage space on the backup targets to be able to keep the retention policy we want.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

I wish I could have forward incrementals but unfortunately we do not have enough storage space on the backup targets to be able to keep the retention policy we want
Were are you storing your backups on? If it's a windows machine, turn on dedupe :):)

You could give Nirsoft's FolderChangesView a try, it's free, small, no bloatware as everything from nirsoft is kinda cool and at least should give you some hints if anything is going on on filesystem level... Just download, run, enter the disks you want to monitor and maybe check the option the export a log every x minutes / hours...

Then get some Coffee and relax :)
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

We do not have raid or storage spaces setup because its a vm that resides on a raided storage system.
Ps. I kinda forgot we are talking about a vm :)
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Uhh today Is the first Friday of the month, maybe you can quickly install it and catch something...
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

Not a windows machine :/ and thank you for that suggestion. I currently have it running.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

How did it go? Again a big backup?
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

You guessed it. the Log file from that program ran but nothing in it looks like it would have caused this issue.

There are a lot of changes related to dedup but there are also a very similar amount of changes on other days so I don't think that is causing it.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Hmm ok... Well dedupe blocks do not change if no dedupe jobs are running, but maybe you see reads from the dedupe chunks (reads ofcourse do not come the files directly, but from the deupded chunks.....)... Strange... Are you sure defrag is disabled? (it's in the task scheduler, but not in the overview but microsoft -> windows -> deduplication), but anyway, the default schedule is once a week and not monthly....
Very strange... Did you already have somebody else have a fresh look at the system? Maybe it's something silly and your just overlooking, stupid issue :(
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

It is definitely disabled. And i think I am up to a total of 6 different people looking at it. it is a very strange issue and i have no doubts that it is something silly and stupid :/ Veeam is currently going over the log from the file monitoring program. Hopefully they can find something I missed!
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Sigh... If nothing works you of course could do a reinstall and leave the data in place (if you do not have a lot of applications running on that thing of course)... Anyway, hope they find something. Keep us updates & good luck!
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

On last idea... Did you try disabling CBT? Obviously incrementals will take much longer, but maybe it could show if it is caused by CBT or actual data changes?!
VladV
Expert
Posts: 224
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Server 2012 File Server random large backups on certain

Post by VladV »

Do you by any chance have Shadow copies scheduled for these disks?
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Adam already wrote he had disabledshadow copies for those volumes :(
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Server 2012 File Server random large backups on certain

Post by mkaec »

I think it is the dedup garbage collection that it causing this problem for you. I tracked this on our systems when I was testing Veeam with Windows dedup. Windows dedup stores the dedup blocks in 1 GB .ccc files in \System Volume Information\Dedup\ChunkStore. The actual files on a dedup volume are mostly empty. It doesn't matter if the files don't change. The garbage collection routine can make changes to the chunk store as it sees fit. It's correct for Veeam to see this as changed blocks and back them up. The problem with Veeam is that there's no way for the user to see what changed in a particular backup. (Though v9 is now looking at the NTFS MFT for a different feature. So, maybe a future version will add better visibility into the contents of a backup file.)

This is one disadvantage of block-based backup. A traditional backup application would ignore the chunk store and only operate on the actual files. Of course, traditional file-based backup has a different set of problems.

There is a scheduled task "Microsoft\Windows\Deduplication\Weekly Garbage Collection" that runs every week by default. You could try changing the schedule to monthly to see if the large Veeam incrementals occur less frequently. Or you could just turn off dedup if you have the space to spare. Or you could set up a repository that is also running Windows dedup (and turn off Veeam compression). I've found that the Veeam incrementals inflated by Windows dedup on the source dedup nicely on a repository also running Windows dedup.
agossett
Enthusiast
Posts: 34
Liked: 1 time
Joined: Apr 17, 2015 12:23 pm
Full Name: Adam Gossett
Contact:

Re: Server 2012 File Server random large backups on certain

Post by agossett »

If it was the weekly garbage collection shouldn't this happen every week? I also have dedup running on other disk but they do not have this same problem.

I really wish i could temporally turn dedup off but i don't have the disk space at the moment.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2012 File Server random large backups on certain

Post by Delo123 »

Adam,

earlier you wrote you had disabled all dedupjobs. So if optimization / garbage collection / Scrubbing jobs are disabled it wouldn't make sense...
If they however are not disabled, do it for testing. (until your next Fridays first!) Dedup itself will still de enabled, there will just be no further backup jobs and there will be no issue with these jobs disabled
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Server 2012 File Server random large backups on certain

Post by mkaec »

It wouldn't happen every week because garbage collection isn't guaranteed to make large changes every week. To test, just disable the "Weekly Garbage Collection" and "WeeklyScrubbing" scheduled tasks for a few weeks. New data will still dedup, but deleted data won't clear out effectively with the tasks disabled.
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Server 2012 File Server random large backups on certain

Post by skrause »

Make sure you have disabled defragmentation/optimization for that drive in the OS. I had a server that was having much larger increments than normal every wednesday night and I found in the task scheduler that defragmentation was running every wednesday at 1am so when my backups ran at 9pm they had a lot more changed blocks than before.
Steve Krause
Veeam Certified Architect
Post Reply

Who is online

Users browsing this forum: Google [Bot], Gostev and 192 guests