Server 2012 File Server random large backups on certain disk

[agossett]

Let me Start by saying this is probably one of the strangest IT problems I have ever had. I have been working on this issue with Veeam Support case 01093141.

We have a Server 2012 R2 file server that is roughly 5TB worth of deduped data crossed 9 different disk, we are running the latest version of Veeam B&R and this is a reverse incremental job. This all runs on a fully updated VMware environment.

My issue is this, after working on this case for about 4 months with Veeam support we have found that 2 of the 9 disk have a much larger then usual backup on the first Friday of every month, about 4-5 times larger then normal, with the latest one happening this past Friday New Years day. As this is a K-12 school I can pretty much guarantee that no one was here and accessing the file server. Now a 3rd disk has the same issue but it happens on the 2nd Friday of every month. We are not able to find anything that could be making this large of a data change and that would also be scheduled to run not only on those specific days but on the specific disk as well. We have checked anti virus software and it is an on access scanning based utility so it scans every new file that comes in as well as running a scan every night at midnight so we have ruled this out as a possible cause. We did have shadow copies set up on all of the disk, except for the C: drive, the tech thought that might be the issue so we disabled it of the first 2 drives, that did not change anything.

Has anyone seen anything like this before? The last thing that tech can think of is to find software that will allow me to view CBT data on the days that we are pretty confident at this point this is going to happen but I have not been able to find anything that allows you to do that and VMware does not have any such tool or even know of a 3rd party tool to do it.

Thank you,
Adam

[tdewin]

Is the deduplication engine running on these inguest disks? I've met customers whose backups explode after enabling windows inguest deduplication.

[skrause]

Do you have a full garbage collection scheduled to run once a month? I have found that when I do a Start-DedupeJob GarbageCollection [VOLUME] -Full , it will "change" a large number of blocks as far as CBT is concerned.

[widmerkarl]

Hi Adam,

I stumbled across this thread. First i wasn't sure if this could be an issue. But the more i think about, the more i think this could be something.

I have something in mind about free space reclamation on virtual servers. If there were deleted many files (big files or thousands of small files for example) the free space is not really free on the LUN.

https://kallesplayground.wordpress.com/ ... ion-part1/

Perhaps it helps to clean up here.

[agossett]

Timothy,

The dedup has been running since the beginning of August and this problem started in October. Do you think this could still be a side effect of that? When we first started enabled dedup we did see pretty large backups but we have got to be passed that point by now, right?

Steve,

I will have to look into that, I have not dealt with any of the dedup here so this vary well could be the cause. Just one question about this, shouldn't it be happening on all of the disk that dedup is enabled on?

Karl,

I will take a look at that.

Thank you all!
Adam

[agossett]

Do you have a full garbage collection scheduled to run once a month? I have found that when I do a Start-DedupeJob GarbageCollection [VOLUME] -Full , it will "change" a large number of blocks as far as CBT is concerned.

I just checked and we have a garbage collection set to run weekly on Saturdays at 2 am. unfortunately I don't think this is what our issue is

[skrause]

I have found that if the volumes that you are deduplicating with Windows Dedupe have fairly high change rates when the dedupe runs it will actually make the number of changed blocks that CBT sees increase quite a bit (we have a DFS group we use for all of our SQL backups which I see this issue on, since we get a lower than expected dedupe rate I am going to be disabling deduplication on those volumes soon).

Are there a lot of active writes/deletes on the volume in question? Is there maybe a batch process that dumps a lot of files at the end of the month to those volumes?

[agossett]

The dedup is set up to dedup files that are older then 3 days so it should be make small changes probably every day or so I would imagine?

There shouldn't be hardly any writes/deletes to at least one of the first 2 disk specially this past Friday because that disk is for the students home drives and they where gone for the past 2.5 weeks.

If there is a batch process i have no idea where it is or how it would have been set up as all of the drives in question cant not be accessed by the same person and even if they did I do not know where they would get enough data to cause such a change.

Our backups are generally 10-20gb on average for each disk during the week. But when that first Friday comes the backup size shoots up to 300+ GB.

Is there a script that I could run on Saturday that would show me all of the new, changed and delete files on the disk in question?

[Delo123]

Did you check if the default defrag job is disabled on the deduped volumes? I believe it's right under the dedup tasks.... If so you could also disable dedup tasks for an amount of time to see if this has any effect... Ps. How do you mean deduped accross 9 disks? Hoe is it setup? Did you create a storage pool, dynamic disk or multiple volumes? Ps even if you schedule your garbage collection & purge jobs at a fixed schedule this doesn't mean they will actually run at that time since all dedup jobs are queued, so if you have 7 optimize tasks scheduled during a week it could be mondays job is still running wednesday if you allow it to... You can check active dedup jobs in an elevated powershell get-dedupjobs , also be sure to check get-dedupstatus to see how many files are actually optimized etc...

[agossett]

The Defrag has been disabled and it has never ran. We have 9 separate disks setup each with there own volume, we have dedup set up on each of the volumes except for the C: drive. At the current time there are no active dedup jobs and based on how quickly dedup ran when it was enabled (we did not have it enabled on any of our drives, some of which where over 1TB) I would be shocked if the jobs are actually exactly a month to run through.

case ID 01093141

[Delo123]

You don't happen to have anothet legacy backup product running like windows backup? Or something which resets archive attribute or something? Otherwise it would probably useful to install some file monitoring utility!? But that also wouldn't help you if it's something underlying in the filesystem....

Ps why do you have the 9 disks all seperated? Why no raid / storage space or something? Having 9 seperate disks doesn't give you any redundancy nor can you utilize the combined speed of all of them (striping), there is nothing wrong with having multiple volumes however

[Delo123]

Maybe a stupid question, but have you tried "normal" forward incrementals? Maybe it's worth switching to test (at least if there is no specific reason to do reverse...)

[agossett]

Nothing, Veeam is the only backup product that has ever been used on this server. There is very very little software on this server so I don't have any idea what could be possibly doing that. Do you have any suggestions on file monitoring software?

I find it pretty odd as well, it was setup this way before I started here and I figure if its not broke don't fix it, right? We do not have raid or storage spaces setup because its a vm that resides on a raided storage system.

Guido,

I wish I could have forward incrementals but unfortunately we do not have enough storage space on the backup targets to be able to keep the retention policy we want.

[Delo123]

I wish I could have forward incrementals but unfortunately we do not have enough storage space on the backup targets to be able to keep the retention policy we want

Were are you storing your backups on? If it's a windows machine, turn on dedupe :)

You could give Nirsoft's FolderChangesView a try, it's free, small, no bloatware as everything from nirsoft is kinda cool and at least should give you some hints if anything is going on on filesystem level... Just download, run, enter the disks you want to monitor and maybe check the option the export a log every x minutes / hours...

Then get some Coffee and relax

[Delo123]

We do not have raid or storage spaces setup because its a vm that resides on a raided storage system.

Ps. I kinda forgot we are talking about a vm

[Delo123]

Uhh today Is the first Friday of the month, maybe you can quickly install it and catch something...

[agossett]

Not a windows machine :/ and thank you for that suggestion. I currently have it running.

[Delo123]

How did it go? Again a big backup?

[agossett]

You guessed it. the Log file from that program ran but nothing in it looks like it would have caused this issue.

There are a lot of changes related to dedup but there are also a very similar amount of changes on other days so I don't think that is causing it.

[Delo123]

Hmm ok... Well dedupe blocks do not change if no dedupe jobs are running, but maybe you see reads from the dedupe chunks (reads ofcourse do not come the files directly, but from the deupded chunks.....)... Strange... Are you sure defrag is disabled? (it's in the task scheduler, but not in the overview but microsoft -> windows -> deduplication), but anyway, the default schedule is once a week and not monthly....
Very strange... Did you already have somebody else have a fresh look at the system? Maybe it's something silly and your just overlooking, stupid issue

[agossett]

It is definitely disabled. And i think I am up to a total of 6 different people looking at it. it is a very strange issue and i have no doubts that it is something silly and stupid :/ Veeam is currently going over the log from the file monitoring program. Hopefully they can find something I missed!

[Delo123]

Sigh... If nothing works you of course could do a reinstall and leave the data in place (if you do not have a lot of applications running on that thing of course)... Anyway, hope they find something. Keep us updates & good luck!

[Delo123]

On last idea... Did you try disabling CBT? Obviously incrementals will take much longer, but maybe it could show if it is caused by CBT or actual data changes?!

[VladV]

Do you by any chance have Shadow copies scheduled for these disks?

[Delo123]

Adam already wrote he had disabledshadow copies for those volumes

[mkaec]

I think it is the dedup garbage collection that it causing this problem for you. I tracked this on our systems when I was testing Veeam with Windows dedup. Windows dedup stores the dedup blocks in 1 GB .ccc files in \System Volume Information\Dedup\ChunkStore. The actual files on a dedup volume are mostly empty. It doesn't matter if the files don't change. The garbage collection routine can make changes to the chunk store as it sees fit. It's correct for Veeam to see this as changed blocks and back them up. The problem with Veeam is that there's no way for the user to see what changed in a particular backup. (Though v9 is now looking at the NTFS MFT for a different feature. So, maybe a future version will add better visibility into the contents of a backup file.)

This is one disadvantage of block-based backup. A traditional backup application would ignore the chunk store and only operate on the actual files. Of course, traditional file-based backup has a different set of problems.

There is a scheduled task "Microsoft\Windows\Deduplication\Weekly Garbage Collection" that runs every week by default. You could try changing the schedule to monthly to see if the large Veeam incrementals occur less frequently. Or you could just turn off dedup if you have the space to spare. Or you could set up a repository that is also running Windows dedup (and turn off Veeam compression). I've found that the Veeam incrementals inflated by Windows dedup on the source dedup nicely on a repository also running Windows dedup.

[agossett]

If it was the weekly garbage collection shouldn't this happen every week? I also have dedup running on other disk but they do not have this same problem.

I really wish i could temporally turn dedup off but i don't have the disk space at the moment.

[Delo123]

Adam,

earlier you wrote you had disabled all dedupjobs. So if optimization / garbage collection / Scrubbing jobs are disabled it wouldn't make sense...
If they however are not disabled, do it for testing. (until your next Fridays first!) Dedup itself will still de enabled, there will just be no further backup jobs and there will be no issue with these jobs disabled

[mkaec]

It wouldn't happen every week because garbage collection isn't guaranteed to make large changes every week. To test, just disable the "Weekly Garbage Collection" and "WeeklyScrubbing" scheduled tasks for a few weeks. New data will still dedup, but deleted data won't clear out effectively with the tasks disabled.

[skrause]

Make sure you have disabled defragmentation/optimization for that drive in the OS. I had a server that was having much larger increments than normal every wednesday night and I found in the task scheduler that defragmentation was running every wednesday at 1am so when my backups ran at 9pm they had a lot more changed blocks than before.