Server 2012 File Server random large backups on certain disk

[foggy]

Some other hints can also be found in this thread.

[Delo123]

Hi Adam,

were you able to solve this? Or gather some new insights?

Guido

[smartsys]

We are experiencing the same problems.
Until 2 months ago we had 2 fileservers running Windows 2012 with dedup enabled.
The first server was about 1TB in size, the other was 2.5TB

Because of frequent very large incremental backups we decided to split the file servers into 6 different servers to prevent the backup copy job files to become way too big for Windows dedup on the Veeam Backup repositories. So we moved almost all shares to new 2012 R2 servers and because of a new SAN we didn't enable dedub on these servers.
Now we have one server left with Dedub enabled. It has a 700GB drive with dedub enabled and this disk ONLY contains user home folders with quota's enabled for max 5GB of data. This is only used for private storage.
On normal days the backup of this server is about 1 to 3 GB of transfered data, but on some days it is almost about 270 tot 290GB. This equals the same about of dedub savings on this volume!

We changed the Scrub and Garbage Collection schedule to run daily. One should expect by running these jobs daily it is not possible to get these very large variations in the daily incremental backup.

The problem we run into is that because we are using normal incremental backups it takes a very long time to merge the last incremental backup in the full backup file.

[agossett]

Hi Delo,

This has not been resolved and I think veeam support has given up on trying to figure it out because they can not see anything wrong with Veeam or my VMWare environment so it must be something with the file server itself. My boss and I have looked I don't know how many times over everything on the server and there is NOTHING that we can find that can be causing this. This is so frustrating. One more example I can give: We have one drive that is only used by 1 person in the communications department to store pictures and what not. She was out sick on Friday and was sick all weekend and I even confirmed with here that she did not do any work over the weekend yet that drive had a 40GB backup. Literally nothing changed on that drive over the weekend but it somehow had an abnormally large backup.

I am at my wits end here trying to figure this out because it is causing issue with out other backups not finishing in a timely manner that also run over the weekend. I believe it to be causing a huge bottleneck with out SQL server for Veeam which is causing other jobs to fail but those jobs are getting all the way through there first run and then failing at the end. What that means for me is that I am doing a Full backup of our exchange server starting on Saturday night. It is a 1tb VM so it take some time to complete on a 100mb connection to our offsite location. Once it completes the job ends up failing because the SQL server is busy trying to work with the messed up file server backups. So then the job does a retry, it is now 1:00 pm on Monday and the job is still running... This is having a huge effect on our bandwidth for our users during business hours.

My apologies for the rant, If any one has any more suggestions I am desperate to figure this out at this point. I am half determined to just bite the bullet and take the file server offline the next time this is supposed to happen and she what happens with the backup. I mean if the server is turned completely off no data can actually change, right? Or have I just gone completely crazy at this point?

[skrause]

Do you have disk optimization enabled in Windows?

I found that it was the cause of us having larger than normal incremental files one night a week on a couple servers that had it enabled (I forgot to disable it on my 2012R2 template when I created it as it is on by default.)

[agossett]

I do not.

[foggy]

Haven't you thought about comparing the files inside the guest OS right before and right after the backup that gets unexpected amount of changes?

[Delo123]

Hi Adam,

I know you disabled defrag, dedup etc... but just to be sure, did you disable all jobs under Task Scheduler -> Task Scheduler Library -> Microsoft -> Windows -> Task Scheduler?
The jobs are idle, manual and regular maintenance, if these (apart from the manual which has no trigger of course) are enabled defrag will happen on a weekly basis I believe... Anyway, always good to double check

Guido

[agossett]

Hi Foggy,

I am currently working on testing this.

Delo,

The only thing that is active under that path is the regular Maintenance but it is set to run every day. Any idea what this does exactly? I have not had to much luck other then finding that it might be doing a defrag. Not sure if this is of any relevance or not but if I open the defrag application it says that it has never been run on any of my disk and that drives are not being optimized automatically.

Adam.

[Delo123]

It does multiple things... from memory it runs chkdsk, application clean up (whatever that may be), remove temp files and run a defrag (once a week), should your volumes be thin I believe it also does an unmap action.
So in your case you should definitely disable this for your testing! Yes, I believe in the defrag application these jobs aren't accounted for, but I'm pretty sure defrag will run when maintenance is enabled...

Guido

[mkaec]

The defrag GUI does list the "Last run" date/time for each drive. It should be easy to determine if a defrag was run recently before the incremental.

[Delo123]

Hi Marc,

so you would leave regular maintenance enabled? Even if Microsoft says it triggers defragmentation once a week?

[mkaec]

For testing I would probably disable it. But, for troubleshooting purposes, the defrag GUI may provide useful information.

[Delo123]

Adam said the Defrag utility didn't show any activity, so apparently either the gui doesn't work or regular maintenance isn't reporting.
Anyway, it would be good to disable and test another month after the next full to be able to see the difference. Every little idea can help Adam I guess...

[mkaec]

I missed that detail that he said the GUI is not reporting.

I checked on one of my systems and the "ScheduledDefrag" scheduled task has no trigger set. The last run time does match with what the defrag GUI is showing. I see in the task history that the task is periodically launched "by user" "System". I would surmise that the system maintenance is manually running this scheduled task as it deems fit.

[skrause]

If you want to make sure the defrag tasks are disabled, just run this in powershell:

Code: Select all

Get-ScheduledTask ScheduledDefrag | Disable-ScheduledTask

[agossett]

I ran Get-ScheduledTask ScheduledDefrag and it is already set to disabled.

[Delo123]

Hi Adam,

If i were you i would disable regular maintenance anyway for the time being, i would say it's your best bet right now, also since it's not "just" defrag which is been done by maintenance...

[skrause]

From what I have read, and my experience running said PowerShell cmdlet so far, the "ScheduledDefrag" task is actually the entire disk optimization batch of things. (Disabling the scheduled task will make the "Disk Optimization" GUI show that optimization is disabled).

[Delo123]

ScheduledDefrag is not the same as "regular maintenance" which is a task of it's own and runs daily. It does a lot of things and once a week it does a defrag this shouldn't be visible by looking at the history of scheduled defrag since it's a task of it's own... So there will be changes, and these will have impact on cbt.

[Delo123]

Hi Adam, no news?