90GB RDS server VM seeing 20GB of change per day

[Jamie Pert]

Hello everyone,

Firstly I'm not really sure if this falls into an error / technical issue - so I have not yet logged a support ticket with Veeam - feel free to leave me a comment if you think I should and I will.

Basically for almost a year I have been running backups of a client's entire infrastructure, not that it's important but one goes from the SAN to a Synology NAS box connected as an iSCSI target and the other goes to a remote location for offsite backups. In terms of setup Reversed incremental backups are ran every 24 hours on weekdays only

Recently we have started seeing the client's RDS server, which is a 90GB VM , take far longer to complete a backup job than ever before. After some investigation Veeam seems to be reading just shy of 20GB of new data per day - this is not a lot for most of our clients but given their connection speed it is a real nightmare. As Veeam backs-up at the block level we at first wondered if it was defragmentation of some sort, but there's no evidence of that, so then we thought perhaps the 22 RDS users had been caching their Exchange OST files which could have accounted for quite a lot of change, but not 20GB! There are a few small archive.pst files, but they aren't updated regularly and total up to less than 5GB.

Here's some more information:

This VM has one disk (a 90GB volume), of which around 70GB is in use and this doesn't change a great deal day to day.

In terms of applications there is Office 2010 of which Word, Excel and Office are used the most. There is also Google Chrome and that's about it. There will obviously be some temp file build up, some users saving data to the wrong directories etc - but nothing that I can think of which would cause such a massive difference in change data day on day...

Any ideas, I can provide lots more info if necessary...

[chrisdearden]

What about profiles ?

[Jamie Pert]

Obviously profiles will be accounting for some change and 20 users all logging on everyday will mean that change will be quite significant - but consistently roughly 19GB every day? It just doesn't seem right....

As previously stated Office applications are used more than anything else which in my experience don't account for a huge amount of change in the user profile areas. Google Chrome's Cache is using on average 300MB in each user's profile - but that's in it's entirety and is not change day on day.

[Vitaliy S.]

Do you have any antivirus installed or probably other maintenance tasks running on this server?

[Jamie Pert]

Yes this VM has Sophos installed. This is set to carry out a weekly scan and we've also enabled Live Protection. I can't see any other applications installed which would carry out maintenance tasks, likewise I've checked task scheduler and aside from some scheduled restarts, generic microsoft window tasks and application updaters (Adobe / Google) there's nothing much there.

[chrisdearden]

Any scheduled shadow copies?

[Jamie Pert]

I really thought you were on to something there Chris, however Shadow Copies have been disabled on this particular server...

[foggy]

Yes this VM has Sophos installed. This is set to carry out a weekly scan and we've also enabled Live Protection.

Could you try to disable Live Protection and see whether this helps?

Also, see this topic for similar discussion.

[Jamie Pert]

whilst I understand the thought behind disabling Live Protection I wouldn't be authorized to disable this feature I'm afraid. Obviously we all know that they'll probably be fine for a small period of no Live Protection if something untowards would happen I'd risk losing our company a contract (and my job) if the worst was to happen!

I have read a little about heavy fragmentation doing this.... (mainly thanks to the link foggy provided). This VM has never been defragged and it has been live a year - perhaps this could be the problem..

[foggy]

Then you could defragment the VM, then run sdelete and full backup after that to see whether this was the issue. However, take these additional steps into consideration if the VM has thin disks.

[tsightler]

I'd look very carefully at any recent changes if you feel that this is something that just started as fragmentation would be more likely to slowly increase rather than start suddenly?

[Jamie Pert]

Ok, I checked the server today and it's just 29% fragmented (much better then I thought).

I was at a bit of a loose end so here's a lot of information which shows just how things have deteriorated over the best part of a year

Two sequential days in November 2012
Status Start time End time Size Read Transferred Duration
Success 04:38:53 04:53:09 60.0 GB 60.0 GB 2.2 GB 0:14:15
Success 04:26:45 04:34:19 60.0 GB 1.7 GB 742.9 MB 0:07:34

Two sequential days in December 2012
Success 03:36:34 03:42:58 60.0 GB 734.8 MB 234.7 MB 0:06:23
Success 03:36:05 03:43:11 60.0 GB 989.0 MB 311.5 MB 0:07:05

Two sequential days in January 2013
Success 05:44:33 07:29:02 60.0 GB 7.3 GB 2.9 GB 1:44:28
Success 04:46:31 05:50:29 60.0 GB 3.7 GB 1.5 GB 1:03:58

Two sequential days in February 2013
Success 04:25:51 04:35:16 60.0 GB 1.7 GB 625.0 MB 0:09:25
Success 06:30:01 07:14:10 60.0 GB 5.6 GB 2.3 GB 0:44:08

Two sequential days in March 2013
Success 04:13:59 06:03:19 70.0 GB 7.1 GB 3.1 GB 1:49:20
Success 05:07:53 06:42:16 70.0 GB 6.3 GB 2.9 GB 1:34:23

Two sequential days in April 2013
Success 05:33:31 07:31:46 70.0 GB 7.5 GB 3.2 GB 1:58:14
Success 05:34:49 07:42:36 70.0 GB 8.0 GB 3.3 GB 2:07:46

Two sequential days in May 2013
Success 05:37:23 08:00:01 75.0 GB 8.9 GB 4.4 GB 2:22:38
Success 08:32:40 12:00:12 75.0 GB 7.1 GB 3.5 GB 3:27:31

Two sequential days in June 2013
Success 04:58:39 05:38:55 80.0 GB 2.7 GB 884.0 MB 0:40:15
Success 03:39:51 03:52:54 80.0 GB 2.0 GB 874.4 MB 0:13:02

Two sequential days in July 2013
Success 04:00:08 04:20:27 90.0 GB 3.8 GB 1.3 GB 0:20:19
Success 01:45:57 02:32:43 90.0 GB 2.5 GB 1.1 GB 0:46:45

August

Success 12:03:03 14:28:04 90.0 GB 16.0 GB 6.9 GB 2:25:01
Success 07:06:49 08:53:17 90.0 GB 13.8 GB 5.6 GB 1:46:27
Success 11:53:25 14:37:39 90.0 GB 19.7 GB 8.3 GB 2:44:14
Success 13:20:41 14:57:20 90.0 GB 10.3 GB 4.5 GB 1:36:39

[Jamie Pert]

I think this does show that it has been more gradual than I thought, but still there is an issue which I don't know the cause

[cparker4486]

Use Powershell to find all the files that have been modified in the past 24 hours. This may help you discover a file (or files) that is changing which you didn't expect.

You could take that even further and schedule a small script to run every x minutes which saves the output to a file. You could then take that log and use Excel to count the occurrences of filenames and see which files are changing frequently.

Again, this isn't a silver bullet solution but it may help you figure out what's happening.

edit:
And just to share stats on my own environment I see an average of 14GB of change per day on my RDS server with about 18 users.

[Jamie Pert]

OK, as a techie it is often quite hard to change an opinion on something like this and maybe admit I have been barking up the wrong tree, but I wonder if this is something as simple as an overloaded Veeam infrastructure which has caused this. Let me explain.

We're just putting in a new SAN to take some load of a Synology NAS box which has been struggling of late. The SAN has a RAID 10 array, whilst the NAS has four disks in a RAID 5 array. Given that we use reversed incremental backups the load on the array is high, I think this is causing the issue. Day to day our jobs have been slowly taking longer and longer.

Because the infrastructure has grown rapidly in terms of demand a nightly backup is rarely starts exactly 24 hours after the last (it is often queued waiting for resources), one delay and before you know it your job could be actually looking at 30 hours of change, obviously if the extra hours don't seep into the working day we're fine, but if it is getting to 10am and users have been working 2 or so hours there's an obvious reason for more data being backed up.

[foggy]

Looks reasonable. Thanks for admission.

Do your jobs typically show target as the bottleneck?

[Jamie Pert]

Yes, we have been aware of this for a while and maxing out the Synology NAS's RAID 5 array for 12 or so hours 5 days a week.

We are now splitting the load and moving jobs onto the much faster 10 disk RAID 10 array. Not to mention when we have finished testing Veeam 7 we will be able to half the amount of backup jobs which run each day (currently 1 to local storage and the other to remote storage) so I'm expecting great improvements with the backup copy feature as well as WAN acceleration.

Less processing, WAN acceleration and an additional faster repository should really sort things out.

[foggy]

Great, keep us updated on how it is going.