Comprehensive data protection for all workloads
Post Reply
DennisV
Novice
Posts: 9
Liked: never
Joined: Feb 24, 2009 2:27 pm
Contact:

VRB's really large for a small environment

Post by DennisV » Jan 06, 2010 3:42 pm

Costumer has several servers with a small number of concurrent users (10).

The daily delta files (VRB) are huge.
For example the Terminal Server has a constant daily VRB of 8GB.
The Exchange server differs from 6-16GB on a daily basis, while this customer would have a max.daily mailtraffic of 50MB.
The swapfiles for the VM's are respectivly 1 and 2 GB.

How come Veeam Backup creates such huge deltas for these VM's ?

canotuna
Lurker
Posts: 2
Liked: never
Joined: Jan 06, 2010 3:59 pm
Full Name: Dan Sanderson
Contact:

Re: VRB's really large for a small environment

Post by canotuna » Jan 06, 2010 4:33 pm

swap file?

mdornfeld
Expert
Posts: 125
Liked: 2 times
Joined: Mar 23, 2009 4:44 pm
Full Name: Matt
Contact:

Re: VRB's really large for a small environment

Post by mdornfeld » Jan 06, 2010 4:53 pm

Auto disk defragger turned on?

Gostev
SVP, Product Management
Posts: 24804
Liked: 3566 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VRB's really large for a small environment

Post by Gostev » Jan 06, 2010 5:15 pm

Veeam Backup VRB size is equal to the amount of VMDK blocks changed since last backup, so generally you need to look for reasons what is causing so many disk block changes.

Exchange and SQL servers generally produce very large incrementals because Exchange uses plain-text transaction log files. Multiple GB of transaction logs is "normal" for busy Exchange servers, but of course 50MB of emails cannot be considered as "busy", so 6-16 GB is strange.

Not so sure about Terminal Services server, but as long as it has many users storing and changing data on it, significant changes are probably expected.

For servers other than Exchange and SQL (for example, file server, DC, DNS, antivirus) typical VRB size is 1-2% of the VM size, according to what most customers are reporting. Have not heard much feedback on Terminal Services servers though.

DennisV
Novice
Posts: 9
Liked: never
Joined: Feb 24, 2009 2:27 pm
Contact:

Re: VRB's really large for a small environment

Post by DennisV » Jan 07, 2010 8:02 am

The swap files are max 2GB.
There is no (auto) defrag running in the VM's.

Terminal server disk size 25GB.
Maildaemon disk size 15GB.

Is there any way to track how much is changing so I can pinpoint the problems ?
Any other tips or suggestions are welcome.

Gostev
SVP, Product Management
Posts: 24804
Liked: 3566 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VRB's really large for a small environment

Post by Gostev » Jan 07, 2010 10:48 am

VRB size created during backup represents exactly the amount of changed VMDK data blocks since last backup (just keep in mind that VRB is compressed, so you should multiple it by 2 on average).

DennisV
Novice
Posts: 9
Liked: never
Joined: Feb 24, 2009 2:27 pm
Contact:

Re: VRB's really large for a small environment

Post by DennisV » Jan 07, 2010 4:15 pm

Ok, so the actual changes are 1,5-2x as much.....
That doesn't make sense for such a small environment.
Is there any way to track these changes ?

Gostev
SVP, Product Management
Posts: 24804
Liked: 3566 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VRB's really large for a small environment

Post by Gostev » Jan 07, 2010 7:17 pm

I suppose you could use Process Monitor (former FileMon) to track all changes on file level - its output contains offsets and data written for each file. You can also apply filter to log only writes. But I am not sure how you could process and make use of all this data after it is generated, since the 24 hours FileMon output will be totally immense based on the amount of activity on these systems...

One other idea I have is that file system on these VMs may be heavily fragmented, so even small change - a few KB - gets scattered across the whole virtual disk, and makes tons of blocks dirty (Veeam incrementals are block-level with 1024KB blocks). I would try performing full defragmentation on those problematic VMs, and starting backup from scratch to see if it helps with incremental sizes.

DennisV
Novice
Posts: 9
Liked: never
Joined: Feb 24, 2009 2:27 pm
Contact:

Re: VRB's really large for a small environment

Post by DennisV » Jan 08, 2010 9:57 am

Gostev,

Thanks for your help.
We are going to perform a few actions :
1) Move swapfile to seperate virtual harddisk and exclude from backup.
2) Defrag VM
3) Upgrade HW version to 7
4) Install/upgrade to B&R 4.0 using CBT

I will let you know what happens after these changes.

Gostev
SVP, Product Management
Posts: 24804
Liked: 3566 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VRB's really large for a small environment

Post by Gostev » Jan 08, 2010 11:10 am

Sounds good. You may want to upgrade straight to Veeam Backup 4.1 :D

awiesen
Influencer
Posts: 24
Liked: never
Joined: Sep 01, 2009 3:31 am
Full Name: Alex Wiesen
Contact:

Re: VRB's really large for a small environment

Post by awiesen » Jan 12, 2010 5:31 pm

We've noticed the same thing. The swapfile really does help, but we still do see pretty big VRB files. For us, it's more like 10% of the data (for a C: drive.)

We're pretty sure in our case, it's due to antivirus scanners marking stuff in a subtle and minor way as files are accessed and loaded. When we turned off antivirus, the size of our incrementals dropped substantially. We've tried a few different antivirus programs, including Symantec Endpoint Protection and AVG, but the problem doesn't go away. Since getting rid of antivirus isn't really an option for us, we're just living with the problem for now.

- Alex

tsightler
VP, Product Management
Posts: 5421
Liked: 2243 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: VRB's really large for a small environment

Post by tsightler » Jan 12, 2010 6:34 pm

One of the things you can do is to disable "Last Accessed" attribute on the NTFS filesystems, assuming you don't need it for anything. Basically, each an ever time a file is read, the "Last Accessed time" in the directory is updates. This is a very small disk write, but since Veeam uses 1MB blocks to track changes, many small updates across even a moderate number of files will create a fairly large VBR due to the way NTFS spreads it's directories around the disk. Disk defragementing should help a good bit, assuming you use a defragementer which consolidates the directories in a single spot (some defrag tools have a "performance" or "speed" options which attempts to place the directories closer to the files to avoid seek time for such updates).

This affects not just Veeam, but also some storage systems which use large pages for their snapshots, and it affects both Linux and Windows filesystems, although for some reason Windows seems worse. Here's a link to an article that talks about it. It's geared to Equallogic and large snapshots, but the problem with Veeam VBR is very similar, although it's actually worse on Equallogic since they use 16MB pages.

http://www.interworks.com/blogs/bfair/2 ... -snapshots

And a link to turning off NTFS Last Access

http://technet.microsoft.com/en-us/libr ... 10%29.aspx

Between running a good defragment, and turning off this feature, we managed to cut many of our servers VBR file sizes in half.

matarvai
Enthusiast
Posts: 30
Liked: never
Joined: Apr 07, 2010 9:49 am
Full Name: Marko Tarvainen
Contact:

Re: VRB's really large for a small environment

Post by matarvai » May 28, 2010 4:40 am

I am also having the same problem. I haven't disable antivirus software yet, but I have disabled last accessed attribute and moved swap file. I did a test with our Windows 2003 Terminal server yesterday and I unplugged it from network. No one used it for a day and still vrb file from yesterday was 1,2GB when original vbk file is 6,2gb. I will try without antivirus software next, but the vrb files are still too big from every server that we are running.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 17 guests