VRB's really large for a small environment

[DennisV]

Costumer has several servers with a small number of concurrent users (10).

The daily delta files (VRB) are huge.
For example the Terminal Server has a constant daily VRB of 8GB.
The Exchange server differs from 6-16GB on a daily basis, while this customer would have a max.daily mailtraffic of 50MB.
The swapfiles for the VM's are respectivly 1 and 2 GB.

How come Veeam Backup creates such huge deltas for these VM's ?

[canotuna]

swap file?

[mdornfeld]

Auto disk defragger turned on?

[Gostev]

Veeam Backup VRB size is equal to the amount of VMDK blocks changed since last backup, so generally you need to look for reasons what is causing so many disk block changes.

Exchange and SQL servers generally produce very large incrementals because Exchange uses plain-text transaction log files. Multiple GB of transaction logs is "normal" for busy Exchange servers, but of course 50MB of emails cannot be considered as "busy", so 6-16 GB is strange.

Not so sure about Terminal Services server, but as long as it has many users storing and changing data on it, significant changes are probably expected.

For servers other than Exchange and SQL (for example, file server, DC, DNS, antivirus) typical VRB size is 1-2% of the VM size, according to what most customers are reporting. Have not heard much feedback on Terminal Services servers though.

[DennisV]

The swap files are max 2GB.
There is no (auto) defrag running in the VM's.

Terminal server disk size 25GB.
Maildaemon disk size 15GB.

Is there any way to track how much is changing so I can pinpoint the problems ?
Any other tips or suggestions are welcome.

[Gostev]

VRB size created during backup represents exactly the amount of changed VMDK data blocks since last backup (just keep in mind that VRB is compressed, so you should multiple it by 2 on average).

[DennisV]

Ok, so the actual changes are 1,5-2x as much.....
That doesn't make sense for such a small environment.
Is there any way to track these changes ?

[Gostev]

I suppose you could use Process Monitor (former FileMon) to track all changes on file level - its output contains offsets and data written for each file. You can also apply filter to log only writes. But I am not sure how you could process and make use of all this data after it is generated, since the 24 hours FileMon output will be totally immense based on the amount of activity on these systems...

One other idea I have is that file system on these VMs may be heavily fragmented, so even small change - a few KB - gets scattered across the whole virtual disk, and makes tons of blocks dirty (Veeam incrementals are block-level with 1024KB blocks). I would try performing full defragmentation on those problematic VMs, and starting backup from scratch to see if it helps with incremental sizes.

[DennisV]

Gostev,

Thanks for your help.
We are going to perform a few actions :
1) Move swapfile to seperate virtual harddisk and exclude from backup.
2) Defrag VM
3) Upgrade HW version to 7
4) Install/upgrade to B&R 4.0 using CBT

I will let you know what happens after these changes.

[Gostev]

Sounds good. You may want to upgrade straight to Veeam Backup 4.1

[awiesen]

We've noticed the same thing. The swapfile really does help, but we still do see pretty big VRB files. For us, it's more like 10% of the data (for a C: drive.)

We're pretty sure in our case, it's due to antivirus scanners marking stuff in a subtle and minor way as files are accessed and loaded. When we turned off antivirus, the size of our incrementals dropped substantially. We've tried a few different antivirus programs, including Symantec Endpoint Protection and AVG, but the problem doesn't go away. Since getting rid of antivirus isn't really an option for us, we're just living with the problem for now.

- Alex

[tsightler]

One of the things you can do is to disable "Last Accessed" attribute on the NTFS filesystems, assuming you don't need it for anything. Basically, each an ever time a file is read, the "Last Accessed time" in the directory is updates. This is a very small disk write, but since Veeam uses 1MB blocks to track changes, many small updates across even a moderate number of files will create a fairly large VBR due to the way NTFS spreads it's directories around the disk. Disk defragementing should help a good bit, assuming you use a defragementer which consolidates the directories in a single spot (some defrag tools have a "performance" or "speed" options which attempts to place the directories closer to the files to avoid seek time for such updates).

This affects not just Veeam, but also some storage systems which use large pages for their snapshots, and it affects both Linux and Windows filesystems, although for some reason Windows seems worse. Here's a link to an article that talks about it. It's geared to Equallogic and large snapshots, but the problem with Veeam VBR is very similar, although it's actually worse on Equallogic since they use 16MB pages.

http://www.interworks.com/blogs/bfair/2 ... -snapshots

And a link to turning off NTFS Last Access

http://technet.microsoft.com/en-us/libr ... 10%29.aspx

Between running a good defragment, and turning off this feature, we managed to cut many of our servers VBR file sizes in half.

[matarvai]

I am also having the same problem. I haven't disable antivirus software yet, but I have disabled last accessed attribute and moved swap file. I did a test with our Windows 2003 Terminal server yesterday and I unplugged it from network. No one used it for a day and still vrb file from yesterday was 1,2GB when original vbk file is 6,2gb. I will try without antivirus software next, but the vrb files are still too big from every server that we are running.