How to reduce backup sizes

[rimvydukas]

Hi,

I've started to use Veeam for my ESX servers infrastructure. As most of my machines are windows ones I noticed one not so happy thing. I use incremental backups to backup my machines and can see that even from not so active machines Veeam transfers about 4GB of data each day backup is run. I also have Tivoli TSM agent on these machines (I'm still on veeam purchase phase). So when I look at the Tivoli client logs I can see that it transfers only 500MB of data from the server when veeam transfers whooping 5GB of data. Why are such differences? Page file? Something else?

P.S. Noticed that V6 skips page file, so I think the problem is not with the page file.

[veremin]

The reason of such behaviour is in the way two products work. TSM is a file-level solution that requires an agent to be installed inside VM and that keeps track about changed files, meanwhile, VB&R is an image-based one, that cares only about blocks. And changed blocks are not necessarily related to files – some activities such as defragmentation, as well as, antivirus one that don’t literally change files, does affect seriously the number of changed block and so on.

So, as a first step of investigation I recommend checking whether there is a security/defragmentation activity that is being performed on regular basis and that might result in such a big number of changed blocks.

Thanks.

[rimvydukas]

Defragmentation are not done. Periodic antivirus scans are not done. One thing I suspect is when Tivoli examines what to back up? Could it be the reason that something changes in that time?

[chrisdearden]

Is TSM backing up every file on disk or is there a more selective policy ? Also is that 5GB the raw size of data or the size of the compressed, deduped incremental ?

[rimvydukas]

TSM does backups for whole machine, not particular directory. Deduplication and compression are set to defaults so both thing are used. It would be nice to find what causes such huge changes count.

[chrisdearden]

Are you using Volume shadow copies at all ? There will be some delta due to the block sizes used - all it takes is for a single bit in a 1Mb block to change and be marked in the CBT file , meaning we will back that block up. If your discs are not aligned , you may be changing more blocks than needed.

[veremin]

I’m wondering whether you get such a huge increment on every VM in your environment or only on certain ones, such as SQL, Exchange, etc. Additionally, completely for the purpose of testing it might be worth disabling TSM temporarily and seeing whether this reduces size of increments or not.

Thanks.

[rimvydukas]

I could understand if the server is Exchange server or SQL, but I noticed the same thing and with another servers.

Ok, did another experiment.

1. Took ordinary windows 2003 server. It has Kaspersky installed.
2. Did full backup. Veeam transferred 10GB of data.
3. Ran backup job once again. Veeam transferred 200MB of data.
4. Ran ntbackup and selected to archive everything. Backup file was configured to use network location. Backup started I saw that it would take lond time to complete and canceled it.
5. Ran Veeam backup job once again. Now I've got 2.3GB of transferred data.

So....?

[chrisdearden]

Archive bit would have changed on the files, causing a high level of changed blocks.

[rimvydukas]

Ok, I'll try to stop TSM client completely on some servers and will try again to see if there will be some changes in few days. Because I have three new sharepoint servers without tsm and kaspersky (remote sql) and increment changes are minimal.

[rimvydukas]

Ok, did another experiment:) Tried to run and cancel ntbackup for the second time and still the same behavior. Will try to do third experiment, will configure veeam to backup only system disk, will let backup to complete and will see what will happen:)

[rimvydukas]

Ok, it seems that I've found root cause of my problem. And it is Kaspersky. So I had following situation:

IIS server and kaspersky installed. We have no kaspersky periodic scans scheduled. So did full backup and increment backup, 10 minutes after the full. I had 5GB of changes.
Ok. Disabled Kaspersky, did full backup and increment after 25 minutes. I've got only 40MB of changes.

From that I can assume that it is definitely Kaspersky. So what are my options then? How can I fix this problem?

[veremin]

Hmm, so, is it just one server that hosts Kaspersky software and is affected badly by the aforesaid behaviour or there are number of them? Thanks.

[rimvydukas]

Problem is that all of our virtual Windows servers have Kaspersky installed and all servers suffer from this behavior. Each generates in average 4-10GB of changes during a day (increments are done once a day).

[veremin]

Probably, you can directly ask Kaspersky representative regarding what certain setting(s) might be responsible for changing blocks inside VMs and what can be done in order to prevent it. Thanks.

[rimvydukas]

No, Kaspersky does not have periodic scans set up. I suspect that this problem somehow is related to indexing. So my next steps:

Changed kaspersky policy to scan files when they are executed. I had this option set to scan files when they are opened, modified, executed.

Another thing I want to to is to exclude veeam exe files from the scan. Can you explain - from where exactly these files are started when they are injected into guest os during vss snapshot and indexing time.

[rimvydukas]

Probably, you can directly ask Kaspersky representative regarding what certain setting(s) might be responsible for changing blocks inside VMs and what can be done in order to prevent it. Thanks.

Vladimir, I'll do this for sure If I'll not find solution:)

[veremin]

Actually, Veeam application aware image processing/indexing has nothing to do with persistent files, but rather with run-time processes. We’re just using run-time process that is deployed on administrative share and instructs Windows VSS to freeze application prior to snapshot taking. And it’s deleted afterwards, once the backup task is finished.

As far as I’m concerned, it’s possible with Kaspersky to specify process exclusion mask, so, just set Kaspersky to overlook process which names match “veeam”. Though, I still doubt that this will have a positive effect on amount of changed blocks.

Hope this helps.
Thanks.

[rimvydukas]

Vladimir,

I'm going to play with exclusions because it seems that I have a lot of changes when I use indexing. So I think that following logic applies here:

Veeam starts VeeamOibIndexer.exe process on my guest machine. This process tries to index (access) a lot of files. Kaspersky sees that the file is accessed, scans it and modifies some kind of attribute of this file (my assumption). And if this happening with a lot of files - I have a lot of changed blocks.

Can you clarify, how exactly does this indexing work? Thanks.

[Vitaliy S.]

Index data is obtained from NTFS MFT directly (rather than by scanning the entire file system). I would suggest trying to disable last access time stamp on the files, might help.

See this topic for additional information: Large VIB file on a small static server

[veremin]

Additionally, in order to confirm whether the indexing in conjunction with Kaspersky affects amount of changed blocks it might be worth disabling indexing temporarily and see whether anything changes or not.

Thanks.

[rimvydukas]

Vitaly, thanks for the suggestion but my servers are 2008 and 2008R2 ones and the thread says that this thing is disabled by default.

[rimvydukas]

Additionally, in order to confirm whether the indexing in conjunction with Kaspersky affects amount of changed blocks it might be worth disabling indexing temporarily and see whether anything changes or not.

Thanks.

Yes, I did that ant it seems that there were only 100MB of changes, like when I had Kaspersky disabled. Can you try to replicate the same thing in your labs? I think that I'm not the only one who uses Kaspersky.

[rimvydukas]

Ok, there is definitely something wrong between Kaspersky and Veeam indexing service. To be completely sure I did the following experiment. Disabled indexing. Did incrementan backup. Restarted server, expanded disk by 10GB, expanded paging file and when I did next incremental backup I had only 200MB of changes, not 2.5GB.

[veremin]

Honestly, I still believe that the best and most efficient you have is to either contact Kaspersky directly regarding this behavior or totally suppress this software while the backup is being taken; since it’s highly unlikely that our QA team will do everything in their power in order to address issue related mostly to the other software, especially, when the release is coming soon.

Thanks.

[Vitaliy S.]

Restarted server, expanded disk by 10GB, expanded paging file and when I did next incremental backup I had only 200MB of changes, not 2.5GB.

Just to be sure we are on the same page - paging file is excluded from the backup. Expanding disk will not increase your backup file unless you fill this disk blocks with new data.

[rimvydukas]

Vitaly, I know this:) I mentioned this just in case:)

[rimvydukas]

Honestly, I still believe that the best and most efficient you have is to either contact Kaspersky directly regarding this behavior or totally suppress this software while the backup is being taken; since it’s highly unlikely that our QA team will do everything in their power in order to address issue related mostly to the other software, especially, when the release is coming soon.

Thanks.

Ok, I'll try to contact Kaspersky after my vacation but if they will ask me to clarify how exactly does this indexing process work I'll not be able to clarify anything for them, so I thought that it would be better for your engineers to contact Kaspersky engineers. But if you say that I must do this myself, I'll try:)

[rimvydukas]

Hi,

Can anyone explain me, how does exactly Veeam indexing process work on virtual machine. I have a problem with my not so busy Windows 2003 virtual servers - daily increments are somewhere between 3-10GB. I removed antivirus, I have no disk defragmentation software, I disabled tivoli tsm client on several servers. I ran filemon and saw one interesting thing - when veeam starts its indexing of virtual machine files I can clearly see that VeeamOibIndexer process creates a lot of tmp files on guest server disk. I suspect that there will be more changed info because of this and because of this I'll have large next incremental job size. So how exactly does this indexing process work on vm? Thanks.

[veremin]

Assuming that the issue is related to indexing stuff, may I ask you try to back this VM up without indexing and see whether large incremental is reproducible without Indexing option or not? Though, please be aware to run the job without indexing at least two times.Thanks.