Comprehensive data protection for all workloads
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

How to reduce backup sizes

Post by rimvydukas »

Hi,

I've started to use Veeam for my ESX servers infrastructure. As most of my machines are windows ones I noticed one not so happy thing. I use incremental backups to backup my machines and can see that even from not so active machines Veeam transfers about 4GB of data each day backup is run. I also have Tivoli TSM agent on these machines (I'm still on veeam purchase phase). So when I look at the Tivoli client logs I can see that it transfers only 500MB of data from the server when veeam transfers whooping 5GB of data. Why are such differences? Page file? Something else?

P.S. Noticed that V6 skips page file, so I think the problem is not with the page file.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

The reason of such behaviour is in the way two products work. TSM is a file-level solution that requires an agent to be installed inside VM and that keeps track about changed files, meanwhile, VB&R is an image-based one, that cares only about blocks. And changed blocks are not necessarily related to files – some activities such as defragmentation, as well as, antivirus one that don’t literally change files, does affect seriously the number of changed block and so on.

So, as a first step of investigation I recommend checking whether there is a security/defragmentation activity that is being performed on regular basis and that might result in such a big number of changed blocks.

Thanks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Defragmentation are not done. Periodic antivirus scans are not done. One thing I suspect is when Tivoli examines what to back up? Could it be the reason that something changes in that time?
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: How to reduce backup sizes

Post by chrisdearden »

Is TSM backing up every file on disk or is there a more selective policy ? Also is that 5GB the raw size of data or the size of the compressed, deduped incremental ?
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

TSM does backups for whole machine, not particular directory. Deduplication and compression are set to defaults so both thing are used. It would be nice to find what causes such huge changes count.
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: How to reduce backup sizes

Post by chrisdearden »

Are you using Volume shadow copies at all ? There will be some delta due to the block sizes used - all it takes is for a single bit in a 1Mb block to change and be marked in the CBT file , meaning we will back that block up. If your discs are not aligned , you may be changing more blocks than needed.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

I’m wondering whether you get such a huge increment on every VM in your environment or only on certain ones, such as SQL, Exchange, etc. Additionally, completely for the purpose of testing it might be worth disabling TSM temporarily and seeing whether this reduces size of increments or not.

Thanks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

I could understand if the server is Exchange server or SQL, but I noticed the same thing and with another servers.

Ok, did another experiment.

1. Took ordinary windows 2003 server. It has Kaspersky installed.
2. Did full backup. Veeam transferred 10GB of data.
3. Ran backup job once again. Veeam transferred 200MB of data.
4. Ran ntbackup and selected to archive everything. Backup file was configured to use network location. Backup started I saw that it would take lond time to complete and canceled it.
5. Ran Veeam backup job once again. Now I've got 2.3GB of transferred data.

So....?
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: How to reduce backup sizes

Post by chrisdearden »

Archive bit would have changed on the files, causing a high level of changed blocks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Ok, I'll try to stop TSM client completely on some servers and will try again to see if there will be some changes in few days. Because I have three new sharepoint servers without tsm and kaspersky (remote sql) and increment changes are minimal.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Ok, did another experiment:) Tried to run and cancel ntbackup for the second time and still the same behavior. Will try to do third experiment, will configure veeam to backup only system disk, will let backup to complete and will see what will happen:)
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Ok, it seems that I've found root cause of my problem. And it is Kaspersky. So I had following situation:

IIS server and kaspersky installed. We have no kaspersky periodic scans scheduled. So did full backup and increment backup, 10 minutes after the full. I had 5GB of changes.
Ok. Disabled Kaspersky, did full backup and increment after 25 minutes. I've got only 40MB of changes.

From that I can assume that it is definitely Kaspersky. So what are my options then? How can I fix this problem?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

Hmm, so, is it just one server that hosts Kaspersky software and is affected badly by the aforesaid behaviour or there are number of them? Thanks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Problem is that all of our virtual Windows servers have Kaspersky installed and all servers suffer from this behavior. Each generates in average 4-10GB of changes during a day (increments are done once a day).
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

Probably, you can directly ask Kaspersky representative regarding what certain setting(s) might be responsible for changing blocks inside VMs and what can be done in order to prevent it. Thanks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

No, Kaspersky does not have periodic scans set up. I suspect that this problem somehow is related to indexing. So my next steps:

Changed kaspersky policy to scan files when they are executed. I had this option set to scan files when they are opened, modified, executed.

Another thing I want to to is to exclude veeam exe files from the scan. Can you explain - from where exactly these files are started when they are injected into guest os during vss snapshot and indexing time.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

v.Eremin wrote:Probably, you can directly ask Kaspersky representative regarding what certain setting(s) might be responsible for changing blocks inside VMs and what can be done in order to prevent it. Thanks.
Vladimir, I'll do this for sure If I'll not find solution:)
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

Actually, Veeam application aware image processing/indexing has nothing to do with persistent files, but rather with run-time processes. We’re just using run-time process that is deployed on administrative share and instructs Windows VSS to freeze application prior to snapshot taking. And it’s deleted afterwards, once the backup task is finished.

As far as I’m concerned, it’s possible with Kaspersky to specify process exclusion mask, so, just set Kaspersky to overlook process which names match “veeam”. Though, I still doubt that this will have a positive effect on amount of changed blocks.

Hope this helps.
Thanks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Vladimir,

I'm going to play with exclusions because it seems that I have a lot of changes when I use indexing. So I think that following logic applies here:

Veeam starts VeeamOibIndexer.exe process on my guest machine. This process tries to index (access) a lot of files. Kaspersky sees that the file is accessed, scans it and modifies some kind of attribute of this file (my assumption). And if this happening with a lot of files - I have a lot of changed blocks.

Can you clarify, how exactly does this indexing work? Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: How to reduce backup sizes

Post by Vitaliy S. »

Index data is obtained from NTFS MFT directly (rather than by scanning the entire file system). I would suggest trying to disable last access time stamp on the files, might help.

See this topic for additional information: Large VIB file on a small static server
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

Additionally, in order to confirm whether the indexing in conjunction with Kaspersky affects amount of changed blocks it might be worth disabling indexing temporarily and see whether anything changes or not.

Thanks.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Vitaly, thanks for the suggestion but my servers are 2008 and 2008R2 ones and the thread says that this thing is disabled by default.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

v.Eremin wrote:Additionally, in order to confirm whether the indexing in conjunction with Kaspersky affects amount of changed blocks it might be worth disabling indexing temporarily and see whether anything changes or not.

Thanks.
Yes, I did that ant it seems that there were only 100MB of changes, like when I had Kaspersky disabled. Can you try to replicate the same thing in your labs? I think that I'm not the only one who uses Kaspersky.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Ok, there is definitely something wrong between Kaspersky and Veeam indexing service. To be completely sure I did the following experiment. Disabled indexing. Did incrementan backup. Restarted server, expanded disk by 10GB, expanded paging file and when I did next incremental backup I had only 200MB of changes, not 2.5GB.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to reduce backup sizes

Post by veremin »

Honestly, I still believe that the best and most efficient you have is to either contact Kaspersky directly regarding this behavior or totally suppress this software while the backup is being taken; since it’s highly unlikely that our QA team will do everything in their power in order to address issue related mostly to the other software, especially, when the release is coming soon.

Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: How to reduce backup sizes

Post by Vitaliy S. »

rimvydukas wrote:Restarted server, expanded disk by 10GB, expanded paging file and when I did next incremental backup I had only 200MB of changes, not 2.5GB.
Just to be sure we are on the same page - paging file is excluded from the backup. Expanding disk will not increase your backup file unless you fill this disk blocks with new data.
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

Vitaly, I know this:) I mentioned this just in case:)
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: How to reduce backup sizes

Post by rimvydukas »

v.Eremin wrote:Honestly, I still believe that the best and most efficient you have is to either contact Kaspersky directly regarding this behavior or totally suppress this software while the backup is being taken; since it’s highly unlikely that our QA team will do everything in their power in order to address issue related mostly to the other software, especially, when the release is coming soon.

Thanks.
Ok, I'll try to contact Kaspersky after my vacation but if they will ask me to clarify how exactly does this indexing process work I'll not be able to clarify anything for them, so I thought that it would be better for your engineers to contact Kaspersky engineers. But if you say that I must do this myself, I'll try:)
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

[MERGED] Veeam indexing

Post by rimvydukas »

Hi,

Can anyone explain me, how does exactly Veeam indexing process work on virtual machine. I have a problem with my not so busy Windows 2003 virtual servers - daily increments are somewhere between 3-10GB. I removed antivirus, I have no disk defragmentation software, I disabled tivoli tsm client on several servers. I ran filemon and saw one interesting thing - when veeam starts its indexing of virtual machine files I can clearly see that VeeamOibIndexer process creates a lot of tmp files on guest server disk. I suspect that there will be more changed info because of this and because of this I'll have large next incremental job size. So how exactly does this indexing process work on vm? Thanks.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam indexing

Post by veremin »

Assuming that the issue is related to indexing stuff, may I ask you try to back this VM up without indexing and see whether large incremental is reproducible without Indexing option or not? Though, please be aware to run the job without indexing at least two times.Thanks.
Post Reply

Who is online

Users browsing this forum: BackItUp2020, Google [Bot], ybarrap2003 and 293 guests