Large VIB file on a small static server

[stevil]

Last night I tested backup across a measly 2Mbps MPLS WAN link. The server I tested wasa a windows print server. I used best compresssion and WAN Target. The initial VBK file size was only 3.3GB. 12 hours later (at 5am) I reran the job, it took 3.5 hrs to backup, the VIB file was 1.5GB in size, nearly half the original server. I don't understand why the VIB file is so enormous when the print server is very static AND the test was done after hours, i.e. 5PM to 8AM.

Any ideas what's going on here?

Cheers
Steve

[Alexey D.]

Hello Steve,

Did you perform any activity such as antivirus scan, defragmentation, etc. in between job runs? B&R queries the number of changed blocks from VMware, so it looks like that VM was actively doing something..

[stevil]

It would have had another B&R job run on it locally at that time (the normal job, not the test job). I wouldn't have thought that would have changed any blocks though. Antivirus is installed on the server but scans are only done on a Sunday. No defrag jobs are ever run.

[Alexey D.]

Do I understand properly - are you having B&R installation inside your 'small static server'? If so, what are you backing up with it?

[stevil]

No, what I mean is that we have another Veeam backup job which back's up this print server on a nightly basis. I setup another test job to do the WAN test using this printserver as a test. So basically it got backed up twice last night.

[Gostev]

Any ideas what's going on here?

Something changed 1.5GB of blocks on that server... there cannot be any miracles with this - if some block does not change, ESX changed block tracking never picks it up, so the block never gets copied to VIB.

[stevil]

OK, I'll check the VM and see what's going on on it, or chose something else to test

[tfleener]

In this post you asked if an anti-virus scan had been ran recently.

I too have noticed sometimes a VIB file is very large, relative to the amount of actual data changes, but I am wondering why an antivirus scan would cause this behavior.

We use Symantec Endpoint, and perform weekly scans, and I am thinking this could be a possibly cause for the large VIB file creation on some days.

Can you please elaborate on why AntiVirus Scans could possibly cause this, and if you know if the Symantec Endpoint could be the catalyst for this behavior.

[Vitaliy S.]

Tom,

When you run defragmentation, move/copy/delete files, perform antivirus checks virtual disk blocks are relocated, thus meaning that all these blocks will be treated as changed ones during next incremental job run.

Thanks.

[tfleener]

I understand the defrag/move/copy... but I do not understand the antivirus checks (did not know it relocated blocks). I was thinking virus scans a basically a read-only option in most cases.

What brings up my question is specifically regarding antivirus checks. I am trying to verify if Symantec Endpoint updates any blocks during its manual scanning process.

What I have seen is very large incremental backups(VIBS) following our weekend virus scan jobs. I have seen references to Symantec updating the USN journal for NTFS, but that is a single file in NTFS, and it seems it should be be unrelated to CBT in Vsphere.

The only other thought I have is the anti-virus software is using a lot of temporary space for the scan process.

[tsightler]

Well, simply scanning through all your files will generate a lot of changes on the filesystem since the "access time" attribute has to be updated for every file entry. That can create a lot of changes. Of course the changes themselves are quite small, only a few bytes, but Veeam uses 1MB blocks, so for 1.5GB of "changes" you only need to have a single byte changed in 1500 blocks spread across the entire disk. Updating the "access time" could easily generate this much changes, and potentially much, much more.

[Gostev]

Hint, you can disable last access time updates to prevent this from happening.

[Daveyd]

[merged]

I just upgraded to v6 but this was going on with v5 as well. I have several jobs that have large vibs with very little change rate occuirring on the VM.

For example:

1 job
1 VM running Windows 2008 x64 and MYSQL DB.
VM has 3 vmdks, 1 for OS (0:0), 1 for Database (0:1) and 1 for MYSQL Database backup dumps (0:2)
Veeam job is using forward incrementals with weekly, CBT, Page File Exclusion, use inline Dedupe, NO compression (backing up to Exagrid), LAN target and is set to only backup 0:1 and 0:2
The job kicks off at 2:15am every day.

On 2/6 at 2:00am adding the Used MB for both my OS and Database drives, it came to 108,357MB (C: 26,962MB/D: 81,395MB). On 2/7 at 2:00am the total Used MB for both drives was 108,577MB (C: 26,966MB/D: 81,611MB). According to my math, in a 24 hour period, the server grew 220MB.

The incremental ran last night on that server. The size of the .vib was 18.2GB!!

Something does not seem right. Am I missing something here??

[foggy]

Dave, what about disks fragmentation? I would recommend that you try to defragment your VM disks plus running sdelete on them and see whether this helps.

[Daveyd]

I have not defragged or run sdelete on any of the VMs yet and they have been running for quite some time. I assume running defrag/sdelete would be best done right before a full backup? It seems like defragging a vmdk would change an awful lot of blocks if it were heavily fragmented and would result in a pretty big vib.

Also, is there a reason why the v6 Veeam Enterprise Manger, in the Reports section when I go into a job, the "Data Transferred" column does not give the actual amount data transferred like it does in the email report I get in the "Transferred" column for each VM?

[foggy]

High fragmentation results in large sizes of incremental backups. Right, you can schedule it just before the full run.

Regarding the transferred data count, this could be a UI bug. I've passed the information to our QC team. Thanks.

[Daveyd]

Thanks for the reply. It looks like the MYSQL DB server I mentioned above is 90% fragmented. Its going to take a considerable amount of time to defrag. I'm interested in seeing what type of reduction in incrementals I get when it finally completes.

Thanks for the bug check. It would be real nice to see the actual amount of data per VM that was tansferred to the repositiory in Enterprise Manger like the email report shows

[Vitaliy S.]

I would recommend that you try to defragment your VM disks plus running sdelete on them and see whether this helps.

This one is good for thick disks only, I wouldn't recommend running sdelete for thin disks without taking these steps afterwards.

[Daveyd]

Hint, you can disable last access time updates to prevent this from happening.

Can that be done via the AV software or does it have to be done on an OS level/registry fix?

[Daveyd]

This one is good for thick disks only, I wouldn't recommend running sdelete for thin disks without taking these steps afterwards.

In my case, I only have thick disks, no thin provisioned

[foggy]

Can that be done via the AV software or does it have to be done on an OS level/registry fix?

I doubt whether I've seen such settings in any of the AV products... Must be something like this.

[Daveyd]

Well, I have a MYSQL VM that I defragged 2 days ago. Here are my results:

Before defrag:
C: 40GB - 11% fragmented
D: 60GB - 83% fragmented
Average daily growth of both drives: 250MB
Veeam Incremental vib: 4.9GB

After defrag:
C: 40 GB - 2% fragmented
D: 60GB - 2% fragmented
Average dail growth of both drives: 260MB
Veeam Incremental - 5.0GB

Obviously the initial incremental after the defrag was HUGE but the subsequent vib was the same size before the defrag. I have not run sdelete on the VM but with such a small change rate and a defrag, I'm surprised that the vibs didn't shrink

[Vitaliy S.]

Dave,

Unfortunately, I didn't notice that you're backing up a MySQL VM, my bad.

Actually this is perfectly normal for servers with high rate of disk changes (Exchange, SQL, Oracle).

Be aware that for every changed block in MySQL VM Veeam will pick up a 512 KB data block (LAN target), so to reduce incremental file size I would recommend switching to WAN target instead. In this case Veeam B&R will use smaller data blocks (256 KB), which will result in the maximum deduplication ratio and the smallest size of backup files.

Please note that changing this setting on existing job only takes effect after full backup is done.

Thanks!

[Daveyd]

Thank you for the info Vitaliy. My backup target is an Exagrid appliance. Will changing the job to WAN target have any adverse affects on deduplication on the Exagrid appliance...Exagrid recommends using LAN target

[Vitaliy S.]

Deduplication ratio might be lower on Exagrid, on the other hand you will have much smaller VIB files.

The best way to figure out what's better for you is to test it yourself. I would appreciate if you could post back your results so we could compare.

[tsightler]

Also, you can't really compare growth rate to VIB size. Growth rate of disk space only accounts for new data added, but with a SQL server, and really with any server, there are likely to be significant changes to existing data.

[Daveyd]

Deduplication ratio might be lower on Exagrid, on the other hand you will have much smaller VIB files.

The best way to figure out what's better for you is to test it yourself. I would appreciate if you could post back your results so we could compare.

Well, I changed the Dedupe process from "Local target" to "LAN target" and also did a defrag. Here are my results thusfar,

VM prior to Veeam settings change and defrag:

C: 40GB - 9% fragmented
D: 120GB - 74% fragmented
Veeam incremental data backed up - 21.1GB

After Veaam settings change to "LAN target" and a full defrag:

C: 40GB - 2% fragmented
D: 120GB - 2% fragmented
Veeam incremental data backed up - 5GB

The new 5GB incremental size is on a weekend incremental where the DB does still get hit but not as hard as during the week. I will be interested to see how much incremental data gets backed up tonight.

[Daveyd]

Regarding the transferred data count, this could be a UI bug. I've passed the information to our QC team. Thanks.

Any update on this?

[foggy]

Any update on this?

This is confirmed as a bug and will be addressed in one of the future product updates.

[Unison]

Hi guys,
Im having a bit of an issue here with Veeam compared to my old imaging software (symantec System Recovery). The full backups of my servers are smaller with veeam (which is great) compared to the old backup solution however each increment that is done with veeam is much bigger than the increments done with the old backup product - im at a loss as to why this is and its blowing out our archive storage and creating issues for tape backup.
For example, some of our 'quieter' servers used to have increments of around 300-500MB....now with veeam, the increments on those same servers are around 4-8GIG! - each time! This adds up real quick.

I suspect that the reason for the huge increments is the OS page file - veeam keeps backing it up with each increment because it changes frequently. Am i wrong or could there be another cause for these large increments?
I read that version 6 of veeam (which i am running - with esxi5 - backing up Server03 VMs) has the ability to 'ignore' or exclude page/swap files from the backup process - GREAT!!! I have checked the settings on all my jobs and this setting is ticked - but it doesnt seem to be working here.

Has anyone else had issues with extremely large increments? been able to work out what is causing the large increments?
I would prefer to not create a new vmdk just for the paging file if possible - and seems like v6 really makes this unnecessary now - is there anything anyone can think of that might be causing these large increments on a server that does not change very much at all daily.

Info:
-CBT is enabled
-Increments are done daily
-Pagefiles are on the C partition of OS
-No synthetic fulls or reverse incrementals in use - all are base weekly with incrementals through the week.

Thanks guys,
Appreciate any help you can offer