Comprehensive data protection for all workloads
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Large VIB file on a small static server

Post by Yuki »

this is driving us crazy.

On our 10TB server, last friday's backup is 1.6TB again! Even though only about 150GB of files changed friday AND saturday. How can we find out what is changing? We know users didn't type up 1.6TB worth of new documents, and our space usage did not increase 1.6tb.

What's going on? How can we figure out what is happening?

Win 2012 Standard
File server
No Anti-virus
No defrag
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Large VIB file on a small static server

Post by Gostev »

Is it possible that the file server's file system is fragmented? In this case, changing 1MB file can easily results in 10 1MB blocks changed (10x of the actual file size).
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Large VIB file on a small static server

Post by tsightler »

Aren't you running dedupe? When are you running chunk store scrubbing and garbage collection? Those processes will make a large number of changes across the entire disks. I think the default is Saturday morning (or Friday night, based on how you look at it) at 2:45AM for Scrubbing, and 3:45AM for Garbage Collection. You can use "Get-DedupSchedule" to see the schedule. That's really the most likely culprit and is simply a side effect of running dedupe. What is the size of a more typical night?
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Large VIB file on a small static server

Post by Yuki »

A typical day would produce between 100-300GB.

It could be fragmentation mixed with dedupe, but at this point we are moving away from veeam as vmware level backups are not a good solution for this system. Not an issue with Veeam directly, just a bad match of existing needs and technologies involved.
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

[MERGED] Any guides on what's "touching" files (large inc. s

Post by pkelly_sts »

I have a file server that only generates around 1gb new data a day maximum but which is generating 36Gb+ incremental backup sizes and I can't figure out why.

There's no A/V installed on this server so it isn't an AV scan modifying last accessed timestamps
There's no defrag job scheduled/running

Because it's veeam that's effectively highlighting to me that /something/ is changing somewhere, thus causing a lot of changed blocks, it's a little difficult to search on in the big-wide-t'internet.

Does anyone have any other clues they could point me towards?

Paul
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Paul, this topic should give you some hints to check. Though, feel free to ask for additional clarification, if required after reviewing it. Thanks.
eric
Enthusiast
Posts: 39
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

[MERGED] Abnormally large incremental backups

Post by eric »

Hello All,

Running VBR 7.0 R2 as a test.

I just backed up a domain controller, which has very low volume, subsequent incremental backups are around 600MB.

That seems too large of a change in just a few hours.

Any thoughts?

~eric
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Large VIB file on a small static server

Post by veremin »

Hi, Eric,

Your post has been merged into existing discussion regarding similar issue; so, please, take a look at the answers provided above. In general, VMs running special applications such as DC, SQL, Exchange are notoriously known for producing large incremental files. Another possible reasons include regular anti-virus scans or de-fragmentation activity.

Thanks.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Large VIB file on a small static server

Post by Gostev »

Antivirus scans are read only, so I don't see how they can introduce disk changes?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Large VIB file on a small static server

Post by veremin »

As far as I know, some anti-virus program might change archive bit of a file, or file attributes, in general. This results in increased amount of changes. Some users have also claimed previously that disabling antivirus activity resulted in decreased incremental files.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Large VIB file on a small static server

Post by dellock6 »

This sounds really strange to me too. Antivirus are supposed to be a problem for IOPS, but not for changing files. In this way, an antivirus can also cause problems to replication or backup software installed at the guest OS level, and is a really bad behaviour...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
pkelly_sts
Veteran
Posts: 600
Liked: 66 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Large VIB file on a small static server

Post by pkelly_sts » 1 person likes this post

Just as an addendum (apologies if already covered in this thread, I've not read through it all) I was recently looking into a similar issue on one of our servers and I think I've found the culprit to be DFS replication. Even though we have a 4Gb DFS cache thingy configured (can't remember proper name right now) I was watching the cache whilst trouble-shooting & noticed it create a 15Gb temp file there which it repeated a few times.

I've not /confirmed/ this to be the cause but I strongly suspect it to be a big contributor in my case...

Paul
speedline
Lurker
Posts: 1
Liked: never
Joined: Jan 31, 2014 9:57 pm
Full Name: SL IT Admin

[MERGED] Any way to determine why my backups are so large?

Post by speedline »

I have a job that backs up 7 virtual machines from ESX. It takes around 12 hours to replicate them off site with "optimal" compression. So far it's 12% through the second to last virtual machine and it says that it's 65% complete and has already moved 35.6 GB of data. The servers it's backing up don't change a heck of a lot, or at least I don't think they do. I've been trying to find way of determining what's changed. Initially I had thought I'd solved the problem with all my SQL servers making backups on themselves (the setup before we had Veeam) but apparently there's still a lot of block change. Any way to track and record what's changing, or which server is causing the most data change?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Any way to determine why my backups are so large?

Post by Gostev »

One thing you can do is disable compression, create a test job that uses incremental backup mode, and try to make sense out of the raw content of VIB file beyond the first 8MB. May be you will be able to recognize the content as belonging to some application.
Eagleman
Lurker
Posts: 1
Liked: never
Joined: Mar 26, 2014 12:05 pm
Full Name: Jan

[MERGED] Big incremental backups for Veeam backup server

Post by Eagleman »

I;ve been using Veeam for over a month and i really like it, i was using ghettovcb before and this felt like a huge step in time.

I do however have one problem.
Whenever i backup my "production machines" i also backup the machines that has Veeam Installed (Windows Server 2012)

Here is a picture of the backup results:
[EDIT] We had to delete a reference to the picture, because Google Chrome started blocking this page due to the image sharing provider chosen by the poster. Sorry about that.

As you can see the Windows Server is eating up 75% of the total backup.
This Windows Server is only using these roles: Ative Directory and DNS.
I think Veeam is somehow changing a lot of data during backups and then the incremental backups will be very large only because of Veeam.

Can i somehow stop these incremental files being so large?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Large VIB file on a small static server

Post by Vitaliy S. »

Veeam does not modify any blocks, so it cannot influence the size of the incremental files of the VMs you protect. Please review this topic for possible reasons of this high change rate. Thanks!
blodsbror
Enthusiast
Posts: 49
Liked: 3 times
Joined: Apr 02, 2014 7:40 pm
Full Name: Evan Williams
Contact:

[MERGED] CBT issues

Post by blodsbror »

Hi,

I have a job with 12 Windows 2003 servers (IIS webservers) that each have a 250GB 'data' disk comprosed of website data. For each incremental backup with CBT enabled, it reads about 1TB of data and generates around 400GB of backup data on average. This number seems really high, as there absolutely wouldn't be that amount of changed data on these servers each day. Even when I run an incremental straight after the last, approximately the same amount is backed up each time - when really the number should be very low (such as 1-10GB).

What could be causing this large incremental backup size ? I know there isn't that amount of changes in the data - so is there any tips to improving this ?. At this rate, I'll need a huge amount of storage to have just 2 weeks retention.

many thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Evan, please review this thread, probably will give you some hints to justify the amount of changes.
metrobg
Lurker
Posts: 1
Liked: never
Joined: Jul 20, 2013 11:45 pm
Full Name: Gilbert Graves
Contact:

[MERGED] Identify cause of high CBT rate

Post by metrobg »

We are using VMware ESXI 5.1u1 as part of a disaster recovery system for a client. We are currently using VEEAM Incrementals to backup the data to a local data store.

VEEAM is backing up 30 - 40 GB a day, which I believe is based on VMware CBT, on one of the VMs. This VM is a file & Exchange server.

What would be useful that we have not found yet is a program that will identify which programs on the Host OS (Windows 2008) that are creating all of the changed blocks. Does anyone know of such a program?


We have made a copy of the entire environment in our lab and with no users accessing the file/exchange server we get similar amounts of data that needs to be backed up.

Any Help is appreciated.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

Gilbert, please review this existing thread for some hint on what could cause such amount of changed blocks. Also, since Veeam B&R is a block-level solution, there's no ability to correlate those blocks to the applications inside the guest OS.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Large VIB file on a small static server

Post by Gostev »

Exchange and SQL servers generate tons of transaction logs, which is the main reason for large incremental backup size. Plus, both make many small and quite random updates of their respective database files, which also contributes to the amount of changed blocks.
ColtsFanMN
Influencer
Posts: 17
Liked: never
Joined: Oct 28, 2013 3:49 pm
Contact:

[MERGED] Backup File Sizes are too Large V6.5

Post by ColtsFanMN »

I'm having an issue with my backup file sizes.. it's not an error yet, but at this rate, I will be out of disk space soon.

I am running B&R 6.5 right now, full backup each day, in a VMware environment to removable media.

My backup sizes are 'out of range' for the actual size of the data on the drives, even without deduplication (which I am using).

All the servers are on 'thick provisioned' drives in VMware. There are no 'orphaned' snapshot files present.

For instance, this are what I am seeing:

Server 1:
Veeam shows:
Drive size: 225GB
Read size: 231GB
Transferred: 126GB
Server shows:
Drive size: 225GB
Used size: 76GB

So VEEAM is showing transferring 50GB more data than is actually present on the drive, even without deduplication.

Server 2:
Veeam shows:
Drive size: 525GB
Read size: 523GB
Transferred: 326GB
Server shows:
Drive size: 525GB
Used size: 290GB

So VEEAM is showing transferring 35GB more data than is actually present on the drive, even without deduplication.

I have 13 VMs backing up, all with similar numbers.. How is VEEAM backing up more than is actually present on the servers? Also, I will note that I deleted 30GB of old files off the first server I listed, this made NO change in my backup file size the next day (or even weeks later). What is going on?

Please help!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Large VIB file on a small static server

Post by Vitaliy S. »

Hello,

Please take a look at the third page of this topic, where Tom explained the behavior you most likely experience.

Thank you!
ColtsFanMN
Influencer
Posts: 17
Liked: never
Joined: Oct 28, 2013 3:49 pm
Contact:

Re: Large VIB file on a small static server

Post by ColtsFanMN »

Vitaliy S. wrote:Hello,

Please take a look at the third page of this topic, where Tom explained the behavior you most likely experience.

Thank you!
NO THIS DOES NOT APPLY. PLEASE READ MY POST AND RETURN IT AS A SEPERATE POST!

THANKS.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Large VIB file on a small static server

Post by Vitaliy S. » 1 person likes this post

ColtsFanMN wrote:Also, I will note that I deleted 30GB of old files off the first server I listed, this made NO change in my backup file size the next day (or even weeks later).
The reason why your post has been merged to this thread can be found in this quote:
tsightler wrote:If you create a 40GB virtual disk, add 20GB of data to it, and then delete it, then take a Veeam backup, the Veeam increment will 20GB (assuming no compression)..... For example, I was once able to use a Veeam backup to recover a file that had been created and deleted on the same day by using an NTFS "undelete" program on a Veeam backup from a week earlier. This is something that would have been completely not possible with a traditional backup product. Also, a Veeam backup of a powered off VM is a forensically accurate image of the system, which can be very valuable for security related analysis.
Please be aware that with NTFS, when you write data to the disk and then delete it, the data doesn't actually get removed from the disk, so virtual disk blocks still contain the same dirty data blocks. If you want to make your full backups smaller, then you should sdelete your virtual disks. Please search these forums for existing topics describing how to use this utility.
ColtsFanMN wrote:NO THIS DOES NOT APPLY. PLEASE READ MY POST AND RETURN IT AS A SEPERATE POST!
It seems like the explanation above about "dirty" blocks on NTFS does apply to your case. Thanks!
out2getyou
Novice
Posts: 3
Liked: never
Joined: Jan 05, 2015 3:56 am
Contact:

[MERGED]Backup Exchange 2010 DAG - files too big!

Post by out2getyou »

Hi All,

I currently have two mailbox servers in a DAG. I created 1 job that contains both passive and active mailbox servers. The initial full backup (which contains both servers) was about 1.31 TB. Every increment after that (1 daily incremental) for 1 week varied between 70-100 GB (both servers). So the obvious question is, why are the incremental backups so big? Surely we cannot be generating 30-50 GBs of email per day? we only have 200+ staff . Thats close to 5-7 GB per person!

Is there a way to make these incremental smaller? its eating up my storage :(

Thanks guys.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Large VIB file on a small static server

Post by Dima P. »

Hello,
I’ve merged your post to the existing discussions. If you suspect increment growing unpredictably large, please review this thread for possible solution. Thank you.
deduplicat3d
Expert
Posts: 114
Liked: 12 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: Large VIB file on a small static server

Post by deduplicat3d »

I have an open case 00746249

They are insisting that I am changing files and that is why I have large incrementals. I had the problem saturday morning and tuesday morning, these are forever incremental backups. Most VM's appears not to have this problem but 2, that are on separate hosts separate domains have the issue. Some jobs are not impacted at all. I have not done any defrag, disk check, virus scanner hasn't changed settings. I've browsed throughout the servers and almost all of the files have old modified and accessed times (it's a file server, people are usually only changing and accessing current documents).

I can't have this happen again as it will fill up the disk on the backup server. Is there a way to cross reference the blocks that have changed with files or to do some sort of analysis. I uploaded logs to support and a bunch of screen shots.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Large VIB file on a small static server

Post by foggy »

As it is mentioned above, there's no ability to correlate the changed blocks with files. What you could do to verify what's changed, is fire up two restore points using Instant Recovery and compare files between them. There's also a couple of other guesses in the referred thread, might give you some thoughts.
tomhkr
Enthusiast
Posts: 42
Liked: 9 times
Joined: Feb 03, 2014 7:40 am
Contact:

[MERGED] Troubleshooting backup size

Post by tomhkr »

I have a virtual Windows 2012 R2 server acting as a file server for users home directories.

Yesterday I looked at the backup sizes this server generates and they are questionable large.
The volume being backed up is sized at 2.3 TB and has about 3,600,000 files on it.
It is a NTFS volume with an allocation unit size of 4096 bytes. Residing on an VMFS-volume with block size 1 MB.

Yesterdays incremental backup of the volume, using CBT, was 23.9 GB.

Looking at all files that were either modified or changed yesterday, they are only about 2 GB in total.
Total of about 3500 files were created or modified, of these about 3100 were smaller than 64 KB and 2500 smaller than 4 KB.
I understand that CBT takes whole changed blocks but this is ridiculous. Even with an assumed CBT block size of 1024 KB; 3500 files would be 3.5 GB, not 23.9 GB.

How can I troubleshoot and find out what is causing this massive leverage in backup size?
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 225 guests