full backup file size too big

[PKaufmann]

Hi,

I have a strange issue here, hope someone has an idea or can help..

We are using a DataDomain as the repository.
Today I was checking the backup file sizes via "files" in the backup&recovery console.

The full backup files (vbk) of all jobs with incremental forever (no synthetic full) are way too large.

Examples:
VM Size: Used Storage in vcenter: 892,64 GB
VBK File on DataDomain: 10,2 TB

VM Size: Used Storage in vcenter: 49,99 GB
VBK File on DataDomain: 2,0 TB

I don't know what's going on. Looks a little bit like the merging process is not working correctly and the VBK File just becomes bigger and bigger..
As Dedup on the DataDomain works pretty well, I can't tell if we loose some diskspace because of this..

Is there kind of a cleanup-task for this or do I have to create new active/synthetic fulls to get rid of these large files?

Thanks for your help.

[PTide]

Hi,

That is definitely something you should show to our support team, have you done that? Also, have you tried to run compact operation?

Thanks

[foggy]

When performing merges on a forever forward incremental chain to a DDBoost repository, Veeam B&R appends data to the file rather than re-uses unused space inside it, for performance reasons. That's why periodic synthetic fulls are recommended in this case, since they prevent full from growing infinitely.

[PKaufmann]

When performing merges on a forever forward incremental chain to a DDBoost repository, Veeam B&R appends data to the file rather than re-uses unused space inside it, for performance reasons. That's why periodic synthetic fulls are recommended in this case, since they prevent full from growing infinitely.

so that is a known issue in combination of DataDomain and Veeam ?

That is definitely something you should show to our support team, have you done that? Also, have you tried to run compact operation?

If it's a known issue, makes no sense to open a support case, right ?
I will try the compact operation with one backup job for testing. thanks.

[foggy]

so that is a known issue in combination of DataDomain and Veeam ?

Not only DataDomain but other dedupe appliances as well.

[PKaufmann]

compact is working

[Virtualredse]

Hi

Now I know this area is probably not a new topic or point of discussion, but I am trying to get it clear in my head so we can better manage some secondary destination capacity.

We have a rather large Linux job which takes around 200 VMs nightly and then we run an offsite backup copy job after this to a DDomain device, with 14 days retention. What I have noticed over the past few months is the rate of consumption has been incredibly high. After starting to dig a bit deeper I found a fairly large number of VMs had a huge "Backup Size" on disk, an example being a 2TB RHEL VM with Backup size listed as 78TB under the copy job properties. This is the largest I could find but many VMs are listed as between 15TB and 30TB for 500GB on disk which when pooled together represents a very large proportion of the storage.

So, I am wondering how other people are managing such growth as incrementals are continually merged ?
Are people scripting to schedule an active full and refresh the full in the chain or are people using Full backup maintenance for removing items / defragmenting ? and if so how effective is that in keeping things at a manageable level.

A customer I am working with is growing fairly rapidly so requirements will only increase in the near future and I would like to get some maintenance practices in place asap

Appreciate any thoughts

[foggy]

Hi Matthew, please see above for explanation. To avoid this behavior, it is recommended to enable periodic compact operation or have GFS retention in place (new fulls are created there as well while old ones go away according to GFS retention).

[Virtualredse]

Ok thanks for that Alexander, so those are the available options to us ..... now to startup a cleanup operation and get one of those practices in place!

Cheers

[chsuscale]

Hi there,

Ive got a VM *130* which is configured 100GB. It's a linux VM. Can anyone help me explaining why the vbk file is that much bigger than the data?
At the "initial" Backup Job al is correct. This "problem" is only there at the Copy Job to the StoreOnce.

[DGrinev]

Hi and welcome to the Veeam community!

The issue comes from the Catalyst StoreOnce storage that doesn't support the cleaning of the removed data blocks. Thanks!

[chsuscale]

Thanks for your answer - did not realize that I got an answer :S
Is there a solution for that?
Or should I test CIFS instead of StoreonceProtocol?

[foggy]

Since you already have GFS retention enabled, this prevents the latest full (which is the part of the regular chain) to grow infinitely - each time GFS full backup is offloaded, the regular one is created from scratch copying only those blocks that comprise the required VM state and skipping unused ones. GFS restore points though will still contain unused data blocks (you can see this on your picture), so there's some space overhead.

[chsuscale]

I'm not sure if I understand you correctly, for me it looks like growing is more or less ifninitely..

[veremin]

But the screenshot proves the foggy's assumption:

- GFS full backups the ones marked with M(onthly) and W(eekly) labels are growing in size, since they contain unused data blocks
- R(egular) full backup occupies much less space

Thanks!

[foggy]

And overall growth of the full backup file with time is also expected since data inside guest OS tends to grow.

[chsuscale]

so in combination with storeone there's no possibility to "reset", "compact" or whatever?
The machine size itself in vmware is 100GB

[foggy]

Right, compact is not supported on StoreOnce. Could you please check what is the size of the regular full at the moment it is created on the day when GFS full is scheduled at (prior to any increment merges)? It should be less than the amount of original data due to compression.

[chsuscale]

for clarification:
you mean, if "Full Backup Restore Point" is setup on Sunday. I check the latest size of full backup before Sunday?

[veremin]

Nope, the size of regular full backup right after GFS restore point will be created on Sunday. Thanks!

[chsuscale]

So this is the view:

You're right that the first full after GFS is correct. But the others are growing infinitely.
Last time we were at 223GBs max now we are in 272GBs.

[foggy]

As I mentioned above, the overall growth of the full backup file with time is expected depending on the data change pattern inside guest OS.

[mytsk]

Sorry for necroing this thread but just realized we're facing the same issue with our StoreOnce and I have a follow up question.

After merge is completed and old data is removed I presume what is left in the file is white space (zeros) yes/no?
If there really is white space gaps in the files how is StoreOnce looking at this, would it be deduped or will it consume space?

When looking at the catalyst items in the StoreOnce perspective all we can see is the data size, what the file size actually is after deduplication cant be seen. Any ideas how to acquire such info (actual or guesstimate)?

Thx!

[foggy]

Unused blocks inside backup files are marked as free and can be re-used later, upon subsequent merges. From the storage perspective though, I think those are seen as regular data blocks since the storage doesn't have visibility into the files themselves.