Remove clients from a backup but VBK file never shrinks.

[mschlott]

I had a backup job that was too large for my liking, so I removed half of the clients from the job. The job is a reversed incremental that runs every day. I expected that during the next backup, I would get a large vrb file and have the vbk file shrink by about half, and then have reasonable sized vrb files following that.

What I experienced was a drop in the size of my vrb files, which I expected, but no significant change in the VBK. It has now been a couple of weeks since I have removed the clients from the group and the retention period for backups and deleted VMs has expired. Looking in backups, I only see the VMs which are currenly in the backup job, none of the VMs that were removed. The VBK file is about 860 GB. The estimated backup size according to the edit backup job window is 639G which after compression and deduplication should create a VBK file under 300GB. According to the backup properties window, my backup data size is 1.59TB and the backup size is 820GB. None if this matches what my backups should look like.

I called support about this and was told that this is normal and that I should run a new full backup any time that I remove a client from a backup. I am backing up over the WAN and the main reason I bought Veeam was to be able to backup over the WAN. If I have to run a full backup over the WAN any time I make changes to backup jobs, that is not going to work for me.

Can anyone confirm that this is infact true? If not, is there some way I can clean up this VBK file and not have to run a full backup to free up disk space?

Thanks in advance.
mike

[Gostev]

Hi Mike, this comes down to the NTFS design (and all other commonly used file systems are no different either). The only way to reduce the size of any file is to recreate from scratch without writing the blocks you do not need. And this exactly what active full backup is doing.

You may consider switching to incremental backup with syntethic fulls for backup over WAN. Do not enable transform. In that case, each week there will be new synthetic full created, and it will consist of only actual data blocks. Of course, you would have to pay for this with having more than 1 full backup on storage. Nothing comes for free

Thanks.

[mschlott]

Does this mean the I could add some VMs to this job and the VBK file will not grow? Are there unused blocks in this VBK waiting to be used again? I guess that makes sense. A defrag feature would be nice. Maybe in V7?

Thanks
mike

[Gostev]

I have added something else to my response above as well while you were responding to the original text.

Does this mean the I could add some VMs to this job and the VBK file will not grow? Are there unused blocks in this VBK waiting to be used again? I guess that makes sense.

That is correct, those blocks are marked as unused and simply reused with the new data.

A defrag feature would be nice. Maybe in V7?

Defrag is not impossible, but would require disk space to hold 2 full backups (old and new) at the time when it is run (which many customers using reversed incremental usually do not have). And if you do have disk space for 2 fulls, then such defrag would be no different than what you can achieve today by using incremental backup mode with synthetic fulls.

[mschlott]

So if I have space, I could switch my backups for one day to run incremental and a synthetic full and I would have a clean VBK file. That might work for me since I have a short retention period on that job, I would only have to maintain 2 copies for 2 weeks.

Thanks for the pointer. I had not thought about this.

mike

[RH832]

We have a backup job that backups up 13 Windows 2003 VMs using reverse incremental and 7 restore points. The job has been in place since December 7 when we upgraded to v6. At this time no drives were excluded from the VMs. A couple weeks later we decided to exclude one of the drives, vmdk, in one of the VMs which contained 200GB of data. Our expectation was the data would drop off from the VBK but here we are a month later and the VBK is still about the same size as the original. I am assuming Veeam does not remove data and I assume a new job with the same configuration would show a decrease in size. Is there a way to keep the existing job and for it to force to drop that data?

[markfellows]

I'm experiencing the same issue as a few people in this thread. Around 50Gb of files was removed from the machine which was being backed up and I expected the .VBK to shrink once a full backup was performed. I’m using Reversed Incremental, I scheduled an Active Full Backup a few days ago which ran but didn’t reduce the size of the .VBK.

Should this have worked?

[RH832]

Should scheduling Active Fulls decrease the vbk size on reverse incremental backups?

[foggy]

Yes, but you need to run sdelete on the VM first to zero all unused blocks.

[markfellows]

Alexander, I've contacted support and they've provided me with the response below :

My apologies for the delay in response.

Actually it's not a requirement and even not related to Veeam software. It's just how NTFS works.

I believe it's safe to run sDelete on a production VM. However to get a proper support on this please contact its manufacturer.

Also the following threads may be helpful:
http://forums.veeam.com/viewtopic.php?f=2&t=10437
http://forums.veeam.com/viewtopic.php?f=24&t=10004
http://forums.veeam.com/viewtopic.php?f=2&t=4017

In these responses it implies that due to how NTFS works it is normal for this to happen to the file (these particular links are mainly describing .vib files).
So with this in mind should a full back up reduce the size of the .vbk?

[foggy]

Mark, VBK size should be reduced after re-creating the actual file (i.e. after performing the active full). If this does not happen in your case, please explain this particular issue to the support engineer clearly (I looked through your support case - you haven't stated that the active full did not help).

[markfellows]

Thanks Alexander, I'll do that.

[rsargeant]

[merged]

Hi guys,

Just wondered if there was a way of 'compacting' a VBK file once a server(s) is removed from a backup job?

This would be a really useful feature so disk space on the backup media is kept to a minimum.

Thanks in advance.

Richard.

[tgiphil]

Hi Mike, this comes down to the NTFS design (and all other commonly used file systems are no different either). The only way to reduce the size of any file is to recreate from scratch without writing the block you do not need. And this exactly what active full backup is doing.

You may consider switching to incremental backup with syntethic fulls for backup over WAN. Do not enable transform. In that case, each week there will be new synthetic full created, and it will consist of only actual data blocks. Of course, you would have to pay for this with having more than 1 full backup on storage. Nothing comes for free

Thanks.

Are you aware of the Window's SetEndOfFile function? It can reduce the size a file. Of course, this only helps to trim the end of a file.

However, NTFS supports Sparse Files. If VEEAM writes large blocks of zeros to the unused space, Windows OS will release blocks with all zeros back to the file system. The file size does not change, but the amount of disk space used by the file is reduced.

http://msdn.microsoft.com/en-us/library ... S.85).aspx

Another option is to perform a bit of garbage collection and compaction by moving used blocks into free space towards the beginning of the file. And then use the SetEndOfFile to trim the end. (SQL Server does something like this when you shrink a database or log).

I think this would be a great feature/option to add. I would use it.

[Gostev]

Yes, but unfortunately this is limited to NTFS only (backup repository being local disks or SAN LUN). Not going to work for any other backup targets we support, which are more commonly used.

[tgiphil]

Yes, but unfortunately this is limited to NTFS only (backup repository being local disks or SAN LUN). Not going to work for any other backup targets we support, which are more commonly used.

Correct, the spare file feature is only beneficial on NTFS. However, if the underlying SAN has any dedup features, it'll have similar results - but maybe at different levels in the storage/file system stack. Also all file system platforms support changing the length of the file. Linux version of SetEndOfFile() which is truncate(). Should apply to all SANs which provide CIF/NFS too.

Ultimately, I think the product is going to have to address how to shrink/compact the backup files as it attempts to reach futher into mid-size and larger enterprise sized organizations. Re-running a full backup over a WAN (or even LAN) is not always a practical solution - just to reclaim disk space.

BTW. I'm surprised that NTFS is not the most common backup target. What is the most common target? CIFs?

Again, keep up the good work!

[Gostev]

Yes, cheap NAS (so CIFS or NFS via Linux). Not an ideal target by any means, but most affordable. I am actually hoping that v6 Windows-based backup repositories will change that (as it is cheaper to stuff some old server box with hard drives, and this makes it much better backup target too).

Mid-size and large enterprise sized organizations all seem to have policy/requirement to perform active (real) full backup periodically, so for those guys this is certainly not a problem.

I personally don't think sparse file is a good idea for a few reasons. For example, copying such file to tape (since you are talking large size customers) will still consume space equal to the actual file size (as tape is unaware of sparse files). However, SetEndOfFile approach should work fine and does not have this issue (although it does require quite a lot of I/O).

[tgiphil]

I agree that leveraging the sparse file feature isn't the best universal solution - I should have stated that. Compacting requires a ton of I/O - so not an ideal solution either. I'd spend valuable R&D elsewhere.

I would really like to store backup data directly to iSCSI LUNs or VMware data stores where very large spans of disk storage can be provisioned - and just bypass the complications and workarounds necessary when using a file system - setting up partitions, I/O for compacting and shrinking files, working around 2TB limitations (!), etc.

[ChrisL]

Hi all, looking to do much the same as this - having moved a load of data from a VM I want to now reclaim that 'free' space. Will probably end up running a defrag on the VM and then using sdelete, which seems to be the general advice.

Few thoughts though.. I'm running reverse incrementals on the jobs, do I need to run sdelete and then specifically run a new full backup (or even start a whole new job and backup chain) to gain the benefit of a smaller VBK file or can I just let the job run as normal? I realise this will most likely create a huge VRB rollback difference file but that's not really a problem if the process is quicker than running a complete new full backup.

As a complete alternative. Can I run a defrag on the VM at OS level and then simply move the VMDK from one datastore to another? If I move the VMDK file and set it to thin provision, would this have the effect of 'rewriting' the file, and reclaiming the garbage space at the time, or does it just copy verbatim the original, including the garbage 'unused' blocks and not actually change anything at all?

[foggy]

I'm running reverse incrementals on the jobs, do I need to run sdelete and then specifically run a new full backup (or even start a whole new job and backup chain) to gain the benefit of a smaller VBK file or can I just let the job run as normal?

In case of reversed incremental mode you need to perform an active full backup as simply running sdelete won't reduce the VBK file size, but just mark all the blocks belonging to deleted stuff as unused (so that they could be reused by some other data). This is how NTFS works.

As a complete alternative. Can I run a defrag on the VM at OS level and then simply move the VMDK from one datastore to another? If I move the VMDK file and set it to thin provision, would this have the effect of 'rewriting' the file, and reclaiming the garbage space at the time, or does it just copy verbatim the original, including the garbage 'unused' blocks and not actually change anything at all?

You have to do sVmotion for thin disks as otherwise running sdelete will expand them.

[ChrisL]

Thanks Foggy, good to know. Running the 'sdelete' sweep now, taking a while, it's a 1.17TB vDisk to trawl through but should be worth it.

I'm dealing with a Thick disk here though (we prefer to have a hard limit and not allow the vDisk to expand beyond the datastore capacity if its been over-provisioned!) so do I need to do the second step of vMotioning off and back on again to have the effect? I assume that if I thin-provision it off to elsewhere and then thick-provision it back where it came from then there's no real overall difference as far as vSphere is concerned - it's still be a 1.17TB VMDK file at the end of it all.

If it's a thick disk, can I just run 'sdelete' on the VM and then run the Active Full in Veeam to get the smaller VBK file?

[foggy]

If it's a thick disk, can I just run 'sdelete' on the VM and then run the Active Full in Veeam to get the smaller VBK file?

Yes, that should be enough for thick disks.

[deduplicat3d]

I'm getting giddy thinking about replacing my slow crappy dedupe device with a Dell server with Windows 2012 dedup

[eiskra]

I'm trying to deal with this same issue and need clarification -

I have a job that used to back up two 850 GB servers, doing reversed incremental. The .vbk is about 1.3 TB in size. (This is two Exchange servers with DAG, so the bulk of that data should have deduped, but despite multiple efforts, dedupe is just not being very effective. And yes, I've ready the threads on dedupe. I've been active in several. We've given up on getting VEEAM to dedupe these DAGs.)

We removed one of the 850 GB servers several weeks ago, so now the total backup is only 850 GB... but teh .vbk is still 1.3 TB in size.
We only have about 250 GB free space in the repository. We would like reclaim the space.

The question: If I set the job to do a periodic active full, the new .vbk will be much smaller than 1.3 TB... but will it REPLACE the current .vbk, or will this ADD a new .vbk in the repository? Obviously, I don't currently have the space for a whole new .vbk to be created!

[foggy]

The question: If I set the job to do a periodic active full, the new .vbk will be much smaller than 1.3 TB... but will it REPLACE the current .vbk, or will this ADD a new .vbk in the repository? Obviously, I don't currently have the space for a whole new .vbk to be created!

The new VBK file will be created in the repository. The old one will go away according to your retention policy.

[eiskra]

That's conflating things, and consequently, I'm not clearly understanding how things work. If I understand you properly:

1. In my circumstance, VEEAM will try to create new .vbk file, and the old 1.3 TB .vbk will not be deleted or overwritten at the time this is done.
2. In fact, the new VBK file will NOT be created, because my repository does not have adequate space. (It would likely require about 588GB.)
3. If there WERE enough space for VEEAM to create the new .vbk, I would have two .vbk files in my repository until the older one aged out based on retention policy.

Are these three sentences correct? I want to be clear on that before I ask one more follow-up question.

[foggy]

Edward, all of your statements are correct.

[eiskra]

Thanks. I'll have to find a way to move things around to different storage until I'm past this hurdle. I can't just add to this storage pool, because it will be non-trivial to shrink it, later. I believe I'll copy the old files out of the repository and have the job do an active full; I'll manually expire (delete) the old .vbk later. If there's no call for an emergency restore along the way, I should be fine, and if there is, there are ways to deal with that.

Last thing to clarify:

If I have a Reverse Incremental job, like this one, and I set it to make a period active full, does it use that periodic full as the basis for the Reverse Incrementals going forward?

In other words, the periodic full creates a new .vbk. When the next reverse incremental AFTER the periodic full runs, does it update that same .vbk?

Trying to understand if I'm out of the woods if I have the system do a periodic full, this one time. Maybe I'm better off just copying the backup files elsewhere and re-creating the job.

[foggy]

If I have a Reverse Incremental job, like this one, and I set it to make a period active full, does it use that periodic full as the basis for the Reverse Incrementals going forward?

Yes, active full resets the current backup chain and starts it anew meaning that the next incremental will be based on the new VBK file created by active full run.

[eiskra]

Last question, in the vein of this thread:

If you set up an Incremental Job with a Synthetic Full every day, the result on disk is the same as a Reversed Incremental.
Is it correct to assume that the job will suffer from the same predicament discussed in this thread - that even if a server is removed from the backup job, the space in the .vbk file remains consumed?