Removed vm from job, VBK and VIB files still same size

[RGijsen]

HHi,
lately we are struggeling with space on our main Veeam repository. New hardware is not in place yet, so I have to be conservative. This week one of our Exchange DAG nodes failed, and it was replaced. I have two nodes in that DAG, both of which are backed up in one job. As the whole node was rebuild, Veeam read all disks of the new vm again, resulting in complete disk full. As all mail data is redundant in that job anyway, and the replaced VM stores about twice as much data as 'needed' now, I decided to delete the crashed VM from the backup in order to free space.
So I removed that VM from the backups, but practically no space, if any, was freed up. As you can see in the image below, there are now only 2 VM's in that job, the failed node was the third and I deleted that from the backup. The 30GB VM is an Edge server by the way, which hostname happens to end with ex002 as well, the 810GB one is the other actual DAG node that was still working. You can see the VBK of 2018-12-16 is bigger than the combined spaced of the VM's (955GB while the VM's in total only are 840GB) and the vib of 2018-12-17, so an incremental, and the first backup where the new VM is in, after deleting the crashed VM's data, holds 41.8GB data but the actual file size is 620GB.



So why isn't that space freed up? What's the use of deleting VM's from a job then on the first place? I know it frees up license slots, so that's one, but it should free up the disk space as well? I wonder what's still in that VIB file, as the incremental data of the Edge server and the remaining node is only 41.8GB. Is the data actually removed at all? How to free up space? I can't do any defrag jobs, as I have periodic fulls it tells me.

[Paulo Balau]

I had a similar issue… Curious about the answers

[PTide]

Hi,

What settings do you have in "Advanced" ? Specifically, "Remove deleted items data after".

Thanks!

[RGijsen]

Not checked, neither is 'Defragment and compact', which is available in my main-job, but not in the copy jobs.
I'm just re-reading the manual about these options, but what do you suggest? Just the delete-data or the defragment as well? If I'm correct, the maintenance jobs are ReFS block-cloning aware, so I'm not defating into another disk-full?

[PTide]

"Defrag and compact" cannot be turned on if periodic full is enabled (no matter if BCJ or BJ).

You can see the VBK of 2018-12-16 is bigger than the combined spaced of the VM's (955GB while the VM's in total only are 840GB) and the vib of 2018-12-17, so an incremental, and the first backup where the new VM is in, after deleting the crashed VM's data, holds 41.8GB data but the actual file size is 620GB.

I am just trying to wrap my head around the situation... So, first you had replaced the failed node, then there was 633GB full (16-12-2018), after that you deleted the node that you had previously replaced from the job (at this point you have only Edge and the other node that hasn't crashed in the job).

And after that you got 620GB .vib, is that correct? Would you also provide pre-vm 'Read' and 'Transferred' figures from the report (you can get one if you right-click on the job and select 'report').

Thanks!

[RGijsen]

Ok. When I read the manual it says the 'Remove deleted items data after' only works when the VM isn't processed for n amount of days. However, that same VM with the same GUID is still in the job, as I re-installed that actual VM with the Windows ISO mounted, as I wanted to keep the Exchange database disk in this case in order not to have to sync that all over again. So that VM is still there, meaning the 'Remove deleted items data after' will most probably never trigger.
While not greyed out as in the copy-job settings, enabling 'Defragment and compact' shows an error; 'Defragment and compact functionality' is unnecessary when periodic fulls are enabled.' Ofcourse that's true to a certain degree, the next synthetic full will probably not have the 'lost' VM in there anymore.

Probably the best way to see the situation now is this: I reinstalled a certain VM, keeping it's GUIDs and all. And I deleted it's previous backups by richt-clicking that VM in the Backups --> disks --> that specific backup job screen. As said, the incremental data for the remaining machines should be around 42GB, while the VIB file still is over 600GB. So it's obvious the data is not actually removed from the vib files.

The report for monday 2018-12-17 reads:



Again as you can see, the failed VM read about 675GB before running into disk-full. The other 2 VM's combined have 22.1+7.9=about 30GB. Not sure why there are different figures here than in the properties of the backup job from the 'Backups --> Disk screen. That still shows the following:



And that simply states there is 41.8GB data in a 620GB file. And just to be clear; I removed the xxxxxxxEX001 data by doing this:



I don't know how to get that data out of those files. It consumes what is extremely precious space until new hardware arrives and is put online.

[PTide]

Ok, here is what I think has happened:

1. You've reconfigured the broken node without actually replacing the VM, therefore CBT detected that some data has changed on the node. I am not sure what exactly was done, but the resulting set of changed data turns out to be around 440GB.
2. Next, you have intermediate fulls, which makes it impossible to use "remove deleted items data" function.

In conclusion, there are also two very weird things about the job:

1. As you've pointed out on this picture, the sizes of the last two incrementals (17.12) look weird.
2. Although one of the VMs has failed to backup due to the lack of free space, it is not clear how two other VMs managed to finish backup given that the repo was already out of space.

I suggest you to contact our support team directly so they can look into logs.

Thanks!

[RGijsen]

The other two completed because they just finished before the space ran out, as simple as that.
I'll log a ticket. But still, is my assumption correct that when I delete a vm from the backup files the way I did, it should free up the space? It certainly does a lot of work when I do that action.

[PTide]

But still, is my assumption correct that when I delete a vm from the backup files the way I did, it should free up the space? It certainly does a lot of work when I do that action.

Unfortunately, it's not. Basically it marks blocks (not filesystem blocks, but backup blocks) inside backup files as 'free'. Those blocks will go away only if transform or compact operation occurs. Since you have periodic fulls enabled, those blocks won't go away until the corresponding backups files are deleted by retention. A somewhat unpleasant workaround to gain some space would be to temporarily reduce the retention value so that the trailing backup chain can be deleted earlier.

Thanks

[RGijsen]

Then I'd like to move this into a feature request. Deleting VM's but not being able to free up the space is non-sense. Even more so - I'm not sure what you mean by 'marked as deleted' but in terms of a lot of systems that means the actual data is still there. With regards with all kinds of laws here in The Netherlands like AVG and the European GDPR, that's really a no no. So basically what I want to request is:

- Either deleting a vm from the backup, the data should be removed from the file and release the diskspace to the OS;
- Or we should be able to run optimization / defragmentation jobs even if we have synthetic fulls enabled.

Reason: Maybe even the most important reason is that deleted data should actually be deleted, not only marked as such. But in addition, VIB files might be on the storage for months depending on the retention requirements of a company. All that space in those VIB's should be made availabe. Storage comes at a cost, and for a small company as ours, it's a big cost.

[RGijsen]

Hi,
can anyone at Veeam respond to this? In the past I've asked where to put feature requests, and the answer was 'by just posting at this forum'. So I'd like a confirmation that this is considered a feature request.

[PTide]

Hi,

Those people are right, as that's exactly what this forum is for. That is, the request has been noted. However, you should keep in mind that even if compact/optimization operation was allowed for synthetic fulls, it would still require enough space to store one full backup. Otherwise such operation would be very I/O heavy. Creation of a new method of compacting backups might take a significant amount of time and effort, so no estimate on when it will be delivered.

Thanks!