Disk Space Issue

[SSR]

Hi folks,

I'm having an issue with disk space on our NAS.

I changed one of our file servers (Hyper-V VM) by adding a new drive; I had to add a new drive with GPT, and migrate the data from the original data drive, because said drive was running out of space and couldn't be increased due to the 2TB limit for MBR.

We only have 1TB of free space on the NAS where Veeam backs up to.

I detached the old data drive from the VM. Understandably perhaps, Veeam treated the new drive as exactly that, and tried to add it to the backup, which then exceeded the free space on the drive. To try and resolve this issue I dropped the restore points on the file server backup to one. We have a further backup system, so it was not an issue to reduce the restore points to try and resolve the issue. It did free up some space, but we have less free space than we did before. I have also changed the restore points on the Replicas job to one, and it's still much the same in terms of disk space usage. This was previously on fourteen restore points.

Is there a maintenance job I can run to try and clear this up?

Currently we have around 4.5TB of VMs that are backed up. The size of the backups directory is 6,5TB. The size of the replicas directory is 7.5TB.

Any ideas of why so much space is taken up, particularly compared to before and with these reduced restore points, would be much appreciated.

This is using Veeam 8.0.0.2030 on Server 2012R2.

Thanks

[Mike Resseler]

What type of backup method are you using?
When you said that you removed the restore points, does that mean you went into the job settings and lowered the # of restore points?

Thanks
Mike

[SSR]

Hi Mike,

This is just a standard backup job. But let me know if you need more detail.

Yes, I changed the backup job's restore points to one. I have done the same for the replication job.

I upgraded to v9, as I had been intending to do that anyway. I enabled "Defragment and compact full backup file" on the file server job, which are by far the biggest servers, again it has used up all the disk space, and the job has failed due to lack of disk space.

Any ideas?

Although I've upgraded to v9 I've not applied the Update 2 patch, which I will apply now.

Thanks

Mark

[foggy]

Unless you remove older restore points from disk (full and related increments), the space will not be released, since retention applies at the end of the successful job run, which you cannot achieve due to insufficient space.

Currently we have around 4.5TB of VMs that are backed up. The size of the backups directory is 6,5TB. The size of the replicas directory is 7.5TB.

When you're mentioning the replicas directory, do you mean the Replicas folder keeping replica VMs metadata, which is located in the same repository?

[SSR]

Unless you remove older restore points from disk (full and related increments), the space will not be released, since retention applies at the end of the successful job run, which you cannot achieve due to insufficient space.

The older restore points seem to have gone though. What's left is just one very large (currently 6TB) VBK file. Prior to last night the backup jobs did have enough space to complete, and the retention points removed.

When you're mentioning the replicas directory, do you mean the Replicas folder keeping replica VMs metadata, which is located in the same repository?

I'm not sure what data exactly it comprises, but it's the directory that the replication job points to. I see directories for each VM there, and VHDX files. There's one replica that is failing, and for that one I note there are also a number of AVHDX files. Restore points perchance?

I think I may have spotted the problem though. I detached the old MBR drive from the file server, but I didn't delete it. I think Veeam is both backing up and replicating this detached drive as well, which is another 2TB, because I can see a copy of the VHD in the Replicas directory. I have taken a chance and deleted it, as the new GPT drive seems to be working correctly. I will run the file server backup job again, and if that works correctly I'll try the replication job.

[foggy]

I'm not sure what data exactly it comprises, but it's the directory that the replication job points to. I see directories for each VM there, and VHDX files. There's one replica that is failing, and for that one I note there are also a number of AVHDX files. Restore points perchance?

Yes, those are replica restore points.

I think I may have spotted the problem though. I detached the old MBR drive from the file server, but I didn't delete it. I think Veeam is both backing up and replicating this detached drive as well, which is another 2TB, because I can see a copy of the VHD in the Replicas directory. I have taken a chance and deleted it, as the new GPT drive seems to be working correctly. I will run the file server backup job again, and if that works correctly I'll try the replication job.

Deleting it from the source might not result in space release as the corresponding data blocks still reside in the backup file and will be marked as free according to deleted VMs retention setting. To effectively compact the full backup file compact operation should be performed, however, it requires additional space that you do not currently have.

[SSR]

The backup job doesn't appear to have compressed, despite being set to. The VKB file is currently 6TB, when the servers combined are 4.3TB.

The replication job isn't completing properly due to the lack of space, but I don't think it can complete until there's sufficient free space, at which point it will probably free up some space. So it's Catch 22.

Would it be wise to delete the file server backup and/or replication job, delete the relevant Backup and Replicas folder, and recreate them? I'm not sure I have any other choice, to free up this space.

[foggy]

The backup job doesn't appear to have compressed, despite being set to. The VKB file is currently 6TB, when the servers combined are 4.3TB.

This is not a compression issue, but due to the fact that data blocks are not actually cleaned up but just marked as free.

Would it be wise to delete the file server backup and/or replication job, delete the relevant Backup and Replicas folder, and recreate them? I'm not sure I have any other choice, to free up this space.

If it is acceptable to live some time without any backup at all, then you can do this. At least you can first do backup, then, once it succeeds, replication.

[SSR]

This is not a compression issue, but due to the fact that data blocks are not actually cleaned up but just marked as free.

With Veeam 9 though they seem to have added a maintenance task, which you can enable to "defragment and compact" full backup file. Would this not remove these cleared data blocks and reduce the backup file size?

If it is acceptable to live some time without any backup at all, then you can do this. At least you can first do backup, then, once it succeeds, replication.

Yes, I think I'll try this overnight. I'll delete the backup job first and recreate, and see if I can get some size reduction.

[foggy]

With Veeam 9 though they seem to have added a maintenance task, which you can enable to "defragment and compact" full backup file. Would this not remove these cleared data blocks and reduce the backup file size?

Right, this allows to release the occupied blocks in the full backup file that are not used and marked as free. However, this requires space for an additional full backup.

[SSR]

For this particular job I had the space, but as per above the VKB file didn't reduce. The recreated file server job is now 2.9TB, so half the size of the job before the trouble started.

I haven't recreated the replication job as it started running last night, and is still running now, 12 hours later. It has almost finished though, so will see if it completes properly and if space is reclaimed.

[foggy]

For this particular job I had the space, but as per above the VKB file didn't reduce.

Probably compact was not scheduled at this particular day. Anyway, now you're back to the clean VBK state and can increase retention back and proceed normally.

[SSR]

It was definitely scheduled. Will keep it enabled on Fridays, and keep an eye on space.

Replica job at 99%, 2.5TB free on the NAS... although unless it recovers the space space at the end there's something odd there also, as 5.5TB of VMs are taking up 8.5TB of space (restore points is still set to zero).

The two file server jobs within the replicas job have either had "failure to create change tracking time stamp for virtual disk" or "change tracking files are not specified". These show as warnings. Not sure how critical these are.

[foggy]

Replica job at 99%, 2.5TB free on the NAS... although unless it recovers the space space at the end there's something odd there also, as 5.5TB of VMs are taking up 8.5TB of space (restore points is still set to zero).

Retention is applied at the end of the job, so I'd let it finish and check the space again.

The two file server jobs within the replicas job have either had "failure to create change tracking time stamp for virtual disk" or "change tracking files are not specified". These show as warnings. Not sure how critical these are.

This means that CBT data is not available and the job needs to read the entire VM data to calculate differences.

[SSR]

It finished, and one of the file server replications failed (this has continually failed), and there was no space change even though retention was applied. So I've wiped the job, wiped the files, and recreated it. Will see tomorrow what the space difference is...

There doesn't seem to be a maintenance option for compressing a replica job, not that it seemed to work on the backup job anyway.

[foggy]

It finished, and one of the file server replications failed (this has continually failed), and there was no space change even though retention was applied.

Retention is applied at the end of the successful job run.

There doesn't seem to be a maintenance option for compressing a replica job, not that it seemed to work on the backup job anyway.

Replicas are stored in native format, so this is not applicable to them.

[SSR]

The recreated replication job ran satisfactorily.

So now I'm back to a Replicas directory of 4.5TB and a Backup directory of 3.3TB, although both will grow a bit, particularly the latter, due to restore points.

A bit of an extreme solution, but as everything's working I think it was the easiest given the circumstances.