Out of Space Recovery

[mkaec]

I had a system backing up using VEB to a dedicated locally attached disk. As a test, I let it completely fill up the disk. As expected, backups started failing. I adjusted the retention settings and found that backups continued to fail. I'm guessing that VEB cleans up old backup files after a successful backup. When I looked at the files, it looked like it was running in reverse incremental with one full backup and a string of incremental backups. I deleted the oldest dated incremental and the next backup failed with an error that merge failed. I finally gave in and deleted all the backups.

I'm wondering if there is a better way to recover from a completely full disk when VEB backups are the only thing on the disk.

[JaxIsland7575]

Marc,
The VEB uses forever forward incremental process for backing up files. Deleting the last incremental will interrupt the chain and cause that failure. The straight forward option would be to add space to the destination to allow it to run successfully and remove points based on a new lower retention setting. Ideally you should setup alerts that will notify you of when space is getting low. As for shortening up a current backup chain, I do not know of a way that can be done without breaking the chain and having the next run create a full.

Cheers!

[mkaec]

Ah, I was fooled by the last modified date on the full backup (which was today's date). Maybe if I had deleted a few of the most recent backups, this would have allowed the backup to complete and the new retention settings to cause a purge.

[Dima P.]

Hi guys,

I would not recommend deleting the latest incrementals – backup job will threat it as the backup chain corruption and start a new full backup. What you can do it to decrease the retention to start the backup merge and free some space.

[mkaec]

Dima,

That was the first thing I tried. I decreased the retention and ran a backup and the backup failed and space was not freed. My guess is it wanted to finish the backup before clearing out old backups, or it needed some free space to do merging and not enough was available.

[Dima P.]

My bad, indeed, if there is not enough space on repository the backup job will fail as merge is performed at the end of backup job. Is it possible to move the backup chain to lager repository? If you have a VBR, you could use a backup copy job, then delete the existing chain and decrease the retention.

[mkaec]

This test probably represented the worst case scenario. It was a physical machine in a remote office with no other server to move the repository to. I suppose if I really wanted to save the backup chain, I could have sent an external drive down there and moved the backup chain to that. It wouldn't have been the speediest of fixes, though. We have the data shipped via DFS-R up to a central file server that is backed up by VBR. So, the local chain was lost, but we are still protected.

Thanks for weighing in on this. I just wanted to make sure that I didn't overlook something. The best practice is to have low disk space alerting set up, but this kind of thing can happen if the admins are not attentive to those alerts.

[Dima P.]

Thanks for sharing the case. This will be a huge ‘plus one’ to an improvement request of making the merge operation at the begging of backup job

[mkaec]

Perhaps it is also a +1 for my request to add a Min/Max retention option so that VEB could take automatic action when space is getting low.

[daphnis]

I believe that VEB does check free disk space prior to running a backup, but I wish it would also look at available space if there will be a merge involved and perform a sanity check to make sure it can actually perform the transform. Use case for this is with a USB drive as a target, the backup might succeed because there is space available, but after that is consumed there isn't adequate space remaining to do the merge afterwards. I know VEB cannot get a good estimation of the next incremental restore point size, but it *does* know the size of the oldest incremental to be merged and so should factor that into its decision. I would recommend implementing this in the form of a pre-flight check before the next incremental restore point is captured and warn the user that sufficient disk space might not exist to complete the backup chain on that run.

[Dima P.]

Hi daphnis,

Thanks. This is a valid feature request and we do have plans to implement it, however I can’t provide any ETA at the moment.

[marcus@vision1.org]

Hello,

I'm using VAW 2.0.0.700 (Free edition), backing up to external USB HDD. The retention policy is set to 23 days. I have 25 backups on the external drive. My expectation is that VAW will merge an incremental to the master backup and retain only 23 backup days, and I don't see that happening. Help?

The sequence that led to where I am:
1. backup target got full. Lots of failed backups.
2. I manually decreased retention to 23 days. no happiness.
3. Manually deleted three large (tens of GB each) .vib files.
4. Re-ran backup. many times. It's never been happy since #1 above. It keeps complaining about " Error: Shared memory connection was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: There is not enough space on the disk. Failed to write data to the file [Q:\VeeamBackup\Backup Job DESKTOP-V7MLR87\Backup Job DESKTOP-V7MLR872018-02-24T100751.vbk]. Failed to download disk.

Help?

Thanks.
-marcus

[chaycock]

It would be nice if Veeam included a utility that would let you remove some of the increments to help free up some space. It is not always practical to "move it to a larger disk". You certainly do not want your backups failing while you are trying to figure out what to do or get a larger disk.