Out of Space Recovery

Backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)

Out of Space Recovery

Veeam Logoby mkaec » Tue May 31, 2016 1:39 pm

I had a system backing up using VEB to a dedicated locally attached disk. As a test, I let it completely fill up the disk. As expected, backups started failing. I adjusted the retention settings and found that backups continued to fail. I'm guessing that VEB cleans up old backup files after a successful backup. When I looked at the files, it looked like it was running in reverse incremental with one full backup and a string of incremental backups. I deleted the oldest dated incremental and the next backup failed with an error that merge failed. I finally gave in and deleted all the backups.

I'm wondering if there is a better way to recover from a completely full disk when VEB backups are the only thing on the disk.
mkaec
Expert
 
Posts: 179
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Out of Space Recovery

Veeam Logoby JaxIsland7575 » Tue May 31, 2016 4:14 pm

Marc,
The VEB uses forever forward incremental process for backing up files. Deleting the last incremental will interrupt the chain and cause that failure. The straight forward option would be to add space to the destination to allow it to run successfully and remove points based on a new lower retention setting. Ideally you should setup alerts that will notify you of when space is getting low. As for shortening up a current backup chain, I do not know of a way that can be done without breaking the chain and having the next run create a full.

Cheers!
VMCE v9
JaxIsland7575
Veeam Vanguard
 
Posts: 364
Liked: 98 times
Joined: Mon Apr 27, 2015 1:59 pm
Location: NY, USA
Full Name: Ryan Jacksland

Re: Out of Space Recovery

Veeam Logoby mkaec » Tue May 31, 2016 6:40 pm

Ah, I was fooled by the last modified date on the full backup (which was today's date). Maybe if I had deleted a few of the most recent backups, this would have allowed the backup to complete and the new retention settings to cause a purge.
mkaec
Expert
 
Posts: 179
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Out of Space Recovery

Veeam Logoby Dima P. » Wed Jun 01, 2016 11:35 am

Hi guys,

I would not recommend deleting the latest incrementals – backup job will threat it as the backup chain corruption and start a new full backup. What you can do it to decrease the retention to start the backup merge and free some space.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Out of Space Recovery

Veeam Logoby mkaec » Wed Jun 01, 2016 12:57 pm

Dima,

That was the first thing I tried. I decreased the retention and ran a backup and the backup failed and space was not freed. My guess is it wanted to finish the backup before clearing out old backups, or it needed some free space to do merging and not enough was available.
mkaec
Expert
 
Posts: 179
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Out of Space Recovery

Veeam Logoby Dima P. » Wed Jun 01, 2016 6:08 pm

My bad, indeed, if there is not enough space on repository the backup job will fail as merge is performed at the end of backup job. Is it possible to move the backup chain to lager repository? If you have a VBR, you could use a backup copy job, then delete the existing chain and decrease the retention.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Out of Space Recovery

Veeam Logoby mkaec » Wed Jun 01, 2016 6:39 pm

This test probably represented the worst case scenario. It was a physical machine in a remote office with no other server to move the repository to. I suppose if I really wanted to save the backup chain, I could have sent an external drive down there and moved the backup chain to that. It wouldn't have been the speediest of fixes, though. We have the data shipped via DFS-R up to a central file server that is backed up by VBR. So, the local chain was lost, but we are still protected.

Thanks for weighing in on this. I just wanted to make sure that I didn't overlook something. The best practice is to have low disk space alerting set up, but this kind of thing can happen if the admins are not attentive to those alerts.
mkaec
Expert
 
Posts: 179
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Out of Space Recovery

Veeam Logoby Dima P. » Wed Jun 01, 2016 11:33 pm

Thanks for sharing the case. This will be a huge ‘plus one’ to an improvement request of making the merge operation at the begging of backup job :wink:
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Out of Space Recovery

Veeam Logoby mkaec » Wed Jun 22, 2016 5:51 pm

Perhaps it is also a +1 for my request to add a Min/Max retention option so that VEB could take automatic action when space is getting low. :)
mkaec
Expert
 
Posts: 179
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

[MERGED] Accurate free disk space check pre-merge

Veeam Logoby daphnis » Wed Jun 29, 2016 1:00 pm

I believe that VEB does check free disk space prior to running a backup, but I wish it would also look at available space if there will be a merge involved and perform a sanity check to make sure it can actually perform the transform. Use case for this is with a USB drive as a target, the backup might succeed because there is space available, but after that is consumed there isn't adequate space remaining to do the merge afterwards. I know VEB cannot get a good estimation of the next incremental restore point size, but it *does* know the size of the oldest incremental to be merged and so should factor that into its decision. I would recommend implementing this in the form of a pre-flight check before the next incremental restore point is captured and warn the user that sufficient disk space might not exist to complete the backup chain on that run.
daphnis
Enthusiast
 
Posts: 41
Liked: 3 times
Joined: Wed Jun 13, 2012 7:01 pm

Re: Out of Space Recovery

Veeam Logoby Dima P. » Wed Jun 29, 2016 5:20 pm

Hi daphnis,

Thanks. This is a valid feature request and we do have plans to implement it, however I can’t provide any ETA at the moment.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov


Return to Veeam Agent for Windows



Who is online

Users browsing this forum: No registered users and 8 guests

cron