First vib file got corrupt during merge, lost all increments

Backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)

First vib file got corrupt during merge, lost all increments

Veeam Logoby mlabbe » Fri Feb 19, 2016 11:16 pm

I posted this in another thread a couple days ago but since I got no feedback at all and I saw alot of activity from Veeam support team in other threads, I thought I'd post again in a new thread to make sure to catch more attention.

As mentionned in the other post, VEB somehow managed to corrupt my first vib file in the chain during the process of merging it into the vbk file. Since everything is incremental, this rendered all my other vib files completely useless (I lost a whole month of retention) and cheduled backups stopped working at this point.

I understand adding differential backups may not be an option because of the way VEB is designed, but then I think some better protection mechanism should to be added to make sure this issue doesn't happen again (e.g. add some PAR2 files or something before doing the merge so you can repair/rollback if there's an issue during the merge process).

I didn't happen to need anything from my lost backup files, but a backup software is as good as what it can restore when you need it.

I also suggest you add some more eye-catching notification (e.g. somewhat like win10 notification window every couple hours) to catch more attention when the backups are failing day after day. I've been sitting at my computer for at least a couple hours before I noticed the tiny red "x" (instead of usual clock) on VEB's tray icon. I know ALOT of non-IT people who wouldn't have noticed anything at all until it's too late.
mlabbe
Influencer
 
Posts: 20
Liked: 4 times
Joined: Fri Oct 16, 2015 4:14 pm
Location: Quebec, Canada
Full Name: Michel Labbé

Re: First vib file got corrupt during merge, lost all increm

Veeam Logoby Dima P. » Sat Feb 20, 2016 12:42 am

Michel,
Thanks again for sharing your thoughts!

I've been sitting at my computer for at least a couple hours before I noticed the tiny red "x" (instead of usual clock) on VEB's tray icon.

How do you think, should it be a pop up notification or may be email notifications are enough for backup history tracking, or maybe both?
Dima P.
Veeam Software
 
Posts: 6249
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: First vib file got corrupt during merge, lost all increm

Veeam Logoby mlabbe » Wed Feb 24, 2016 4:41 am

How do you think, should it be a pop up notification or may be email notifications are enough for backup history tracking, or maybe both?

Well, I'd say both would be best, but if you don't want to implement email notifications in the free version, a pop up notifications should do it.

You don't want to implement a 5-10 secs notification that the user will miss because he was afk when the pop up happened, and not something that will be instantly dismissed because it happened to pop while the user was typing on keyboard either.
mlabbe
Influencer
 
Posts: 20
Liked: 4 times
Joined: Fri Oct 16, 2015 4:14 pm
Location: Quebec, Canada
Full Name: Michel Labbé

Re: First vib file got corrupt during merge, lost all increm

Veeam Logoby mlabbe » Tue Mar 01, 2016 11:15 pm 1 person likes this post

For whoever may end up on this thread later don't freak out yet :), the issue I faced prevented new incremental backups, but most important: the data in the corrupt backup chain was still recoverable. The follow-up has been done in the original thread.

mlabbe wrote:(e.g. add some PAR2 files or something before doing the merge so you can repair/rollback if there's an issue during the merge process).

I sent some more details about these PAR2 files to Adam in the support ticket and he said it might also help that I post this on the forums so here it is for the records:

Have a look at Parchive if you haven't checked this already. I used PAR2 files often when I was a student and was grabbing files from newsgroups. My ISP's newsgroup servers were skipping posts every now and then so more than half of the binary files were incomplete or corrupt. It's very powerful: a small PAR2 file (even if it's damaged itself) could repair a dozen corrupt files, as long as you have enough healthy parity blocks to cover the number of corrupt blocks in a number of files.

See example images:
Image Image.

Parchive now seems to implement PAR3 with better support for multithreading.

You could use something like this (it's open source) to create some parity blocks before starting the merge (and maybe even add another set after completing for extra safety). Then whatever happen (even if there are real bad blocks on the disk), you'd be able to repair/rollback the vbk/vib files as long as there's enough healthy parity blocks available.
mlabbe
Influencer
 
Posts: 20
Liked: 4 times
Joined: Fri Oct 16, 2015 4:14 pm
Location: Quebec, Canada
Full Name: Michel Labbé

Re: First vib file got corrupt during merge, lost all increm

Veeam Logoby Dima P. » Wed Mar 02, 2016 2:56 pm

Michel,

Once again thanks for sharing! You issue report helped us a lot and we will make the needed corrections in upcoming version.
Dima P.
Veeam Software
 
Posts: 6249
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: First vib file got corrupt during merge, lost all increm

Veeam Logoby ftcnet » Thu Mar 03, 2016 5:25 pm

Yes, this area of VEB is still a bit weak (IMO). I lost a client due to a missing .vib file. The .vem and all .vbk files were still present. Veeam support worked on it for several hours, but apparently there's no way to recreate a missing or corrupted .vim file. It seems like using .PAR files might substantially improve our confidence in VEB's ability to successfully restore when needed.

Email notification at completion of backup jobs would be very much appreciated.
ftcnet
Enthusiast
 
Posts: 25
Liked: 3 times
Joined: Fri May 15, 2015 1:59 am
Full Name: Bernard Klatt

Re: First vib file got corrupt during merge, lost all increm

Veeam Logoby Dima P. » Fri Mar 04, 2016 11:10 am

Hello Bernard,

Can you share the case ID please? Thanks.
Dima P.
Veeam Software
 
Posts: 6249
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov


Return to Veeam Agent for Windows



Who is online

Users browsing this forum: UT2015 and 11 guests