Feature Request - Differential backups

[mlabbe]

Hi,

I'm new to Veeam. Heard the name several times in the past years, and we've been impressed by the Veeam demo at SIP 2015 in Quebec a couple weeks ago.
We started testing VEB on a couple desktops after the show. It's a great product that we'd be very interested to use, but we kinda hate using incremental backups.

Any chance you add support for differential backups in a future version?

There are alot of desktops we support that don't have alot of file changes every day so the file size/bandwidth difference between differentials and incrementals isn't that much. However, differentials allow cleanups more easily e.g. in cases like this:
http://forums.veeam.com/veeam-endpoint- ... 30584.html

...and differential backups don't make you lose a whole bunch of restore points when a single of them becomes lost or unusable and breaks the chain (which is the main reason we always avoided using differentials).

[Dima P.]

Hello Michel,

We don’t plan to add differentials to the Endpoint for now, as we don't have support for them in our backup engine. However, that’s for sharing your use case – it highly appreciated and will be taken to internal discussion. Cheers!

[Edmondo]

Hi,

just wanted to chime in and suggest that the differential option would be VERY useful because the incremental backup isnt very secure, it just needs one file to be corrupted and the you lose a lot of days of backup.

[Dima P.]

Hi Edmondo,

Can you please clarify what you are going to backup with differential backup (I bet it’s a physical server and not a desktop computer)?

[mlabbe]

Oh well, I've been testing VEB for a couple months now, and what I feared may happen just did happen to me a couple weeks ago. I didn't have time to write to the forum earlier.

I noticed VEB backups were failing for the last couple days. Error was :

"Full backup file merge failed Error: CRC Error code: 23 Failed to read data from the file [\\server\share$\folder\Backup Job mlabbe\Backup Job mlabbe2015-12-29T003418.vib]."

It first looked like disk corruption on the server I'm backing to, but it's a RAID volume and the RAID manager showed no error. Scandisk didn't return anything wrong with the volume either.

It turned out the disk errors in the server's eventlog weren't from the RAID volume, but from an external USB drive that was dying. I guess the flood of disk errors on the USB drive may have caused some timeouts on the server. Somehow, VEB ended up corrupting the first vib file in the chain while merging it to the vbk, so all incremental backup files are unusable and I lost a whole month of retention.

While it wasn't a big issue to me (I renamed the backup folder and started over, it's been running fine since), but IMHO this proves a major flaw in VEB. No matter the reason, the merge thing ended up corrupting a backup file that was perfectly fine before, and because everything is incremental, it broke the whole backup chain.

I still like the product, I've used it a couple times to restore files/volumes and it worked great, but I can't say I'd sleep well if I had important data to backup on a daily basis and it was my only backup solution.

You should really reconsider adding support for differential backups to the backup engine, that would prevent the issue I got from happening: no need to merge and modify files (risking corrupting them in the process), just delete the older differentials and keep creating new ones. All valid backup files in the chain would remain untouched.

[Dima P.]

Hello and sorry for keeping silence Michel,

"Full backup file merge failed Error: CRC Error code: 23 Failed to read data from the file [\\server\share$\folder\Backup Job mlabbe\Backup Job mlabbe2015-12-29T003418.vib]."

This error indeed may refer to the destination disk corruption but if I were you I’d trust the check disk output (moreover if it’s a server). We got several cases in QA department when check disk did not properly report disk corruption. Try any third party tool to be sure.

I can't say I'd sleep well if I had important data to backup on a daily basis and it was my only backup solution.

I think hardware failure is still one of the main reasons why backup should be used. We have pretty decent blog post - How to follow the 3-2-1 backup rule with Veeam Backup & Replication. The main idea of this post and one of best backup practices is at least three copies of your data, two different media, keep one backup copy offsite (and offline). For instance, if you use Veeam Backup and Replication repository you can use a backup copy job or backup to tape to save VEB backups to another location or offline.

the merge thing ended up corrupting a backup file that was perfectly fine before

Strange because there is a built-in health check – which basically confirms that existing chain is restorable before creating a new incremental backup or performing merge.

While it wasn't a big issue to me (I renamed the backup folder and started over, it's been running fine since)

Thanks for sharing your case and glad to hear that no sensitive data was lost. A bit sad that you did not open a support case since then we could analyze the VEB logs and get better understanding on what actually has happened.

As for differential backup, this is counted as a feature request and I’ll take your case to the weekly team meeting. Thanks again.

[mlabbe]

We got several cases in QA department when check disk did not properly report disk corruption. Try any third party tool to be sure.

Do you have any third party to recommend? I'm not sure I'd blindly try any unknown third party for a data consistency check on a 12 TB RAID volume. I googled a bit and didn't find any tool's name that is mentioned repeatedly as being more reliable than chkdsk or RAID manager softwares.

We have pretty decent blog post - How to follow the 3-2-1 backup rule with Veeam Backup & Replication.

Great blog post indeed. I would add "also keep an offline backup" to that blog post though. Everything is connected these days and you need to think more than just hardware failure or physical disaster. No matter how many different copies/medias/sites your data is backed up to, you still could be screwed if a hacker gains access to your servers and he is able to nuke all your local and offsite backups. While your blog post does mention "storage location" and you mentionned offline backup in your last post, the blog doesn't make it clear that keeping an offline backup is also important.

Strange because there is a built-in health check – which basically confirms that existing chain is restorable before creating a new incremental backup or performing merge.

Well, yes that's my point: I believe the built-in health check did find the existing chain was restorable before performing merge, but it didn't check after the merge completed (it looks like some problem happened during the files merge and VEB missed it). The built-in health check it also found it was not restorable on the next backup schedule so it could not add new incremental backups to it.

A bit sad that you did not open a support case since then we could analyze the VEB logs and get better understanding on what actually has happened.

I still have the corrupt backup chain files and I'm pretty sure I still also have these VEB logs (I didn't deleted them so unless VEB deleted them when I started a new backup chain over I should still have them). I'll try to open a support case tomorrow so you can investigate.

[mlabbe]

For the records, I opened support case #01708701 for logs investigation.

Btw, I also had an "issue" with the email registration process (required before you can submit a support request): from the "Report an issue" window, I clicked the "register" button next to my email and received the verification email. But when I click on the URL inside the verification email, the website says "Unfortunately we didn't receive confirmation of your email. Please try again." on every try. I ended up clicking the "update" button in the report window and my email turned green so I could submit my case, but the message returned by the website is confusing.

[Dima P.]

Michel,
First of all thanks for opening the case – now I can share it with QA team and we can investigate your problem.

health check did find the existing chain was restorable before performing merge, but it didn't check after the merge completed

True. That’s why we do plan to add the additional health checks in upcoming major version.

Do you have any third party to recommend?

I tested Victoria disk checker. It has a really ugly UI, but no doubts it performs more complex hard drive analysis. Let me check with the team if it can handle a RAID like yours - I’ll follow up tomorrow.

the blog doesn't make it clear that keeping an offline backup is also important

That’s why I always keep adding ‘offline’ when referring to this post

the message returned by the website is confusing

Could be a temporary glitch – I’ve tested the case submission with new email address and it went thru. Please let me know if this issue appear again.

[Dima P.]

Michel,

We can’t find the logs you are referring to in the case. Can you just grab and share the C:\ProgramData\Veeam folder from the affected machine (Drobox/OneDrive or any other cloud hosting).

In addition, QA team wanted to check if you could start a File Level Recovery from the restore point in the backup chain that seems to be corrupted?

[mlabbe]

We can’t find the logs you are referring to in the case.

I sent the logs to Adam via ftp a couple hours ago (you probably already know at this time).

QA team wanted to check if you could start a File Level Recovery from the restore point in the backup chain that seems to be corrupted?

The merge issue happened on 2016-02-02 (merging the oldest vib dated 2015-12-29 with the vbk dated 2015-12-28).
I was able to restore a file from the 2016-01-29 vib with success.
So it's not as bad as I thought: I couldn't add new backups to this broken chain, but data is still recoverable so that's great news on the reliability side

I was also able to successfully restore a file from the 2015-12-29 backup (the corrupt vib file).
I don't see the full backup (vbk, dated 2015-12-28) in my list of restore points though.

Let me know if you need anything else.

[Dima P.]

Michel,

Awesome news – thanks for sharing. We’ve identified the issue you faced - it was caused by local SQL misbehaving - it seems that it was temporary inaccessible during merge, and that caused the merge issues. Frankly speaking only, the metadata file was damaged and that’s why recovery is still possible.

We will add additional logic to handle such issues in the upcoming update – so thanks again for all your help in investigation!

[NickViz1]

Hello Dima,

I'm also voting for a differential backup feature - for the safety reason: I don't know exactly how do you merge the incremental backup data to the main "trunk", but if you do it in place - the data could be corrupted with the system crash/power loss/reset. If you copy and merge data to another "trunk" file and then delete the old one - then you copy forth and back the whole main "trunk" file, which is a waste of resources.
Also, it's not very convenient that the big trunk file is changed every day - if I use a USB media to copy data off-site - I need to copy not only the changed data, but the big "trunk" file as well, as it changes daily.

The differential backup fixes at least 3 issues:
1. It keeps main file untouched, thus reducing load/possibility to damage whole backup,
2. It's probably faster to restore data (only 2 files are in use),
3. It's much easier to copy backup to offline storage.

If I see that differential backup takes too much space (as data changes are too big comparing to the main file), I can run the full backup again (renaming folder with my previous backup). And again - if something failed during this backup - I'm always have the decent previous one.
Indeed it takes more space, but with modern disks it's acceptable.

With best regards,
Nikolai

[mlabbe]

We’ve identified the issue you faced [...] We will add additional logic to handle such issues in the upcoming update – so thanks again for all your help in investigation!

Glad I could help! Thanks for the explanations.