Comprehensive data protection for all workloads
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Support for differential backups

Post by Gerben »

Hi,

We would love to see the support for differential backups.
Because we don't want to use incremental backups and build a large chain of dependencies.
If one incremental backup get corrupt or missing, then we can't use that backup to restore our data.

I know that we can create multiple full backups, but that is not efficient.
I also know that we can create reverse incremental backups, but that also creates a chain that we don't want.

So, please add the support for differential backups.
If you also think that differential backup is essential, reply to this thread to make this thread stand out.

Case # 01807321
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Support for differential backups

Post by veremin »

Just to check whether we're in the same boat here - what do you understand by "differential" backup? Thanks.
Pat490
Expert
Posts: 170
Liked: 29 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: Support for differential backups

Post by Pat490 »

maybe you should take a look at "reverse incremental"feature
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

v.Eremin wrote:Just to check whether we're in the same boat here - what do you understand by "differential" backup? Thanks.
Thank you for your reply.

We want to create a full backup on Friday and differential backups from Saturday trough Thursday.
At Friday our backup disks (single disks, no RAID) are rotated with disks from our fire safe.

Because there is no redundancy in those single disks, we want to minimize the dependencies for the restore procedure.
If we need to restore we want to use the full backup from Friday and only one differential backup.
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Pat490 wrote:maybe you should take a look at "reverse incremental"feature
I am familiar with the the reverse incremental feature.
But this also creates a chain of dependencies, but then in reverse order.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Support for differential backups

Post by veremin »

We want to create a full backup on Friday and differential backups from Saturday trough Thursday. At Friday our backup disks (single disks, no RAID) are rotated with disks from our fire safe.
Create a backup repository, enable "rotated drive" option in its settings. Then, add a backup job, set a daily schedule for it and make it create an active full backup on Friday.

This is pretty much it. The described approach should meet your expectations.

Thanks.
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

v.Eremin wrote:Create a backup repository, enable "rotated drive" option in its settings. Then, add a backup job, set a daily schedule for it and make it create an active full backup on Friday.
Yes, I'm also familiar with the "rotated drive" option.
But this does not allow us to create differential backup.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Support for differential backups

Post by veremin »

By differential you mean a point containing changes occurred since the latest full backup, right? And idea here is to avoid building a large chain of dependencies?

I'm wondering because with weekly full backup there won't be "a large chain of dependencies". Also, you can perform regular health check to avoid data corruption.

Thanks.
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Yes, we want to be able to select a differential backup which is only depended on the last Full backup.
Yes, we want to avoid any chain longer than two dependencies for the restore procedure.

Yes, I am also familiar with the "health check", and I have enabled this to run this health check each day.
But this does not reduce the the dependencies for the restore procedure.
Anders
Enthusiast
Posts: 58
Liked: 12 times
Joined: Sep 09, 2010 9:45 am
Full Name: Anders Lorensen
Contact:

Re: Support for differential backups

Post by Anders »

So the drive in your fire safe should contrain:
1 full backup
6 differential backups

instead of:
1 full backup
6 incremental backups

Thats how I understand what you want. If that is correct, you will gain nothing! You will still be depending on the same amount of physical Blocks on the physical disk. And you will need to write more data to the disk.

/Anders
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Support for differential backups

Post by dellock6 »

can i ask why the need to have at most 2 files to start any restore is so important?
As you store data in single disk media, the chances to lose data over those disks are depending on the resiliency of those disks, bot the type of backup method you use. If the disk breaks,it loses the entire week you've stored during the rotation, even if you would stor full backups daily on it...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Anders wrote:So the drive in your fire safe should contrain:
1 full backup
6 differential backups

instead of:
1 full backup
6 incremental backups

Thats how I understand what you want. If that is correct, you will gain nothing! You will still be depending on the same amount of physical Blocks on the physical disk. And you will need to write more data to the disk.

/Anders
Yes, that is how we want it.

I understand what you mean. And yes, if you want to restore 100GB of data, you have to at least read 100GB of data (if not compressed/deduped/etc).
But, to restore the data there several factors that play a role, and the physical sectors are one of them.

1) If you have a chain of incremental backups, it has to read the (meta)data of those incremental backups to reconstruct the original data. That way it has to read more data than only full backup + incremental backup. And as we agree, reading more data = more prone to error.

2) If you have filesystem corruption, for example with NTFS, then it might be possible that a incremental backup is not visible on the drive. This means that you can't restore from the incremental backup that is not visible/corrupt and from the incremental backups after that. If you have differential backup, you can ignore that one differential backup and choose a different restore point. If you average that out, then you cannot restore from 50% of the restore points if a incremental backup file is corrupt. If you have differential backup than that is in our example only one of the six differential backups, and that is 17% (1/6*100) of the restore points.

3) If there is an error while creating the incremental backup, that goes undetected, which will cause the incremental backup to be corrupt, then again you can't use the that incremental backup and the backups after that. Such error might occur with a unpatched bug in Veeam, Hardware error, I/O error, etc. As stated before a "Health check" might find this error, and might fix/correct this error. But it is possible that the "Health check" is not capable of detecting this error. Once again, you can't use the backups from this point on. Unless there is a new full backup.

4) If you need to create a copy of the backup before you can use it to restore a VM, then you would need to copy the whole chain if you are using incremental backups. This takes more time then it would to copy the full backup and the differential that you want before you can start the restore.

5) If you use incremental backups, the whole chain needs to be present. That means that you need to maintain the whole chain. Of course that would not be a problem for a specialized IT-person, but each step that makes the whole process more complex, makes it more prone to user errors. Someone might delete a incremental file because they think that it is a old file and not needed anymore. In our example a disk contains only a backup for one week. But again, each bit of complexity makes it more prone to errors.

I am really convinced that incremental backups are more prone to errors then differential backups.
To be fair, it would be nice if Veeam supports parity with for example PAR2, and that you can set the parity to for example 20%. But that still would not solve all the issues stated above.

Sorry for the delayed response.
I had to type this message again, because the Veeam forum decided to ask for my user credentials before I can post my reply, and this caused that my message was lost.
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

dellock6 wrote:can i ask why the need to have at most 2 files to start any restore is so important?
As you store data in single disk media, the chances to lose data over those disks are depending on the resiliency of those disks, bot the type of backup method you use. If the disk breaks,it loses the entire week you've stored during the rotation, even if you would stor full backups daily on it...
Yes, I agree. Disk resiliency is important, but there are more factors.
I've just retyped my message, see my reply above.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Support for differential backups

Post by dellock6 »

Just a comment on performance: we have tested many times in the past incremental chain performance, and up to hundreds of restore points, there is no read penalty restoring from a large chain compared to a small chain.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

dellock6 wrote:Just a comment on performance: we have tested many times in the past incremental chain performance, and up to hundreds of restore points, there is no read penalty restoring from a large chain compared to a small chain.
The restore performance is not that important. It would be a nice benefit, but not a must. In the end the data matters for us. I know that that is contradictory with point number 4 which I stated, but again, it is a benefit, which only costs a bit more disk space.

The issue here is the chances that an error might occur, and that you can't use the backup that you would like to restore from.
Because we create backups, to restore from, right?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Support for differential backups

Post by dellock6 »

It's also contradictory with point 2: corruptions and bit rot happen in percentage to the amount of written data. The more date, the more chances to have a corruption. Guess which backup method writes more data to the media? Yes, the differential.

To be honest, I remember differential backups in the days of physical backups, offered by other companies at that time. I never questioned the reason to have such a method, but the simple fact that no-one today is offering it would raise a flag for me. And the simplest reason is that, with the amount of data we are dealing these days in production environment, no-one can afford to read multiple days the same data from a production storage: too many I/O consumed on the production storage, too long backup window, too many data to be stored in the backup storage.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

dellock6 wrote:It's also contradictory with point 2: corruptions and bit rot happen in percentage to the amount of written data. The more date, the more chances to have a corruption.
I agree, the more data you have stored, the higher the chance that some data is corrupt.
dellock6 wrote: Guess which backup method writes more data to the media? Yes, the differential.
Yes, that is true. Differential backups writes more data to the disk. This will cause that there is redundant data on the disk. The advantage of this that you can choose which part of the redundant data you want to restore from. And do you know why that is? Because corruption and bit rot can occur.
dellock6 wrote: To be honest, I remember differential backups in the days of physical backups, offered by other companies at that time.
Yes, other companies do currently support differential backups. Acronis for example. Or Backup Exec from Symantec/Veritas. I'm not familiar with other backup solutions, so maybe they are the only two, or maybe there are more, but that if offtopic.
Acronis: https://kb.acronis.com/content/1536
Backup Exec: https://www.veritas.com/support/en_US/article.000075525
dellock6 wrote: I never questioned the reason to have such a method
Feel free to question it now. :)
dellock6 wrote:I never questioned the reason to have such a method, but the simple fact that no-one today is offering it would raise a flag for me.
No-one? A stated above, I know about Acronis and Backup Exec, and they support differential backup. I'm not familiar with other backup solutions, and I guess that I don't have discuss this in this topic?
dellock6 wrote: And the simplest reason is that, with the amount of data we are dealing these days in production environment, no-one can afford to read multiple days the same data from a production storage
Again no-one? We can afford it. I guess that if you don't have the resources (money, time, people, hardware, etc) to create full/differential backups, how are you going to do it when you have to restore it?
dellock6 wrote: too many I/O consumed on the production storage
I hope that you don't live on the edge with the I/O resources, and that you have resources available to create a decent backup. But again, that is a matter of resources. And if you don't have that, you need to bring that to the attention of the respective manager. But that is off-topic, because we do have the resources available.
dellock6 wrote: too long backup window
We can run the backups through the night and in the weekend, and that is for us enough time. I don't know about other companies, but I guess that if you have enough resources you can run the backup during work hours. If you don't have enough resources, then yes, then you might encounter a drop in performance. But the Veeam software also has some configuration options to prevent that from happening. Once again, this is offtopic.
dellock6 wrote: too many data to be stored in the backup storage.
Differential backup requires more storage, yes that is true. But again, if you have the resources, then it is not a problem.

To conclude, please give the end-users the possiblility to set the backup schedule to their needs.
No-one can decide that you don't have the resources available to create the backup, other than the people who manage it.
Differential backup is less efficient, but it has advantages. (I'm not going to tell the advantages again, as they are already stated before in this thread)

May I quote from Acronis? Source: https://kb.acronis.com/content/1536
Acronis wrote: So, If your objective is to reduce backup size and backup time, then the best strategy would be to create incremental backups. On the other hand, if you want to increase backup reliability, by not having to rely on a chain of incremental backups, then differential backups would be the best solution.
I completly agree with Acronis, but once again, it depends on the needs for the end-users.
We obviously want backup reliability, and to achieve that we want to use differential backups.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Support for differential backups

Post by Gostev »

Hi, Gerben. I understand your points, but just to set expectations straight, we are not planning on adding differential backups option to our solution. This is based on the amount of requests for this functionality for the past 8+ years, all the success our immense customer base has had with incremental backups, as well as the fact that forever-incremental backup is considered an industry best practice for enterprise backup these days - including by leading analysts such as Gartner (legacy vendors will certainly tell you otherwise).

We do recognize that incremental backup approach has certain disadvantages (everything has its pros and cons), which is why our product has some unique functionality such as automated backup health check and self healing.

That said, the best way to make sure your data is safe is to maintain at least one copy of your backups as per 3-2-1 rule. This will protect you from virtually all possible issues with your backups. Because honestly, according to our support statistics, the data corruption issues you are raising above happen 10 times more rarely than those which you can protect against only by having a complete copy of your backups on a separate media.

Thanks!
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Thank you for your reply.

We both agree "that incremental backup approach has certain disadvantages (everything has its pros and cons)."
But why not offer the option for the end-user?

I know that the Veeam software won't change at the request for one user.
But I can also imagine that it would not take that much effort to create that option, but I can't say that because I obviously didn't create the software.

I understand the choice, and I respect that.
Still I have to figure a way to use Veeam with differential backup, search for a different backup solution, or convince myself and the IT-department that incremental backup is the way to go.
Because of that, do you have a link to the referenced Gartner article, stating that "forever-incremental backup is considered these days an industry best practice for enterprise backup"?

No offense, but I would be really interested in that from a IT-technical-perspective. :wink:
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Support for differential backups

Post by Delo123 » 1 person likes this post

i can't imagine how differential backups would work with image-level backups...
Doing file based backups i see the point, but with this one is always dependent on the first full. Doing full or at least synthetic fulls daily would be the only real alternative i guess...
Dennis_Ayala
Novice
Posts: 3
Liked: never
Joined: Jul 22, 2012 10:50 pm
Full Name: Dennis Ayala
Location: San Juan, PR
Contact:

Re: Support for differential backups

Post by Dennis_Ayala »

dellock6 wrote:can i ask why the need to have at most 2 files to start any restore is so important?
As you store data in single disk media, the chances to lose data over those disks are depending on the resiliency of those disks, bot the type of backup method you use. If the disk breaks,it loses the entire week you've stored during the rotation, even if you would stor full backups daily on it...
You're right. If rotating disks or media for that matter, full backups and differentials should be stored on different disks and rotated differently too.

For example:
Full Backup Disks - Rotated Weekly and a 1 month rotation should contain 4 different disks.
Differentials Disks - Rotated daily and a weekly rotation should be 6 different disks.

Although the strategy is valid, it would be expensive and includes the proper management/identification of each disk so you don't use the wrong one.

I think that he should stay using one week rotation of 7 disks and do a FULL every day. In this case, if he loses any of the disks, he'll always have yesterday's full backup.

D.
Dennis_Ayala
Novice
Posts: 3
Liked: never
Joined: Jul 22, 2012 10:50 pm
Full Name: Dennis Ayala
Location: San Juan, PR
Contact:

Re: Support for differential backups

Post by Dennis_Ayala »

v.Eremin wrote:Just to check whether we're in the same boat here - what do you understand by "differential" backup? Thanks.
This is how I understand the difference of Incremental and Differentials:

Incremental backups - the changes since the last full backup or incremental. During incremental backups, the amount of data being backed up is smaller. To achieve a full restore to the most recent point in time available, the restore needs the last full plus ALL incrementals.

Differential backups - the cumulative of previous differentials since the last FULL backup plus the changes on the day of the backup. As the description states, its cumulative so each day the differential backup runs, the amount of data is larger backed up is larger. To achieve a full restore to the most recent point in time available, the restore needs the last full plus the latest differential.

Regards,

D.
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

I think that the main factor here is the chance for data corruption (bitrot)
I think that if we could reduce or eliminate the chance for data corruption, incremental backups would be accepteble.
The assumption here is that no incremental files get deleted by accident (user-error), or that Veeam creates a undetectable corrupt incremental backup file (unfixed bug)

Does anyone have experience or ideas how bitrot can be prevented for rotary disks?
I think that that is a diffecult question to answer.

I do think that if bitrot occurs, that a restore from differential backups have a greater chance than incremental backup.
Because of that we prefer to use differential backups.

If anyone knows about the Gartner article, stating that "forever-incremental backup is considered these days an industry best practice for enterprise backup", I would love to hear about that.
I guess that they would also talk about bitrot.
The only thing that I could find with Google, after looking on the Garner site, is this PDF file with a presentation.
https://cdn2.hubspot.net/hub/30865/file ... 9)_(1).pdf
On page 5 it states that "Too many incremental backups" is a reason that backups can't be used for a recovery.
That PDF is from december 2012, so this statement might be outdated.
If anyone has some more recent information, I would love to hear it.

Thanks for all the replies so far. :)
smoore33
Influencer
Posts: 11
Liked: 4 times
Joined: Mar 14, 2016 7:31 pm
Full Name: Steve
Contact:

Re: Support for differential backups

Post by smoore33 »

I share the concern about a long backup chain to the last full. Even thought I have not been using Veeam that long, I have run into one situation where a backup, that had not generated an error message on creation, would not restore. My solution has been to do more active full backups (daily) and only use incrementals during the day.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Support for differential backups

Post by HannesK »

ReFS has features against bit rot / data corruption. Would it be an option for you to use that instead of NTFS?

And to check whether backups are corrupt you have two ways with Veeam
- Data Corruption guard: he checks whether data corruption happened and can repair it depending how much data was lost. It is enabled for backup copy jobs per default and you can enable it also in backup jobs (storage -> advanced -> maintenance)
- automatic verification / restore of backups with SureBackup: this ensures that the VM is really bootable as it spins it up and really tests whether the operating system boots and you can configure to test even services
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Thanks for the suggestions
My experiences with ReFS and Storage Spaces is not that great (Slow performance, not that great monitoring), and it's quite new filesystem, so I guess that it still has some mayor bugs in it. I know that ReFS requires Storage Spaces to tolerate bitrot, and I have not read anything about the ability to hotswap Storage Spaces pools.

Yes, I do have Data Corruption guard (health check) enabled to be run each day.
I was not familiar with SureBackup. Interesting feature. I will take a look at it.
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Is there anyone who can comment on this?
If anyone knows about the Gartner article, stating that "forever-incremental backup is considered these days an industry best practice for enterprise backup", I would love to hear about that.
I guess that they would also talk about bitrot.
The only thing that I could find with Google, after looking on the Garner site, is this PDF file with a presentation.
https://cdn2.hubspot.net/hub/30865/file ... ckup_(6-09)_(1).pdf
On page 5 it states that "Too many incremental backups" is a reason that backups can't be used for a recovery.
That PDF is from december 2012, so this statement might be outdated.
If anyone has some more recent information, I would love to hear it.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Support for differential backups

Post by HannesK »

yes: use automated restore tests with SureBackup :-)
Gerben
Influencer
Posts: 17
Liked: never
Joined: May 25, 2016 8:13 am
Full Name: Gerben

Re: Support for differential backups

Post by Gerben »

Okay, so that statement can not be substantiated with a source from Gartner or any other source?

If so, then there is no reason to believe that incremental backups are the way to go. Especially if you don't want to make compromises with the ability to restore from the backup when data corruption occurs.

Also, why use incremental backup, if incremental backup is not a leading "industry best practice for enterprise backup" ?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Support for differential backups

Post by HannesK »

Well, I'm an engineer and I trust in mathematics, in this case in hash functions and check-sums. If you have a different opinion, you might want to use an "active full" backup mode instead of an "incremental forever" mode. Veeam has these options, feel free to use them. There might be Gartner documents but I personally do not read this documents anyway so I'm not aware of them (and honestly I do not really care as I trust mathematics more than any Gartner guy).

You seem to be a lucky guy that has an environment that can do backups without incremental methods. That's fine, be happy :-) In enterprise environments almost nobody has the resources to do backups in a different way than incremental, often combined with active fulls "from time to time".

If you want to protect your data from corruption, you might want to use the 3-2-1 rule https://www.veeam.com/blog/the-3-2-1-0- ... ility.html and https://www.veeam.com/blog/how-to-follo ... ation.html

It makes no difference whether you do incremental or differential backup, if your full backup is corrupted. "Copy your data" (to different media) is the way to go.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 162 guests