Anti-"Feature" Request: Dont remove backward/reverse incremental

[Gostev]

It depends solely on which particular data chunk is lost. As I've already explained, your repository hard drives store all the same data chunks irrespective of backup mode (only thing different is how they are organized in files on the file system).

Importantly, losing a chunk of a full backup file with reverse incremental mode means you can't recover any restore points at all. And given this one file is constantly updated with your most recent data in case of reverse-incremental backup, the chance is high because storage-level data corruptions typically happens on write operations.

[BackupBytesTim]

Just on the note of that most recent point:

given this one file is constantly updated with your most recent data in case of reverse-incremental backup

I think it's important to mention my biggest issue with the change, the original "full backup" file is constantly updated as well, due to data merges and deletions. So by that explanation "forward incremental" is only better when employed in conjunction with periodic full backups or synthetic full backups and configured so that no "merging" ever occurs. Unless I'm missing something there.

So, to explain my reasoning, with the risk of losing the entire backup chain (due to loss of the full backup) in either scenario, reverse incremental backups seem to mitigate the risk because there's only one single file required for the restore of the latest data. Forward incremental backups would require several files (or hundreds of files, depending on the configuration) to recover the latest data. So while "most" corruption occurs when the file is written, that isn't when "all" corruption occurs, so those other incremental files could become corrupted or unrecoverable for other reasons, in the case of a reverse-incremental backup, those other files aren't needed, which I see as reduced risk.

I will say that right now we don't generally find issues until we go to attempt a recovery and find one or more of the recent forward-incremental increments is inaccessible. Things might be different if we had everything configured for SureBackup verification processes though.

[Gostev]

That is correct, "constantly updated full backup file" is NOT the case with default backup job settings. If you keep periodic synthetic fulls enabled, no backup files are never updated once they have been created, which - aside of massive reliability implications - would also be a blocker for immutability for obvious reasons.

While I did not mention this part much before and honestly reversed incremental would have been removed regardless of this consideration, just based on its low usage and multiple technical drawbacks + proliferation of block cloning compatible backup repositories which make synthetic fulls spaceless; it's removal is also a part of the bigger journey to "immutability everywhere", the place where every backup mode we provide should be compatible with hardened repository immutability feature.

[BackupBytesTim]

I can certainly agree with the benefits of block cloning and the preference for immutable backups, so definitely a positive thing from my perspective, but this is one area where (while a positive move altogether) Veeam seems to be neglecting some particular types of environments (like ours) where block cloning and immutability are not implemented per Veeam's recommendations, reverse incremental seems to be the better overall choice as periodic fulls, even synthetic ones, without block cloning aren't as realistic.

Admittedly though a lot of that comes from my own company's management repeatedly declining my recommendations to modernize our Veeam storage, presently to either the recently released Managed Hardened Repository or to some cloud object storage service like Wasabi.

[zadrian]

It depends solely on which particular data chunk is lost. As I've already explained, your repository hard drives store all the same data chunks irrespective of backup mode (only thing different is how they are organized in files on the file system).

Importantly, losing a chunk of a full backup file with reverse incremental mode means you can't recover any restore points at all. And given this one file is constantly updated with your most recent data in case of reverse-incremental backup, the chance is high because storage-level data corruptions typically happens on write operations.

Yes, but if you look at the 2 real life scenario I given as example.....

How often in a recovery would you want to recover 60 days ago vs most recent ?
Given that if your DB or files got corrupted now or "just now" or some updates implemented crashed the VM ?

If somehow we lost the data chunk due to some HDD issue on the data repository and lost the 1st March 2025 file....
If I need to recover the VM or DB etc...
- I could only recover as of 28 Feb using increment ?
- I could recover yesterdays version using yesterdays synthetic full if used reverse increment ?

[Gostev]

The chance not to be able to recover any restore point is the same. It only depends on whether the required data chunk that the desired restore point consists of is corrupted or not. Backup mode is irrelevant.

[BackupBytesTim]

I was thinking it over and came up with a better illustration for how reverse incremental decreases the risk of data loss for the most recent restore point due to "bitrot" or other completely random (across the whole disk) forms of data corruption.

We have 1 backup, consisting of 7 "restore points", 1 "full backup" that is 70 GB and 6 "incremental" backups, each are 5 GB. The total size is 100 GB.

If this is a forward incremental backup, there is 100 GB of total data (split across multiple files) required to recover the latest restore point.
If this is a reverse incremental backup, there is only 70 GB of data required to recover the most recent restore point.

So this has been my long-standing comparison, that the reverse incremental is inherently safer due to lower total data required for the most recent restore point. However, more recently I've decided I should take my own advice more and not "fight the system" and just use the software as it's intended, even if that means changing the environment and how it's implemented. So now I'm moving more into urging management to approve a restructuring of the Veeam infrastructure, and implementing some additional functionality.

[Gostev]

That is not correct. In both cases, the same amount of total data is required for recovery and the same exact data blocks will be read from backup repository. Never more, never less than what the restored VM state actually consists of. There's simply no point to read 100GB when the restore process only needs 70GB?

Maybe your thinking how some other backup software works and just assume Veeam does the same nonsensical stuff like reading data we don't actually need?

[BackupBytesTim]

Well, it's not about reading the data, it's about the file itself needs to be intact. If Veeam can read the required data from an incremental file, ignoring a corrupted block somewhere else in the file, then that's definitely great. But if a corrupt block in an unneeded part of the file makes Veeam fail to read the backed up data from the intact part of the file, then I would say the entire file is required for recovery. Can you confirm a partially corrupted file, which would no longer pass file integrity checks, can still be accessed to recover the intact data stored in that file (ignoring the corrupted portion)? That would definitely change my opinion on the practical "recoverability" situation.

[Gostev]

Correct, the restore process will never even know if some part of backup file is corrupted because it only reads data blocks it specifically requires. It cannot know about the state of any other blocks because the validation of the retrieved content against its checksum is only performed for blocks as they are read.

[BackupBytesTim]

Ah, okay. That actually sounds like a logical design then. I did not assume that was how that worked, assumed a partially corrupted file would cause Veeam to fail to load any data from the file, but if that is not an issue then I'd say that eliminates my last good reason for opposing the switch. Now if we can get our Veeam infrastructure modernized some, sounds like the change will end up being beneficial. Although we only have a few customers left using reverse-incremental jobs anyways.

Thanks for clarifying that.

For comparison against other backup software, the only other backup software I've ever actually used at a similar scale stores all backup data in a single file per source, so there's not really "forward" or "reverse" in the same manner as Veeam as there aren't separate files. So I have no good comparison to make to other software, was just an assumption about how data verification and data access for incremental files would work, glad to see it was an incorrect assumption, although in my opinion having SureBackup set up would still be beneficial for full data verification purposes.

[zadrian]

The chance not to be able to recover any restore point is the same. It only depends on whether the required data chunk that the desired restore point consists of is corrupted or not. Backup mode is irrelevant.

How can it be irrelevant ?

If we need to recover to the most recent version....
1. For reverse incremental backups, all we need is yesterday's synthetic full backup
2. For incremental backups, we need the synthetic full backup which was created 60 days ago and all of the 60 days of incremental. One lose incremental and it makes the whole backup useless ?

[Gostev]

Storage-level data corruption is not aware of file system above, it happens on much lower level. Therefore it doesn't cause you to lose entire files but rather the content of particular file's block gets corrupted. All your backup files will be in place and looking like nothing is happening even in case of continuous heavy corruption. Until you try to restore, at which point it's pure luck: did the specific data blocks required for restore get corrupted or not, irrespective of which backup file they are located in.

[BackupBytesTim]

How can it be irrelevant ?

If we need to recover to the most recent version....
1. For reverse incremental backups, all we need is yesterday's synthetic full backup
2. For incremental backups, we need the synthetic full backup which was created 60 days ago and all of the 60 days of incremental. One lose incremental and it makes the whole backup useless ?

That was my assumption as well, basically that the entirety of an incremental backup file needed to be intact in order for even just a part of it to be accessed, but apparently not. So the actual number of blocks on the storage volume needed to recover data in either case remains the same. So, actually not the issue I believed it would be.

Admittedly though I see why this sort of thing was confusing, Gostev understood how it worked so from his perspective there was no difference, but it was never explained to me (and others) that that was how it worked until just a couple posts ago, so we went on trying to explain a problem that we saw because we assumed it worked one way, which was incorrect. Not sure if the ability to reliably access non-corrupted blocks of a file that contains corrupted blocks is actually outlined in any docs somewhere, but I didn't know it was a thing. Actually has gone towards my recently improved viewpoint of Veeam.

[WoenK]

How can it be irrelevant ?

If we need to recover to the most recent version....
1. For reverse incremental backups, all we need is yesterday's synthetic full backup
2. For incremental backups, we need the synthetic full backup which was created 60 days ago and all of the 60 days of incremental. One lose incremental and it makes the whole backup useless ?

3-2-1 ?

[zadrian]

That was my assumption as well, basically that the entirety of an incremental backup file needed to be intact in order for even just a part of it to be accessed, but apparently not. So the actual number of blocks on the storage volume needed to recover data in either case remains the same. So, actually not the issue I believed it would be.

Admittedly though I see why this sort of thing was confusing, Gostev understood how it worked so from his perspective there was no difference, but it was never explained to me (and others) that that was how it worked until just a couple posts ago, so we went on trying to explain a problem that we saw because we assumed it worked one way, which was incorrect. Not sure if the ability to reliably access non-corrupted blocks of a file that contains corrupted blocks is actually outlined in any docs somewhere, but I didn't know it was a thing. Actually has gone towards my recently improved viewpoint of Veeam.

But there are real events where we needed to actually move the Backup Repository from the remote site to another site (it was a test of DR when we "assimilate the move of data center due to some massive event like DC burnt down). But do not ask me how or why.....some files went missing and/or corrupted.

So if the NAS had Full backup File (60 days ago), increment backup files of 59th, 58th....45th, 20th, 19th, 18th till yesterday (missing were 44th day till the 21st day). Can we still restore to yesterday backup if using incremental ?

[BackupBytesTim]

In your message it was unclear to me whether you meant an entire file was "missing" or just "corrupted", but...

For your backup repository transfer to the secondary location are you using Veeam's Backup Copy feature? If yes, you should be able to enable the Health Check functionality. This will run a scheduled CRC verification of the transferred restore points, and replace any missing or corrupted data by retrieving it from the source repository again. https://helpcenter.veeam.com/docs/backu ... ml?ver=120

If the health check detects corrupted data blocks, together with data blocks for the new restore point, Veeam Backup & Replication transports valid data blocks for the corrupted restore point. The valid data blocks are stored to the new incremental restore point created by this backup copy session. As a result, the backup chain gets “fixed”, and you get a possibility to restore data from restore points following the corrupted restore point.

Similarly you can do the same for the original backup files, not just a Backup Copy: https://helpcenter.veeam.com/docs/backu ... ml?ver=120

As a more complex measure, you can configure SureBackup to actually test restoring backup data to a VM to verify the recovery completes and is bootable.

To try to elaborate on my previous explanation:

Every restore point contains "blocks" of data. When you recover the latest restore point with Reverse Incremental, you only need the most recent file because it has all the blocks required to restore the latest restore point for an entire disk recovery. When you recover the latest restore point with Forward Incremental it will retrieve the required blocks from across multiple files, the total number of blocks required is the same in either scenario. So if we assume all files exist and the issue is random corruption of blocks that is in no way isolated to a single file, as long as the required blocks are available the restore point can be restored.

In theory the Forward Incremental method is better here because you can write the file to a disk and then never modify it again, reducing the chances of corruption due to failed processing or data transfer errors. So then you only transfer "new" data whenever a new restore point is created, you don't reprocess old data to "merge" the files. This works best when you use a storage location with block cloning, such as XFS or ReFS, also various storage appliances such as Dell's DataDomain or HPE's StoreOnce. This enables the use of Fast Clone to create Synthetic Full Backups. Which reuses blocks of data from previous files to create a new full backup file periodically, but without taking up additional disk space, so the older files can be deleted while retaining all the data required for a newer restore point.

If your issue you described was in fact that entire files were missing from your transfer, that sounds like you'd need to address that with whatever software was doing the data transfer.

If you were actually missing those files you mentioned, then no, you wouldn't have been able to restore the latest restore point, but that would be because files were missing entirely rather than corrupted. Which sounds like an issue with whatever did your data transfer to the DR site, which is different than data corruption.

What method did you use to duplicate your data to the DR site?