Anti-"Feature" Request: Dont remove backward/reverse incremental

[Poweruser]

Hi,

Veeam plans to remove reverse incremental, which is - in my opinion - the best option ever.
So please vote, that .vrb will be kept instead of .vib only.

Maybe it is a performance impact, but its better to keep the latest backup full, because - normally - everyone wants the latest backup and full is better than delta..

[Gostev]

normally - everyone wants the latest backup and full is better than delta

That's because they don't understand that in case of Veeam specifically, it does not make any difference to the restoring of the latest backup.

In legacy solutions, this made a huge difference because users needed to restore a full backup first, and then - as a separate process - apply "delta" on top of the previously restored data. However, this is not how it works with Veeam: in our case, the restore process is the same regardless of how you choose to store your backups, and it always restores the latest state immediately.

[Poweruser]

In my use-case i backup all VMs daily fully to a tape.
so i want only one tape, which includes all my data, without having the need to grab 2 tapes for restore.
if i want t full backup (base+delta) on my tape, i have to force a full backup daily or use the reverse incremental.
if i do a full backup daily, i will waste my HDD space.

am i wrong, or is there a solution to keep 1 full/base and n deltas on HDD and keep base+latest(deltas) on ONE tape with forward incremental (without storing more than 1 base/full on HDD)?

--edit:
also i prefer to delete all the repo data someday to create free diskspace, then i do a repo refresh. i fear, that veeam would not notice, that i deleted n deltas and/or base so its not reconstructable..

[Gostev]

Tape is a whole different and perpendicular topic because Veeam does support synthesizing fulls on-the-fly during tape export, so that what goes to tape is a self-contained full backup file. As such, the need for tape exports does not make a good argument for keeping reversed incremental for disk-based backups. Remember that with Veeam, backups can be stored even in object storage, which has no notions of backup file chains in principle (just millions of separate objects) and yet we still need to let users export this "mess" to tape in a form of full backup files

[Poweruser]

okay gostev, so you say, that if i do a tape job, it will always work standalone, even if 01.01.2024 is base and 15.04.2024 is delta, then (when running jobs on 15.04.2024) it would always pack base onto the tape alongside with the delta of the day?
if yes, could i do sth. wrong that it wont happen?
if no, what exactly do we have to configure to make it so and what must we NEVER do?

is there an easy way to check whats on tape and if its 100% stand-alone?

[Gostev]

Yes but please ask any tape related questions in the corresponding subforum, where the responsible PM will be able to provide a more knowledgeable answer.

[Poweruser]

okay, fork for tape question goes here:
tape-f29/q-is-a-daily-backup-tape-alway ... 93355.html

[richardbradley]

I have was a huge fan of reverse incremental and yes with older file systems this was much faster to mount the restore than a weekly full with forward incremental.

Hiwever with ReFS and XFS this isn't a thing any more and there is no difference between Forward with a synthetic full with forward incremental and reverse incremental.

Plus ReFS and XFS have been around for ages and much better space utilisation, so everyone should be using this.

Yes it can help with tape, but Veeam was the only company I knew to have reverse incremental and they allow you to pick when you want your weekely synthetic full so you can make it the day prior the tape job and behaves the same as the reverse incremental.

So sorry I don't agree, is time reverse incremental goes.

[NickKulkarni]

Hi Everyone,

Thanks to Gostev for the detailed explanation but it raises more questions than it answers for me. I am worried because we have all our jobs on Reverse Incremental. If I roll out v12 (which I have hesitated to do because I wasn't certain how our aging hardware and file systems would handle it) am I going be hit by this.

Background we are on V11, with Veeam ESXi on aging servers and NAS using ISCSI and NTFS. We also have a very slow WAN link to an offsite storage NAS that has poor ISCSI performance as well.

I believe that switching to Incremental will force a new backup chain; is that the case?
What is the performance impact if we also turn on the Create Synthetic Full backups periodically option? Is this necessary or even a good idea.
Our Backup Copy jobs over the WAN link use the option "Read the entire restore point from source backup instead of synthesizing it from increments" because of the performance hit doing this has on the NAS at that end. Is this also going to be affected?

As you can see it opens up a can of worms for me, just don't have the budget for an upgrade of NAS and if ReFS is required will be forced to reformat our entire NAS estate and restore from HDD backups. Its a significant project as far as I can see this . Then again I may have completely misunderstood it. Any and all information / suggestions gratefully accepted.

[Gostev]

V12 still supports reversed incremental. It will only be removed in V13. So feel free to upgrade

[Outlaw]

That's because they don't understand that in case of Veeam specifically, it does not make any difference to the restoring of the latest backup.

In legacy solutions, this made a huge difference because users needed to restore a full backup first, and then - as a separate process - apply "delta" on top of the previously restored data. However, this is not how it works with Veeam: in our case, the restore process is the same regardless of how you choose to store your backups, and it always restores the latest state immediately.

Hello

Did I understand correctly that the recovery time for forward and reverse increment is the same?

[Gostev]

Hello, that is correct.

[Poweruser]

To clarify: i dont want to use reverse incremental to save time.
reasons are:
- i mainly use tapes only as backup (no files on HDD/NAS are saved secure)
- with one daily tape i want to have all backups on one tape only, so i dont have to switch tapes for recovery
- i just use NTFS as it is the best and most compatible and unproblematic FS for very small servers. (one host only for everything with a hand of Hyper-V VMs)
- the host is repo, tapeserver, hyper-v server.. everything...
- if i cant use r-inc. anymore, than i have to use full backups, which will fill every day my disk space quickly.
- maybe i could tell the tape job, to store all deltas on one tape. the documentation how to make sure, that my daily tapes is 100% of all data is very unclear, so i feel unsafe that everything is on one tape only.
- why was r-inc. implemented in past if its so useless??

I bet Gostev thinks like most veeam guys, that we store our data in a repo/NAS whatever. this is not true. its just a temporary storage as B&R does not support direct to tape.
so its a waste of the rare diskspace of the host, and i prefer to keep it very small or empty, as it has not backup. the tape is the only safe thing i have, the repo is considered to fail everytime,
as its on the same host as the vms..

[Gostev]

As agreed, let's please keep the tape export needs to the corresponding subforum thread, instead of keep bringing them here. As I explained, backup file chain format must be completely irrelevant for tape exports because they supports even object storage as the source, which has no backup files at all. For this reason, any uncovered tape export scenarios - which I actually don't believe there's any - should be covered in the tape export functionality. Instead of forcing customers into using some weird backup mode that is not even compatible with most popular types of backup repositories these days (which are hardened repositories and object storage).

"why was r-inc. implemented in past if its so useless?"
This is a bit like asking why was abacus implemented in past if its so useless now. The answer is, because both had its uses in the past? But now there are simply way better options. And even if you feel more comfortable with abacus, it's really the time to switch to calculators!

Likewise, 16 years ago when Veeam started, there was simply no file systems with block cloning nor object storage. So some first Veeam architect came up with this backup mode in order to avoid periodic fulls. He didn't think this through in terms of IO load though, as later we added forever forward incremental which could achieve the same benefit with 50% less IO load on backup storage.

[Poweruser]

yes, you may be right for big systems, low resources and high 24/7 demand and big NAS/etc object storages.
iam just an "old" guy living in a simple world:
1 Host for everything including veeam and 5 VMs (Hyper-V)
1 Tape Drive
All NTFS filesystem
offline backup after office hours
plenty enough i/o resources as hardware is much bigger than the demand.

i agree, that in future (my future) i prefer to have some NAS as object storage, where i want to write data to a hard repo, which cant be deleted by malware and i can go forevery backward for 10 years or so. thats the far goal. but for this i need to buy a big and expensive NAS. maybe qnap oder synology.. but i cant find sth. which is very good, has much features.. so still figuring out.

so currently, i just need on a daily basis just one tape which includes just one full backup ;-)

[anicolai]

Hi

You will get my vote for keeping the reverse incremental.
My usecase is latest backup is also the full one which makes it easy to export to offline media.

-Alex Sørensen

[Gostev]

Hi, Alex. Easy export of any restore point as full backup is available regardless of backup mode you're using > Export Backup

[Poweruser]

@Gostev:
Export Backup should be run using the daily backup job, not manually, and to tape.
is this also possible?

[Gostev]

I kindly ask you (for the 3rd time now) to discuss tape exports in the corresponding topic you yourself has created in the Tape subforum. As this is where Tape PMs live

[Poweruser]

anyway, Alex was talking about exporting to an offline media, so just change my "tape" with a job for an auto export to anykind of media, like external USB device, usb stick, \tape, other filesystem or share without being a veeam repo like a windows share, etc.

[Gostev]

So if we leave tape out of the list, this makes it "rotated drives". These are natively supported for Veeam backup repositories, also irrespective of backup chain type. Just swap the drive and the job having detected a new drive will land a new full backup on it automatically. You can swap them as often or as rare as you like, and each time you swap you will get a new full backup created automatically.

[Poweruser]

ah, so i dont need to do rescans of repos, if i move or delete files from repos?

[Gostev]

Not for backup repositories which have the rotated drives mode enabled.

[ChriFue]

Just my 5 Cents:

As default, synthetic full backups to tape are created only on Weekly, Quaterly, Monthly and Yearly -tape backup schedule in a GFS Pool setting.

But if there is a .vib as the actual daily backup file on disk, you cannot "force" VEEAM to create a synthetic full to tape for this restore point. It will always copy that small .vib as daily to tape.

That is the behavior in traditional weekly vbk. + daily inc. chain on disk.

But if you let your backup to disk write in reverse incremental mode, also the daily restore point will be a .vbk AND the GFS Tape Job with the GFS Pool will write this .vbk to Tape as a daily backup!
Because for the daily-to-tape-job veeam always copies what it "sees" on the disk, and that is the .vbk file of the reverse incremental chain.

And we have lots of customers who want a full backup on tape on every single tape run AND have the comfort of the GFS pool with logging which tape is daily, monthly, yearly and so on with different overwrite protection schedules.

A workaround if no reverse mode is available (v13?) could be to create a synthetic full every day, so the latest restore point is always a vbk. and with ReFS/XFS there is no space wasted.

So I still like the option of reverse incrementals.

[Poweruser]

anyway (gostev will blame me for posting taped related things here) we should have an option on the tape job side, to tell the tape job to save a latest full copy (synthetic?) to the tape under any circumstances (GFS/whatever).
this will always work for people with small data and big tapes ;-)
currently the jobs and the status of diff/delta or full is very confusing and imho it is very unintuivtive, what happens.
i think veeam has to rewrite the gui and clarify the settings much more, so we are aware, what will happen diff or full.

[Gostev]

Folks, reversed incremental backup mode is going away with V13 in any case.

If our Backup to Tape functionality is missing some important feature that forces you into using this backup mode today, please create a feature request in the corresponding subforum, so that the tape PM can prioritize it for delivery in V13/V13.1 as to ensure your migration to the next major release is a smooth one.

Any other approach is simply meaningless because it would not address the same tape export needs of hardened repository and object storage users, who could never use reversed incremental backup mode in the first place.

[JPMS]

I don't want to see the removal for Reverse Incrementals for the reasons of performance.

We run a 12 drive repository with hardware RAID controller and have always struggled with fragmentation of backup files. Any backup system that is based on mechanical drives and involves incrementals is going to have issues with fragmentation over time, getting worse as the number of incrementals increases and is the reason we won't move to REFS (we tried REFS for a couple of months a couple of years ago and found the resultant fragmentation massively degraded performance).

Our solution has been to run Reverse Incrementals and then defragment them once a week before all our major jobs (Health Check, Surebackup and Tape Backup) and that has allowed us to retain excellent performance. It also means any restores are much quicker. If Reverse Incrementals are removed, our only solution will be to run Active or Synthetic Fulls each week which will massively reduce the number of backups we have room for.

[Gostev]

"ReFS a couple of years ago" is not necessarily a great benchmark as Microsoft kept working on optimizing it all this time. And then there's XFS that is yet to see ANY performance complaints on these forums.

Having said that, keep in mind you can schedule Active Fulls monthly or quarterly for example. There's no point in scheduling it each week because significant fragmentation with meaningful I/O performance impact takes much longer to generate.

[BackupBytesTim]

So, just going to share my concern, because I don't see that anyone else mentioned it, but I do not actually expect it to change Veeam's decision, it's been a while since it was decided and I've been aware of it.

My concern actually is unrelated to tape jobs, or other single backup export methods, and more focused around actual daily backups. With forward incremental backups, as is Veeam's recommendation, an initial full backup and every single incremental backup file is required to restore the latest backup, with a reverse incremental backup, you need only the one latest file in order to restore the latest backup. In my experience, most people are more concerned about being able to restore the latest backup than the oldest backup. As Veeam has periodic issues with accessing some incremental files, I would describe it as an issue of how reliably can a particular version be recovered, for forward incremental the oldest existing recovery point is most reliably recoverable due to not relying on multiple files being accessible, with decreasing reliability for every additional recovery point needed to reach the desired version. The opposite is true for reverse incremental, where the latest recovery point requires only a single file to be accessible, with decreasing reliability going backwards to reach older recovery points.

We only actually use reverse incremental for a small number of our customers due to Veeam's recommendation and my awareness that using it would increase work later on to change them when it gets removed, but because we use forward incremental in most of our customers I am aware of periodic issues where Veeam loses a VIB file, says it doesn't exist when it does, says it can't find one that doesn't exist, or a file gets corrupted, different things that are all basically "this file is broke, so the backup chain is useless now". Not knowing what causes these VIB files to become missing or corrupted I typically just restart the backup chain and move on, but every time I do that I think about how this probably wouldn't be a problem for the reverse incremental backup format because the latest data is all just in the latest file, rather than spread across a bunch of files as with the forward incremental backup format.

[BackupBytesTim]

I am curious how forward incremental backups are supposedly more IO efficient, from my thinking through the process, noting I don't actually have access to the code that's doing it, it seems it would be the same amount of reading and writing and deleting data, just that the particular files involved in each step differ between the two methods.