Anti-"Feature" Request: Dont remove backward/reverse incremental

[Gostev]

Code knowledge is not needed, it's all about the number of I/O operation on repository. For each incoming block, reversed incremental mode needs to read replace block from VBK, write it into VRB, and write incoming block into VBK (so 3 I/O operations). While with forward incremental, we just write the incoming block into the VIB.

Slow backups means snapshots stay open for much longer on production machines, resulting in longer on-going impact following by heavy I/O due to snapshots growing large. This was basically the very reason why most customers abandoned reversed incremental by now, and its actual usage simply does not justify keeping it around any longer.

[JPMS]

"ReFS a couple of years ago" is not necessarily a great benchmark as Microsoft kept working on optimizing it all this time. And then there's XFS that is yet to see ANY performance complaints on these forums.

This isn't an issue of whether ReFS is better now, the whole nature of the disconnect between Veeam software and any underlying file system that uses block cloning means you will always end up with fragmentation without active fulls, however good the filing system. I suspect the reason you don't see performance complaints with XFS is the vast majority of operations with B&R are writes. The effects of fragmentation only really show up with reads (assuming you have reasonable hard disk space), at least as far as Forward Incremental is concerned.

Folks, reversed incremental backup mode is going away with V13 in any case.

It's good to see you listening

[JPMS]

I have was a huge fan of reverse incremental and yes with older file systems this was much faster to mount the restore than a weekly full with forward incremental.
Hiwever with ReFS and XFS this isn't a thing any more and there is no difference between Forward with a synthetic full with forward incremental and reverse incremental.

You clearly don't understand how the technology works, You are quite correct that there is "no difference between Forward with a synthetic full with forward incremental and reverse incremental" with block cloning because none of them produce a defragmented file. That is certainly not the same as "this isn't a thing any more". Fragmentation is still very much a "thing" with block cloning. If you want to create a defragmented file you have to run an Active Full.

Plus ReFS and XFS have been around for ages and much better space utilisation, so everyone should be using this.

It's not a great space saver if you still want to create defragmented files by running Active Fulls and hard disk space is (comparatively) very cheap. If you are running flash storage then it becomes a very different story, both in terms of the impact of fragmentation and storage costs but that is not a cost effective solution for us.

There is the benefit of immutable backups but that is not a high priority for us as it would mean supporting an additional OS and we backup to tape anyway.

[JPMS]

Hi, Alex. Easy export of any restore point as full backup is available regardless of backup mode you're using > Export Backup

Having spent a good few hours looking at this (so certainly not an expert), running a Backup Copy Job (as Active Full) seems a better solution as it appears to be better supported by Tape Backup options.

These are early days though, so I'm quite prepared to be shot down in flames.

[richardbradley]

You clearly don't understand how the technology works, You are quite correct that there is "no difference between Forward with a synthetic full with forward incremental and reverse incremental" with block cloning because none of them produce a defragmented file. That is certainly not the same as "this isn't a thing any more". Fragmentation is still very much a "thing" with block cloning. If you want to create a defragmented file you have to run an Active Full.

You are correct re your extra detail however it's not that I don't understand the technology I just didn't feel I needed to explain block cloning.

It's not a great space saver if you still want to create defragmented files by running Active Fulls and hard disk space is (comparatively) very cheap. If you are running flash storage then it becomes a very different story, both in terms of the impact of fragmentation and storage costs but that is not a cost effective solution for us.

It should be noted Veeam do have a defragmentation option in the job settings that fixes this. Hopefully this fixes your concern with forward incremental and Synthetic full backup.
And before you say hey the synthetic full that is defragmented is not the same as a active full you are right, however it does get the space close and gives you the benefit of block cloning using ReFS and XFS.
I am also not sure where you work or work with but I have never been lucky enough to find any company that is willing to pay for SSD storage for backups, all my clients still use cheaper storage or cloud storage. Generally spinning storage still gives you more storage per $ than SSD.
But in saying the above, there are uses cases for active fulls and not defragmenting there absolutely is but in all the environment I deal with I haven't needed them when I have ReFS and XFS with Synthetic Fulls.

[richardbradley]

This isn't an issue of whether ReFS is better now, the whole nature of the disconnect between Veeam software and any underlying file system that uses block cloning means you will always end up with fragmentation without active fulls, however good the filing system. I suspect the reason you don't see performance complaints with XFS is the vast majority of operations with B&R are writes. The effects of fragmentation only really show up with reads (assuming you have reasonable hard disk space), at least as far as Forward Incremental is concerned.

JPMS, I work for a SP that runs hosted environments, on prem/BaaS and provide cloud storage and we don't see any impact when it comes to read slowness of the backups. I would suggest if you are getting this you need to review your environment as there is an issue there.

[BackupBytesTim]

Code knowledge is not needed, it's all about the number of I/O operation on repository. For each incoming block, reversed incremental mode needs to read replace block from VBK, write it into VRB, and write incoming block into VBK (so 3 I/O operations). While with forward incremental, we just write the incoming block into the VIB.

Slow backups means snapshots stay open for much longer on production machines, resulting in longer on-going impact following by heavy I/O due to snapshots growing large. This was basically the very reason why most customers abandoned reversed incremental by now, and its actual usage simply does not justify keeping it around any longer.

Your explanation of IO for reverse incremental is as I assumed it to be, basically 3 steps, however from my understanding, forward incremental is the same 3 steps. I do see your confusion though, your understanding of how it actually reduces IO load assumes unlimited storage space. Which isn't actually a thing in any environment I've seen. So in a real, not your internal testing and development, environment, Veeam writes incoming data to a new VIB file, reads oldest VIB file data, writes that data to the VBK, and deletes expired data from the VBK. The last step also occurs with the reverse incremental scenario in the form of deleting the expired VRB file.

I do like the idea of the hypothetical benefits you speak of, so if you know of a way we can actually achieve your scenario of unlimited storage and thus actually gain the benefits of reduced IO from using forward incremental backups, let me know.

If you're sure about it, then okay, I can't prove it one way or the other so I won't debate it, but are you sure people switched because of your hypothetical benefits? Or did they switch because they either use the default setting and you changed the default setting, or otherwise because you recommend forward incremental backups now as the preferred option, possibly because you've been saying reverse incremental will be removed eventually, so like me they do it just to avoid work in the future of changing things?

[Gostev]

Yes, I'm very sure simply because what I explained was the very specific feedback we acted on when we added forward incremental backup mode to the product you may not know this but for the first few versions, Veeam did not provide any backup chain options with reverse incremental was the only mode available.

Next big jump of migrations off of reverse incremental was when we introduced block cloning integration, which made those periodic synthetic fulls of forward incremental backup mode not consume any disk space.

And the final wave of migrations was when we introduced hardened repository with immutable backups.

[BackupBytesTim]

I did not know that, but I assumed it. The additional option does make sense, and if most people moved away from using it, especially for performance reasons, then removing it can make sense too. Though I do stand by my analysis of observed issues with forward incrementals that are not present for reverse incrementals in my experience, though my experience is limited somewhat by our small customer base.

I've mentioned before, and so I'll mention it again as it would seem relevant, but I still believe a single-file backup format would be more efficient still over either forward or reverse incremental formats and potentially less prone to errors.

A single file wouldn't need to involve reading and rewriting data to move it amongst different files as new restore points are created, instead simply writing data one time and later deleting it one time when it expires. This would bring the performance benefit of forward incremental backups to the more realistic environment of not unlimited storage space. Also having only a single file would seem to reduce the likelihood of errors arising from a maintaining series of files, errors that do result in some files seemingly going missing, and errors that result in orphaned files, which is something I still deal with regularly, manually cleaning up files that Veeam created and later forgot about so just got left behind in the repository.

[Outlaw]

Hello, that is correct.

Hello

But how? During recovery, I read out all the increments from the last full in sequence. Instead, during the reverse increment, I always have the last full.

[Gostev]

Hello, I've already addressed in my very first reply to this topic. Veeam does not work like that: "read out all the increments from the last full in sequence". For each restored block, we read its actual state in the selected restore point right away, so it does not matter which backup file it is located in. Thanks

[RubinCompServ]

As Veeam has periodic issues with accessing some incremental files, I would describe it as an issue of how reliably can a particular version be recovered, for forward incremental the oldest existing recovery point is most reliably recoverable due to not relying on multiple files being accessible, with decreasing reliability for every additional recovery point needed to reach the desired version. The opposite is true for reverse incremental, where the latest recovery point requires only a single file to be accessible, with decreasing reliability going backwards to reach older recovery points.

...I am aware of periodic issues where Veeam loses a VIB file, says it doesn't exist when it does, says it can't find one that doesn't exist, or a file gets corrupted, different things that are all basically "this file is broke, so the backup chain is useless now".

This is one reason why I run Reverse Incrementals for some of my customers.

Not knowing what causes these VIB files to become missing or corrupted I typically just restart the backup chain and move on,

That's also been the "official" Veeam support stance when I opened tickets for these issues. The response has basically been, "Rescan isn't finding/reading the file, it must have gotten corrupted somehow, just run a new Active Full to fix the problem", which then puts me in the less-than-pleasurable position of explaining to a customer why they no longer have any recent restore points for their critical server.

[BackupBytesTim]

Thank you for joining in, nice to know I'm not the only one with that issue.

I suppose theoretically we could make things work more reliably with forward incrementals if we upgraded our infrastructure to handle the increased load and configured regular automated health checks that automatically repaired broken chains by running backups again. I assume that's something that can be configured to run completely automatically, though currently I don't believe our infrastructure could handle daily health checks for all our backups.

[RubinCompServ]

Funny that you bring up the automated health checks, because there's another thread detailing how running backup jobs will break the health check jobs. To your larger point, though, I have upwards of 500 jobs running daily - there's no way I can handle daily health checks for all my jobs.

[Outlaw]

Hello, I've already addressed in my very first reply to this topic. Veeam does not work like that: "read out all the increments from the last full in sequence". For each restored block, we read its actual state in the selected restore point right away, so it does not matter which backup file it is located in. Thanks

Thanks, but I still don't understand how this magic works "we read its actual state"

[BackupBytesTim]

Funny that you bring up the automated health checks, because there's another thread detailing how running backup jobs will break the health check jobs. To your larger point, though, I have upwards of 500 jobs running daily - there's no way I can handle daily health checks for all my jobs.

I do remember reading that somewhere now that you mention it. So that's not the best option either then. Even if we ignore the infrastructure requirements.

[JPMS]

I still haven't seen anybody from Veeam respond regarding the issue of broken chains.

We have some large data stores (c. 15TB) that change little and aren't business critical that we backup with Reverse Incremental (and dump out to take every 3-6 months after an Active Full backup). We don't really want to be doing this more frequently and deliberately chose Reverse Incremental because we knew that if one of the delta files became corrupted, we could at least recover the latest backups.

Any follow up on this @Gostev ?

[Gostev]

There are no known issues with the product resulting in "broken chains" nor there are similar cases from other users lately so you will need to investigate your situation with our Customer Support if you want to get to the bottom of this. Which I would say is important because most issues that break backups usually have to deal with storage-based corruption and this will make your backups unrecoverable regardless of the backup mode used to create them. Thanks

[Regnor]

Backup files shouldn't get corrupted or 'lost' by themselves. If you experience this regularly then you should search for the root cause, and not rely on reverse incremental.
So far I've only seen that because of storage issues, and in many cases more than one backup file got corrupted. Especially with block cloning, a single disk error will have greater effects. So there's also a high chance that your VBK will broken and you end up with no backups at all.
So make sure that doesn't happen regularly and follow the 3-2-1 rule, just in case you lose the primary backups.

[JPMS]

I think you have misinterpreted my question; I have never suffered from file corruption or loss.

I do, however, try to consider all the possible eventualities and what is the best way to mitigate against those possibilities. In the event that I do , at some time in the future, suffer a hardware issue that causes file loss or corruption, I will have a better outcome with Reverse Incremental than Forward Incremental.

[Regnor]

I would change the "will" to "could" (I "could" have a better outcome)
Like I said, I've seen issues affecting a high amount of disk blocks or files on volumes. And those resulted in many or all backup files getting damaged and/or corrupted.
So in your case, you could also end up with losing your only full backup.

Therefore, 3-2-1 and backup copies will increase the reliability of your backup solution, and not the backup method.

[Servior]

As some said already, reverse incremental is a good way to do full backup to tape, for example as daily full to tape.
You can have weekly active or synthetic full on disk and it still works.

How to do that with incremental?
Virtual full on tape is not possible with periodic full configured. I know workarounds, but workarounds are bad. I don’t want to use forever forward incremental.

[Gostev]

As I also already said to those some please direct any and all tape export related requests to the corresponding subforum. There, the tape PM will explain how to achieve your requirements today, or will take feedback to ensure any missing but commonly requested tape export functionality is implemented. Thanks