Reversed incremental backup mode

[zadrian]

I really still do not understand why to remove Reversed incremental backup mode ?

Reversed incremental backup mode creates a smaller foot print which
- reduces load on the VMs (eg requirements to quiescence the VMs)
- can be easily transported via WAN
- produces "latest" synthetic full back up set (in times of DR or restoration, how often do we need latest version vs a version 2 weeks old) ?

Is there an alternative or is it time to move off VBR ?

[haslund]

It seems there are some misunderstandings here. There is no reduced load on the production VM when comparing forward incremental vs reverse incremental. The first full will create exactly the same load, after this Veeam will leverage changed block tracking and extract only changed blocks = equal load from either forward/reverse incremental.

When restoring from the latest restore point, there is no benefit to restore from VBK versus a chain. The reason is Veeam metadata. This means sure the VBK contains all the blocks needed, but if you restore from a forward incremental chain, we don't read the entire chain, only the exact blocks needed to restore the latest restore point, but all of the changed blocks that are not needed will not be read/processed.

[Gostev]

Reversed incremental backup mode significantly *increases* load on production VM, which is why it was never used by larger customers.

The reason is much longer incremental backup times (caused by this mode requiring up to 3x more I/O operations on backup repository during incremental run) making VM snapshot remain open for longer time, and the bigger snapshot grows the more impact production VMs experience during the snapshot commit phase.

Reversed incremental mode offers no technical benefits (only technically incorrect perceptions of benefits) but it does have a number of meaningful drawbacks that were previously discussed in existing topics about reverse incremental discontinuation.

Please please please use search before creating yet another duplicate discussion.

[zadrian]

It seems there are some misunderstandings here. There is no reduced load on the production VM when comparing forward incremental vs reverse incremental. The first full will create exactly the same load, after this Veeam will leverage changed block tracking and extract only changed blocks = equal load from either forward/reverse incremental.

When restoring from the latest restore point, there is no benefit to restore from VBK versus a chain. The reason is Veeam metadata. This means sure the VBK contains all the blocks needed, but if you restore from a forward incremental chain, we don't read the entire chain, only the exact blocks needed to restore the latest restore point, but all of the changed blocks that are not needed will not be read/processed.

But when you say "incremental", it is increment from the last full backup data set ?
So if the retention period is 21 days, it will have full + 21 increments with each of the 21 increments being critical component to recovery of the latest backup ?
Example if today is 10th April, I would have "synthetic" full backup at 15 Feb with 21 increments ? If one of the increments (like 1st April was lost or corrupted), can I still recover 9th April ?

Then if there would be daily "synthetic" full due to the retention period, I really do not see the point of wasting that resources to create a full backup that old ?

[zadrian]

Reversed incremental backup mode significantly *increases* load on production VM, which is why it was never used by larger customers.

The reason is much longer incremental backup times (caused by this mode requiring up to 3x more I/O operations on backup repository during incremental run) making VM snapshot remain open for longer time, and the bigger snapshot grows the more impact production VMs experience during the snapshot commit phase.

Reversed incremental mode offers no technical benefits (only technically incorrect perceptions of benefits) but it does have a number of meaningful drawbacks that were previously discussed in existing topics about reverse incremental discontinuation.

Please please please use search before creating yet another duplicate discussion.

Firstly, are you saying that reverse increment is 3x more IO than full backup or increment backup ?

How "big" is bigger ?
- I have a 1.3TB VM that takes 1:22 (82 seconds) to backup.
- I have a 4.7TB VM that takes 7:36 (456 seconds) to backup.

How do you answer for usage of Veeam Backup Copy to copy backup data sets to secondary backup repositories which can be on other sites ?


The previous post you link is mainly all about how much slower is reverse or what to do if there are broken chains or why would there be broken chains etc ?
Then what the post is kindly asking for are "ALTERNATIVES" ?

[Gostev]

1/ Yes, it is increment from the last full backup data set. Chance of one of the increments "lost or corrupted" is exactly the same as chance for the same data block to be "lost or corrupted" within reversed incremental full backup file, as storage-level corruption happens on much lower level than file system.

2/ Wrt. to 3x I/O I'm talking about incremental runs, which is the only thing that matters most users do forever-incremental incremental irrespective of backup mode.

3/ "Big" is 3 times bigger, because incremental backups in reversed incremental mode are 3x slower.

4/ For Backup Copy, backup mode of primary backup jobs makes no difference.

[zadrian]

Like I mentioned to backup 1.3TB and 4.7TB, the respective reverse backup jobs only takes 82s and 456s respectively ?
So are you saying that forward increments for the 2 VMs would take no more than 25s and 150s respectively ? Do you have real figures to support that claim ?

So in short, what are the alternatives for Reverse Increment backup if you need the following ??

1. Most recent "synthetic full backup" with 60 days retention period ?

2. Remote site Backup Repository to have most recent "synthetic full backup" with 60 days retention period ?
- Veeam Backup copy is transferring approx 1.3GB (processed 11.1GB) from a 1.3TB VM (taking approx 11-15 minutes, once every 120 minutes)
- There are 5 Veeam Backup Copy jobs transferring 5 backup data sets (5 VMs) using Reverse Increment backup

[Gostev]

In most environments, backup speed is determined by limited backup repository IOPS capacity, simply because secondary storage is typically slower than primary (production) storage. Reversed incremental required 3x IOs from backup storage during backup, so it will always be 3x slower to extract and store incremental changes than forward incremental. It's just pure physics and math, 3x IOs demand against limited IOPS capacity of a secondary storage device.

Veeam recommends the following setup:
1/ Using ReFS, XFS or object storage for backup target (basically, anything that supports block cloning)
2/ Default backup mode (which is forward incremental backup with weekly synthetic fulls) with custom GFS backups schedule according to business needs
3/ At least one backup target must support immutability (hardened repository, object storage, rotated drives, tape, etc.)
4/ At least one backup target must be off-site (for DR)

[zadrian]

"In most environments, backup speed is determined by limited backup repository IOPS capacity, simply because secondary storage is typically slower than primary (production) storage. Reversed incremental required 3x IOs from backup storage during backup, so it will always be 3x slower to extract and store incremental changes than forward incremental"

I do agree the Backup repository might be shower then the production storage......for example a NAS using SSDs vs a production SAN.

But in terms of forward increment vs reverse increment backup, both are basically the same techniques where increment delta data are taken from the VM, then written to the backup repository ?
The main difference is that
- for forward increment, the synthetic full is created using the oldest retention data before deleting the oldest retention data
- for reverse increment, the synthetic full is created using the current retention data before deleting the oldest retention data

"Reversed incremental backup mode significantly *increases* load on production VM, which is why it was never used by larger customers."
Thats where if this topic is focused on load of production VM (or the VM being backup), what "load" are we referring to ?
Then if the customer (like myself) cannot afford 10 minutes of "load" to backup a 5TB VM, what alternatives are there ?
Lastly, keep saying 3x slower or 3x faster, are there any data for comparison ?

[Gostev]

I explained everything multiple times in both threads you're asking me questions in, but you are just not listening (or not reading carefully). Unfortunately, I can't continue this exchange. I really tried my best to address every item you raised but we're just going in circles at this point and it's not the best use of my time.

[stephen.mintrom]

A reverse Incremental on the repository is 1 Read IO and 2 write OP operations per block, whereas the forward incremental (And active full backup) is 1 write IO per operation.

In a reverse Incremental, Veeam does the following.
1. Receives the new data block from the Production side.
2. Before the block can be updated in the file, Veeam must read the block that is changing from the backup file.
3. The block that is changing is written to the new Reverse Incremental file.
4. The new block from production is written to the new full backup file.

On top of this, because Veeam is reading, writing new files, and writing updates to a previous file, the 3x IO is compounded to be 3x IO that is also Random, as for an Active full or forward incremental on the repository side is 1x write IO of Sequential operations on the disk.

There is also double the requirement of memory on the repository server during this operation, as both data blocks must be stored in memory until the disk operations are completed. In recent years, memory has become cheaper, resulting in systems having more of it in general. This has become less of an issue.

Also, with the requirement to modify the most recent file every backup run with a reverse incremental, it is incompatible with immutable repositories.

Synthetic full operations, for the most part, are only 2 IO operations, read then write, and the only time the read operation occurs is when data is retrieved from previous restore points.
If you are using a repository that Veeam supports, block cloning the read operation doesn't even occur; the existing block is linked to the new file on the File system level.

[zadrian]

I explained everything multiple times in both threads you're asking me questions in, but you are just not listening (or not reading carefully). Unfortunately, I can't continue this exchange. I really tried my best to address every item you raised but we're just going in circles at this point and it's not the best use of my time.

I am getting that it is 3x faster...but is there actual data to compare what is 3x faster ?
I can provide data and timing that says 4.7TB VM takes 5 minute to backup using reverse incremental.
Qn 1 : How much time will it take for forward increment ?

Qn2 : Then what alternatives do we have ?

Qn3 : Then what alternatives do we have if we need to use Veeam Backup Copy to remote sites ?

Qn4 : Then what alternatives do we have if we need a latest synthetic full backup ?

[zadrian]

A reverse Incremental on the repository is 1 Read IO and 2 write OP operations per block, whereas the forward incremental (And active full backup) is 1 write IO per operation.

In a reverse Incremental, Veeam does the following.
1. Receives the new data block from the Production side.
2. Before the block can be updated in the file, Veeam must read the block that is changing from the backup file.
3. The block that is changing is written to the new Reverse Incremental file.
4. The new block from production is written to the new full backup file.

On top of this, because Veeam is reading, writing new files, and writing updates to a previous file, the 3x IO is compounded to be 3x IO that is also Random, as for an Active full or forward incremental on the repository side is 1x write IO of Sequential operations on the disk.

There is also double the requirement of memory on the repository server during this operation, as both data blocks must be stored in memory until the disk operations are completed. In recent years, memory has become cheaper, resulting in systems having more of it in general. This has become less of an issue.

Also, with the requirement to modify the most recent file every backup run with a reverse incremental, it is incompatible with immutable repositories.

Synthetic full operations, for the most part, are only 2 IO operations, read then write, and the only time the read operation occurs is when data is retrieved from previous restore points.
If you are using a repository that Veeam supports, block cloning the read operation doesn't even occur; the existing block is linked to the new file on the File system level.

Thanks.... I do understand this.
But Gostev kept saying that the above process would take longer and "locks" or snapshots the production VM ?'
Qn 1. Maybe I am wrong, but should the creating of the incremental files for both reverse & forward increments take approx same amount of time ?
The only difference is that reverse increment would then use the latest reverse increment file to create a synthetic full backup while forward incremental backup would use the oldest increment file to create a synthetic full backup ?

For my 4.7TB VM, the whole reverse increment backup takes approx 5 minutes. How much time will it take for increment backup ?

[zadrian]

If we "convert" reverse increment backups to forward increment backups and use "create synthetic full backup" on Sat & Sun.
Would the synthetic full backups be deleted after a certain period (eg after retention period is over) ?

[veremin]

Yes, you are right, the oldest synthetic full backup along with its dependent incremental restore points will be deleted when the retention is reached, and when the required number of restore points starting from the next full backup is created.

More information on how the system works can be found in this KB article and in our User Guide.

Thanks!

[BackupBytesTim]

I am getting that it is 3x faster...but is there actual data to compare what is 3x faster ?
I can provide data and timing that says 4.7TB VM takes 5 minute to backup using reverse incremental.
Qn 1 : How much time will it take for forward increment ?

Qn2 : Then what alternatives do we have ?

Qn3 : Then what alternatives do we have if we need to use Veeam Backup Copy to remote sites ?

Qn4 : Then what alternatives do we have if we need a latest synthetic full backup ?

As someone who was recently convinced by @Gostev that the change is actually beneficial, after many months of also being opposed to it, I'll jump in here and answer what I can.

1. There is no way to predict that, due to various factors like compression, I/O speeds, network speeds, etc... it's impossible to predict a specific time any backup will take. I would expect it to take approximately the same time, depending on the configuration, if it is different by much then it's probably something that can be improved by adjusting settings, make sure you're using block-cloning and periodic synthetic fulls.

2. Alternatives, use forward incrementals with block cloning, periodic synthetic fulls, don't update Veeam, or use a different software. There's no "alternative way to do reverse incremental on Veeam".

3. The built-in backup copy feature does not copy backup files the same as you would using Explorer in Windows (or other file browser software), it reads data from all files in a backup chain and copies the needed data to new files in the destination location for the backup copy job, similar to when you restore a device, except you're restoring to a file in another storage location. The backup copy functionality isn't going away, so no alternative is needed.

4. You can schedule synthetic full backups to occur periodically, you can also create one manually at any time if you need it, such as to copy a particular restore point outside of your Veeam backup storage infrastructure.

[clintbergman]

@BackupBytesTim

Perhaps an ignorant question but I'll ask it anyway. How do you create a manual synthetic full? I've been in a few spots over the years where that would have come in very handy.

[BackupBytesTim]

Okay, I see where I created confusion. What I was thinking of isn't referred to as a "manual synthetic full" but actually "Exporting" a backup. Which creates a copy of all data required for a particular already existing restore point. See here: https://helpcenter.veeam.com/docs/backu ... ml?ver=120

It will use Fast Clone if available to create what is essentially a synthetic full backup of the latest restore point (or you can choose a specific restore point), which you can save elsewhere after creation as a fully restorable independent item (1 VBK and 1 VBM) that can be recovered independently of any other incremental files. Because it uses Fast Clone, backups stored in a Cloud Connect repository can only be exported into that repository.

You can also export a restore point as a virtual disk to be used by Hyper-V or VMWare if you want something more independent of Veeam's software, which has always been my "go to" format for archiving decommissioned devices as that format (specifically VHDX) is more conveniently accessible in the future in the event we ever were to not be using Veeam.

[zadrian]

@BackupBytesTim

Perhaps an ignorant question but I'll ask it anyway. How do you create a manual synthetic full? I've been in a few spots over the years where that would have come in very handy.

In forward increment, you can tick "create synthetic full backup periodically on (choose days of the week OR days of the month)".

In Reverse increment (only when you have secondary backup repository), you can use GFS to retain the synthetic full of certain days.

[zadrian]

As someone who was recently convinced by @Gostev that the change is actually beneficial, after many months of also being opposed to it, I'll jump in here and answer what I can.

1. There is no way to predict that, due to various factors like compression, I/O speeds, network speeds, etc... it's impossible to predict a specific time any backup will take. I would expect it to take approximately the same time, depending on the configuration, if it is different by much then it's probably something that can be improved by adjusting settings, make sure you're using block-cloning and periodic synthetic fulls.

2. Alternatives, use forward incrementals with block cloning, periodic synthetic fulls, don't update Veeam, or use a different software. There's no "alternative way to do reverse incremental on Veeam".

3. The built-in backup copy feature does not copy backup files the same as you would using Explorer in Windows (or other file browser software), it reads data from all files in a backup chain and copies the needed data to new files in the destination location for the backup copy job, similar to when you restore a device, except you're restoring to a file in another storage location. The backup copy functionality isn't going away, so no alternative is needed.

4. You can schedule synthetic full backups to occur periodically, you can also create one manually at any time if you need it, such as to copy a particular restore point outside of your Veeam backup storage infrastructure.

Here are the responses.....

1. I am not asking for a prediction....but Gostev kept saying 3x faster, 3 times smaller etc....but are there any actual data ?
You mentioned that you switched from Reverse to forward increment. On the average like if you compared to last month of using reverse incremental and first month of using incremental on your VMs.....
did you see the backup window shrink (like 10 hours to 3 hours) or using less storage (Backup Repository using 3TB instead of 10TB previously) ?

2. What is "block cloning" ? I cannot seem to find this option.

3. I actually really forgot why Veeam Backup Copy was brought into the picture....this is my bad.

4. The "schedule synthetic full backups to occur periodically" is actually an alternatives. Literally we can turn Increment backups into a Dual Increment backup where we run the synthetic full to create a latest synthetic full after the increment backup creates the vbk based on the oldest retention

[BackupBytesTim]

1. The hypothetical benefits only come from a full proper setup in my experience, so you must be using Block Cloning, you must be using Forward Incremental, and you must use periodic Synthetic Full Backups. I can confirm the speed benefit, mainly as a result of not needing to perform file "merges" every time the backup runs. I can't say I noticed any big difference in storage used. For either of "faster" or "smaller" though, I still would say it's harder to calculate or predict anything without actually doing it. I can believe there was a Veeam user with 3x faster speed and 3x smaller storage used, but I would not expect to see that number across the board. It will vary depending on the specific data you have and your device processing speed, network transfer speed, disk I/O speed, etc...

4. Periodic synthetic fulls works well when you're using block cloning and forward incremental jobs.

2. "Block Cloning" is a feature used in XFS or ReFS filesystems, also available on many dedicated storage appliances such as Dell's DataDomain or HPE's StoreOnce, which enables you to "copy" blocks without processing them in the software, the filesystem or storage appliance will copy them instead, so data doesn't need to be transferred. When Veeam uses Block Cloning to create a synthetic full, it generates a new "full" backup using assorted blocks of data from previous incremental and full backup files, so the new full backup is created without transferring any data. This allows those older incremental and full backup files to be deleted while retaining the recoverability for the restore point in the Synthetic Full Backup file, and that file is also used as a base file for future incremental files to be built on to.

On the disk, don't think of a "file" as a container with a lot of data in it, think of it as a list of blocks. "Blocks" represent a container for data on the disk, a small amount of data, usually many blocks are required to make up a file. For some familiarity, you may have defragmented your disks before, which takes the many blocks for a file and moves them physically to sit next to eachother on a disk. In a filesystem that supports Block Cloning, when you "copy" a file, what happens is the filesystem stores a new "file" which is just a list of blocks, so you now have two "files" that reference the same "blocks" on the disk and thus use the same physical space.

When Veeam uses Block Cloning to create a Synthetic Full Backup, it creates a new VBK file (a list of blocks) by using blocks that are already used in previous VIB or VBK files, so your new Synthetic Full Backup doesn't actually need to write new data to the disk and it doesn't actually take up more space, because the data is already there.

Similarly when you use an object storage repository, such as Wasabi, AWS S3, or Azure Blob Storage, Veeam will store all data as a collection of objects (functionally equivalent to blocks), and there are no VIB or VBK files at all. In that scenario Veeam stores the list of "objects" that contain the data for a restore point. So when you upload a new increment, new data is written, but data from the source that didn't change is already there in the existing objects, and so does not need to be reuploaded.