Transform previous backup chains into rollbacks

[MattFlowers]

Hello,

This is in response to the community digests email that was sent this week about removing the "Transform previous backup chains into rollbacks".

We currently use the forever incremental backups with Transform previous backup chains into rollbacks. We do the transform every day (not once a week). I have had support reps ask me why I do this and not just use the reverse incremental jobs. Here is why:

When the reverse incremental job runs it works like this:
1. Take snapshot of VM
2. Do incremental backup of VM
3. Create synthetic full for today and create the reverse incremental.
4. Remove snapshots.

How the "Transform previous backup chains into rollbacks" works:
1. Take snapshot of VM
2. Do incremental backup of VM
3. Remove snapshots.
4. Create synthetic full for today and create the reverse incremental.


The big difference between the reverse incremental and the "transform previous backup chains into rollbacks" is when it releases the snapshot of the VM. The incremental backup will be relativity quick but the synthetic full could take much longer. This means that the VMs will be sitting on on their snapshots for longer than necessary and will lead to longer stun times/IO usage during the snapshot consolidation. This is less of a problem with storage integration and ReFS but I still dont understand why Veeam works differently with reverse incremental and "Transform previous backup chains into rollbacks".


If you are going to remove the "Transform previous backup chains into rollbacks", please (if possible) change the logic on reverse incremental to remove the snapshot after the incremental backup is complete and before the synthetic full occurs.

[Andreas Neufert]

Thanks for sharing. Yes, I can confirm that the Snapshot and the Proxy Task slot are kept occupied during the whole backup at Reverse Incremental.
In the old times daily processing of the transform helped with both as you have described while not having to waste space with the Synthetic Full + Incremental processing.

But since some years we have the Forward Forever incremental chains (Select Incremental and disable Synthetic or Active Fulls). This will basically address both of the things. It will just create and incremental and then close snapshot and as well close the Proxy task slot. Then it will "merge" the backup in a similar way as the transform. With this we achieved way smaller backup windows because the proxy task slot and snapshots are closed more early.

Did you tried this backup chain format?

[MattFlowers]

What backup chain format are you talking about? I am sorry but you lost me a little bit in your post.

[foggy]

Hi Mathew, Andreas is talking about forever forward incremental backup. What you're using now is simple forward incremental with periodic (daily in your case) synthetic fulls and transformation to rollbacks. With forever forward method, there's no periodic fulls at all, just the initial full with subsequent increments.

[MattFlowers]

Yes, I can do forever forward incremental backups but I dont want to do that. The backup style that I use is the reverse incremental but I do forward incremental backups with daily synthetics and rollbacks. This works fine for me and I dont have any issues with it.

My original post was in response to Gostev's Veeam Community Forums Digest email from Sunday.

"Specifically, I'm talking about the Transform previous backup chains into rollbacks option of the forward incremental backup. This option was added in the early days of Veeam to reduce the disk space usage via eliminating previous full backups by transforming them into rollbacks. So, very little difference comparing to our reverse incremental backup, except if there it is done "inline" during every backup, here this whole process happens once a week after the new scheduled full is created. "

Gostev is talking about Veeam's plan to remove the "Transform previous backup chains into rollbacks" option in the future because "there is very little difference" between that and the reverse incremental.

I was trying to point out that the way that reverse incremental works is that it will not remove the VM snapshots until the job completes the full backup chain transformation.

Reverse incremental - Doesnt remove the VM snapshot until the synthetic full/rollbacks are finished.
Forever forward with rollbacks - Removes the VM snapshot once the incremental is done and before the synthetic full/rollbacks are finished.

I dont have any issues that I need help with, I just want to point out about how the two work differently and how the forever forward with rollbacks is a better way to do the reverse incremental job. I also would like to ask, if Veeam removes the "Transform previous backup chains into rollbacks" please change the logic of reverse incrementals to remove the VM snapshot before the synthetic full/rollbacks happen. It is always better to remove the VM snapshot as soon as possible to limit the effect it has on IO/stun times.

[HannesK]

Hello,
we are aware that there are differences between the backups modes (I came up with the idea to remove transforms). On the other hand, the big amount of backup modes is confusing customers for no real benefit.

Yes, I can do forever forward incremental backups but I dont want to do that.

could you maybe explain what the reason technical is? I see that it works fine for you, but I also see the downsides of the many backup modes. For agent and Nutanix backup we only have forward with / without active or synthetic full and nobody ever asked for reverse incremental or transforms to rollbacks. Sure, agents do not cause a VM stun. But this topic is the same for forward incremental and your daily transforms scenario.

please change the logic of reverse incrementals to remove the VM snapshot before the synthetic full/rollbacks happen

that would cause more IO than reverse incremental or forever forward incremental. that would result in many problems for customers that do not have block cloning (REFS / XFS)

Best regards,
Hannes

[jmacdaddy]

I second much of what Matt has said. Forward incremental with transform to rollbacks is far superior to Reverse Incremental because it decreases the size of snapshots and the length of time in which a snapshot is held open. Also, it decreases the amount of time that the source VM's virtual disks are attached to the proxy server when using HotAdd mode. Basically, it cuts the probability that a failed component (proxy server or backup server) will impact a production VM. Think of it like choosing to run across a highway or walk across a highway. Getting across faster cuts your chances of getting hit.

The reason we want to use rollbacks is that many of my customers backup their Veeam vbk files to tape every night. We set our daily backup to disk jobs as forward incrementals with nightly transforms to rollbacks. This assures that when the backup job completes that we only have to grab one file (the transformed .vbk) to ensure that we have everything we need. We configure our file to tape jobs with a *.vbk filter to only grab that full backup file. If we did forever forward incrementals, we would always have to backup to tape the full backup and all of the forward incrementals in order to have all of our systems in their current state on tape. Obviously this would take far more space on the tapes which is an added expense, and if we have to add in additional tapes per job, added complexity (which means added risk).

[Gostev]

Regarding the tape part: actually, Backup to Tape can do what you need regardless of the source backup chain type, as they send synthetic fulls to tape (creating VBK on the fly from the selected restore point).

[jmacdaddy]

Gostev - I thought that Backup to Tape was only in the Enterprise edition? Almost all of our customers, especially in the SMB market, are using Standard edition. Is Veeam going to include the Backup to Tape feature in the Standard edition if you remove the rollback transform feature?

[Gostev]

Yes, that's definitely an option we can consider, if it helps us to get rid of this backup mode.

[JPMS]

We output to tape as a secondary backup and use forward incremental with daily synthetic and transform rollbacks to minimise fragmentation.

I'm sure fragmentation is an issue for everyone but there is quite a difference in impact between the occasional restore and having to copy your entire backup to tape every week (or night). It's the same reason that REFS has no appeal for our requirements.

SSDs still have quite a way to go before they can offer the capacities we are using at a cost effective price.

[HannesK]

@JPMS: uhm, that does not minimize fragmentation

"defragment and compact full" minimizes fragmentation.
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

To reduce fragmentation, it makes no difference whether you do reverse incremental, forward forever incremental or use transforms with "defrag and compact full"

Best regards,
Hannes

[JPMS]

@HannesK

Thanks for the reply even if it exposed my ignorance!

I appreciate the purpose of "defragment and compact full" but I'm really talking about the best way of minimising fragmentation during backup rather than how to produce a completely defragmented backup file (We run monthly active full, partly for that reason, partly to guard against losing a long chain of backups if the vbk gets corrupted).

I spent some time checking what you said - not because I didn't believe you but I wanted to try and understand how and why my understanding was wrong. There are so many sources of information both here, help files, KBs and blogs. After about an hour I gave up trying to find exactly what I wanted but one example shows how it is easy to get confused.

https://helpcenter.veeam.com/docs/backu ... ml?ver=100 (How Synthetic Full Backup Works) says that it creates a new vbk file. It is easy to read that and believe that this is a completely new (and therefore defragmented file). So, I've now looked at our own backups and realise that this can't be the case because they don't take long enough and if I look at the vbk file it has 3209 fragments (Active Full is due to take place tomorrow). What I presume is happening is that the Synthetic Full is actually created by block injection into the existing vbk file (and block removal into the vrb file?) It isn't really a 'new' file at all but the help topic doesn't mention this at all.

So, assuming this new understanding is correct, and from what you have said, vbk files created with reverse incremental vs forward incremental with synthetic full are equally fragmented. Is there any advantage to synthetic fulls with rollbacks compared to reverse incremental? Are the file sizes the same?

[HannesK]

minimising fragmentation during backup

I don't understand the idea of that. But the same happens with (forever) forward incremental.

There are so many sources of information both here, help files, KBs and blogs

that's why I suggested to remove that backup mode

The link you mention is not for transforms. It's the normal "forward incremental with synthetic full". In that mode, the UI does not even allow to defragment. Same by the way if you have transforms + active full.



https://helpcenter.veeam.com/docs/backu ... ml?ver=100 has an animation how transforms work. yes, it injects to the full.

As this thread is about discussions on transforms, I would like to not describe every backup mode. https://www.veeam.com/kb1799 has nice animations. I also always recommend http://rps.dewin.me -> check the "manual run" and see how retention works

[JPMS]

OK, I think I understand the source of my confusion and I'm blaming your user interface!

The option to 'Transform previous backup chains' appear as a 'sub' option of 'Create synthetic full backups'. I therefore assumed that it first created a synthetic full (which would be a defragmented file) then did the transform. What you are saying is when this option is ticked it doesn't do a synthetic full at all, it does a 'transform' backup using the existing vrb.

This topic is about whether people still need the transform option. I thought I did but now realise that my understanding was wrong and it isn't achieving what I want. I will change to using reverse incremental.

[HannesK]

OK, I think I understand the source of my confusion and I'm blaming your user interface!

that's why I want to remove it

thanks for confirming.

[cmaier]

Regarding the tape part: actually, Backup to Tape can do what you need regardless of the source backup chain type, as they send synthetic fulls to tape (creating VBK on the fly from the selected restore point).

Sorry to say that but the Veeam tape backup is still completely unusable for us. We still use for tape backups - yes! - Backup Exec 2010. Not because I like it that much, it's because of VBR.

In first place VBR is still not able to handle different drive generations in a single loader based on the tape's bar code label suffixes. I cannot tell it that an LTO 7 drive can only write to LTO 7 and LTO 6 tapes but not to LTO 5 tapes. I cannot tell it that an LTO 6 drive is not able to handle an LTO 7 cartridge at all. Your only solution is to partition the loader which I definitely don't want to for different reasons. That's really sad.

Second there is still a lack of differential backups, you are only able to create incrementals. Yes, I need this here and there especially for File 2 Tape backups.

For File 2 Tape backups there are no GFS Media Pools available. Why?

And back to topic: We need "Transform previous backup chains into rollbacks" for our tape backups, too. Removing it would completely kill our backup strategy at the moment. We write the daily .vib to the daily tapes and the weekly .vbk to the weekly tapes while ignoring the .vrb files. We don't want to have more than one .vbk in a folder because of it's size. But I can have tens or hundreds of .vrb in a folder to get back some weeks or months without using the slow tape. The transformation on a ReFS volume is done very fast.

And I don't like the idea to remove features because they could confuse someone. There are still some power users out there, not only noobs.

Best regards,
Christian

[Gostev]

Hi, Christian.

Please share your tape feedback in the tape subforum, where it can be processed by the responsible PM. We still have time to address any shortcomings before this feature goes away.

We're not removing this feature "because it can confuse someone", but because it is very costly to maintain from R&D perspective, while less than 5% of customers are using it. With even fewer doing this for a valid reason (like yourself).

Thanks!

[cmaier]

Hi Anton,

I posted all the points years ago in your forums.
post175692.html#p175692
post191340.html#p191340
post321668.html#p321668

I contacted one of your instructors after a Tape webinar in the year 2017 and told him all that. I asked the same things again after a webinar in 2019. I told it your support guys and so on.

Obviously your company is not really interested in a good and competitive Tape Backup solution. It's all about cloud, cloud and cloud. That's really annoying because your product was an eye opener in VMware Backups when I came to you about 8 years ago. We have our own data center and are not interested in cloud solutions at all. But we really need a good Tape solution for air gapped backups. I have to use the described workarounds for years and a really outdated Backup Exec until now. Such a shame.

[HannesK]

Hello,
Your first request should already be solved. Daily incremental backups are possible in GFS media pools: https://helpcenter.veeam.com/docs/backu ... ml?ver=100 (I know, that requires synthetic / active full. Block-cloning does synthetic fulls without using additional disk space)

The second request looks like a performance request: transforms are the slowest option possible. I mean, it works fast with REFS / XFS. But with classic filesystems it's painful. The question is, whether you want to go with filesystems that do not support block-cloning in future. Synthetic fulls with block cloning do not cost disk space. That would allow daily incrementals to tape.

your third request to support multiple LTO generations is absolutely valid. And we need to do that before removing transforms to cover situations like yours. Anton (head of product management) already confirmed your request in his post.

Best regards,
Hannes

[mcz]

I think block-cloning is a great feature but if you don't have it right now, you have to swallow the bitter pill of transformation. I've searched a little bit on this forum and checked the helpcenter but I couldn't really find an answer why during the merge of a forever forward incremental veeam would create a complete new full (temp-file). I've got a 1 TB vm and this means that during every run it would create a whole new 1 TB file. What is the reason why veeam can't just merge the vib into the vbk like it would when injecting the vib's into the full and transforming into rollbacks?

My current chain looked like this:

vrb - vrb - vrb - vbk - vib - vib - vib

Transforming is one thing, but creating a whole new full (starting as a temp file) is another one - at least for me.

Thanks!

[HannesK]

I've got a 1 TB vm and this means that during every run it would create a whole new 1 TB file.

@mcz: do you have a case number for that behavior with forward incremental forever?

That's not how it is supposed to work. the animation for forward incremental forever merges is available in www.veeam.com/kb1932

[cmaier]

Hello Hannes,

Hello,
Your first request should already be solved. Daily incremental backups are possible in GFS media pools: https://helpcenter.veeam.com/docs/backu ... ml?ver=100 (I know, that requires synthetic / active full. Block-cloning does synthetic fulls without using additional disk space)

Yes, it's ok. I justed posted some example links.

The second request looks like a performance request: transforms are the slowest option possible. I mean, it works fast with REFS / XFS. But with classic filesystems it's painful. The question is, whether you want to go with filesystems that do not support block-cloning in future. Synthetic fulls with block cloning do not cost disk space. That would allow daily incrementals to tape.

Yes, it's also ok. I use ReFS for more than a year and fast cloning is an amazing advantage.

your third request to support multiple LTO generations is absolutely valid. And we need to do that before removing transforms to cover situations like yours. Anton (head of product management) already confirmed your request in his post.

Ok, thanks. This is the biggest show stopper at the moment.

What do you think about differential backups? Until now you don't have it at all. We have some jobs that run a full backups only once or twice a year. In the mean time we need differential backups to have valid backup chains. We would need to use 50 tapes for a restore if we would have to use incremental backups for those jobs.

For File 2 Tape backups there are no GFS Media Pools available. Why?

[HannesK]

@cmaier: could you please post you questions to the tape forum? This topic is about transforms of disk backups. thanks.

[mcz]

Ok, looks like I maybe missed something, will that check again in the future. So to prevent misunderstandins: forward incremental forever should merge the oldest vib into the vbk, correct?

[Gostev]

Correct.

[cmaier]

@cmaier: could you please post you questions to the tape forum? This topic is about transforms of disk backups. thanks.

I already did. 4 years ago.
post204214.html#p204214

[RubinCompServ]

could you maybe explain what the reason technical is?

Doing a Reverse Incremental (or as the original poster suggested a Forward with daily Synthetic Fulls + rollbacks) keeps the Full as the most recent backup, which is beneficial when doing a recovery as most recoveries (in my experience) are for items deleted in the last 24-48 hours. If I have a chain with 30 points and the Full is at the earliest point in the chain, it takes time for Veeam to reassemble the Full and all of the Incrementals, just to get a single file back from yesterday (and that's if Veeam is able to reassemble it which, depending on the size, I've seen fail). So it would seem desirable to keep the Full as the most recent.

Thank you,
David

[HannesK]

@RubinCompServ: nobody wants to remove reverse incremental. It's about transforms. And transforms do for example not even exist for agents and Nutanix.

If you see a measurable difference in doing recoveries for a chain with 30 restore points, then I suggest to investigate that. I could maybe understand that with 500 restore or more points in one chain, but not with 30.

[RubinCompServ]

@HannesK

Perhaps I'm slipping on the terminology, but what does a Reverse Incremental do if not transform the previous backup into an Incremental? I understand that transforms do not exist for agents and Nutanix, but that doesn't negate my point that they have a use case.

I have a job that backs up every 4 hours with a 30 day retention. Everything worked fine (including transforms), until the day that I had to restore from 2 weeks (79 points) prior, and the restore simply wouldn't run. It would chug along for 30 minutes and then time out. Support had me increase the timeout, which just meant it chugged longer and then timed out. Their resolution to the case was, "I'd suggest running a weekly Active Full against this job, because even 42 points can be problematic". Was the engineer mistaken?