Why can't "Move" work with Active Chains?

[YouGotServered]

Lengthy post, so questions are in bold

I know that Veeam v10 can do 2 things:

1. Move inactive chains to object storage
2. Copy active chains to object storage

My initial thoughts were we can't "move" active chains because of API overhead to update the full backup file in S3 storage during something like a merge operation, but doesn't Veeam just update metadata pointers? Also, if we can "copy" a chain up to object storage, then we would be battling the same issue, because copied chains with active VBKs can always be changing, so that wouldn't make sense either.

So my question is: Why can't we "move" an active chain? It seems like there's little functional difference in updating a "copied" chain vs a moved one that just doesn't exist on prem. I guess instead of merging vbk changes and then copying, Veeam would have to make changes directly without doing on-prem first, which doesn't seem like a big issue, but what am I missing? If it was easy, surely Veeam would have allowed that, right?

Here's the use case:
The customer has a daily backup job, plus another backup job that runs once a week that acts as an archival weekly snapshot of the servers. They have one full backup and 51 increments every year. What they wanted to do was have that whole chain in object storage, so that every week when a new incremental is taken, it gets sent up to S3 right after the job and lives with the rest of the chain, therefore using very little on prem storage. The customer has explained that they don't have enough space on their hardware for the normal backup chain, the archival chain, and then a second archival full backup that would need to be taken that would be needed to seal the first chain so that it can be deleted from on prem.

Here's what I feel like you guys are going to tell me to do now:
Use the new GFS retention scheme for normal backups, put that on a SOBR linked to object storage, and then tell that to move the sealed GFS chains up to S3 after creation so that I only have 1-2 full backups on disk, and not 2-3. Is that accurate, or is there a better way?

Is there anything I'm missing? Is there a better way to configure this?

Bonus questions:
Is there a hack or workaround to force the active chain to "move" to S3?
Is this a potential change coming in vNext, or is this the functionality that I need to be happy with long-term?
Can I make the current chain inactive just by a manual right click on the job > active full, or does it need to be some sort of scheduled operation? Not sure why it would matter, but I know there are a few things Veeam treats differently in scheduled vs non-scheduled jobs.
Veeam only moves unique blocks up to S3, right? So if we have a 10TB backup in S3, and our next sealed chain is 10TB with only 1TB of unique data, only 1TB will be uploaded - is that correct?
If we decided that we were done with the old way of doing things with our separate archival backup job and went to GFS, what would be the proper Veeam-supported way to seal it off and have it deleted from on-prem? Should I take all VMs out of the job except one tiny one and do an active full of the tiny VM so that Veeam offloads the previous chain into S3, then manually delete the most recent full restore point of the random VM and disable / delete the job?
Since the chain is already "copied" when the "move" retention kicks in, Veeam is essentially just going to update the local metadata and delete the data blocks in all of the backup files on-prem, which should be a quick operation, right?

Whoever decides to tackle this post and answer these questions will be awarded a gold star and my everlasting gratitude.

PS - this is where I got my info, let me know if I missed something:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

[veremin]

Is there a hack or workaround to force the active chain to "move" to S3?

No

or is this the functionality that I need to be happy with long-term?

This

Can I make the current chain inactive just by a manual right click on the job > active full

Correct

Veeam only moves unique blocks up to S3, right?

Correct

So if we have a 10TB backup in S3, and our next sealed chain is 10TB with only 1TB of unique data, only 1TB will be uploaded - is that correct?

Correct

If we decided that we were done with the old way of doing things with our separate archival backup job and went to GFS, what would be the proper Veeam-supported way to seal it off and have it deleted from on-prem?

Backup Copy Job pointed to SOBR with Capacity Tier move policy enabled. No need to seal or delete restore points from on-prem manually - GFS restore points are selaed by definition, so, as soon as they fall out of the specified restore window (7 days or lower), they will be offloaded object storage, freeing space on on-prem repository.

Since the chain is already "copied" when the "move" retention kicks in, Veeam is essentially just going to update the local metadata and delete the data blocks in all of the backup files on-prem, which should be a quick operation, right?

Correct.

Thanks!

[soncscy]

Hey Cory,

Why can't your client use XFS/ReFS block cloning here and get some spaceless fulls to allow archiving?? I've had quite a few tell me something similar ("We don't want to pay up for local storage"), but block cloning is a thing now and yes, it does mean you'll have to re-do the backup strategy, it puts them in a much safer spot in the long run and that's something that just has to be done sometimes.

[YouGotServered]

@veremin - thanks for those answers but there's some confusion on this question: If we decided that we were done with the old way of doing things with our separate archival backup job and went to GFS, what would be the proper Veeam-supported way to seal it off and have it deleted from on-prem? I'm talking about for our current job that we are using as an "archival" job. I know that Veeam does this automatically when things fall out of retention, but my issue for example is that I have a 10TB chain on site that is copied up to S3 already, how can I "safely and correctly", "seal" that chain so that Veeam correctly cleans up the on-prem storage? To seal the chain, I would need to create another full backup, which is what I want to avoid due to storage concerns. So what should I do? Remove all VMs from the job, add a small 100GB VM, do an "active full" to seal the chain so that the SOBR applies retention to remove the big backup files from the local extent and then manually remove the restore point for the little 100GB VM I added manually?

I just need to know the best way to safely seal and stop an existing chain without creating more backup files on-prem. How do I do that?

@soncscy - I knew someone was going to suggest this, I should have preemptively answered this in my original post We're a long-time Veeam shop and have been following ReFS updates since it was made available in V9.5. It just doesn't seem stable enough for me to start moving my customers to it yet. Yes, issues are rare, but when you have a large customer base, the chances of having an issue are extremely high. The kind of issues that ReFS has generally aren't simple issues either. I do not have the time or resources to dedicate troubleshooting ReFS issues with MS right now (literally, I do not have the time at all). With the amount of customers we have, some sort of large ReFS issue would be an inevitability and I simply cannot handle that right now. I've opted to wait on ReFS for another year or two as MS is still clearly working out the bugs. Every time I think ReFS is good to go, the forums light up with another small nightmare

Trust me, I'm not a technology hold-out. I've often been accused of adopting technologies too soon, but my backup storage is one place I refuse to make a gamble, no matter how small.

[veremin]

Remove all VMs from the job, add a small 100GB VM, do an "active full" to seal the chain so that the SOBR applies retention to remove the big backup files from the local extent and then manually remove the restore point for the little 100GB VM I added manually?

This should do the trick. Make sure to disable deleted VM retention in the job settings. Otherwise, the data belonging to removed VMs will be cleared. Thanks!

[YouGotServered]

Ok, so here's what I've done. I've got a couple of more questions.

After 2 weeks, we have finally finished "copying" 18TB of data to Wasabi with the Capacity Tier. We set capacity tier to "move" backups after 0 days, so essentially, as soon as the chain is sealed. I took all VMs out of the job (ensuring deleted VM retention) was disabled, except one small VM. I right clicked the job, ran an active full. An active full was created with the small 100GB VM, and then my 18TB of other backups were "moved" to S3 in a matter of minutes (Veeam essentially just deleted it) and now all I have on-prem is about 30GB of metadata for my 18TB of backups, and the 100GB backup for the most recent active full for the 1 VM I left in the job. Awesome! I went from 18TB to about 100GB of used space on-prem.

For fun and to see what would happen, I cut and pasted the backup folder (metadata from moved backups and the on-prem VBK of the 100GB VM I just did an active full for) from the SOBR to a different location, rescanned SOBR, and then Veeam automatically downloaded all of the metadata from Object Storage. That's awesome, but expected. What I didn't expect (and I don't know why I didn't expect this) is that Veeam also downloaded just the metadata for the small 100GB VM for which we had an on-site full backup. I guess for some reason, I thought Veeam would want to re-download it since it wasn't supposed to be "sealed", but I'm happy with this behavior as well, so no problem.

So my questions:
1. Instead of doing this workaround with this active full with a single vm and letting Veeam delete the on-prem backups once everything was uploaded to S3, could I just have deleted (or temporarily moved first for safety) the 18TB of backups and then let Veeam redownload the metadata from S3? Is there a problem with this method that I'm not aware of? That seems easier, and then I don't have a 100GB active full hanging around
2. Let's say I lost all of my on-prem performance tier data (hardware failure or attack) - and I was in the middle of a chain that was not sealed. I would fix my storage, rescan the SOBR, Veeam would redownload the metadata. Could I continue with that non-sealed incremental chain, despite the fact it only exists in object storage now, or would Veeam have to re-download the active chain?

• For example: My Sunday VBK and Monday VIB are only in Object Storage - could I continue the chain with a Tuesday VIB on site?

• Per retention policy, if the Monday VIB in object storage needed to merge with the VBK in object storage, I assume pointers would be updated in object storage and the Monday restore point would be removed, correct?

• What would happen to the Tuesday VIB on site when it reaches retention? Would it merge with the VBK in object storage and delete from on-prem?

Thanks!

[veremin]

1) Just to be on the safe side - restore data from restore points stored in object storage repository prior to removing on-prem "dummy" chain. Also, don't forget about deleted VM retention mentioned earlier. Other than, I think, you should be good.

2) What Capacity Tier policy are you talking about? Move or Copy one? If former, I don't understand how part of active backup chain might be present in Capacity Tier.

Thanks!

[YouGotServered]

1. Of course, test restores are key

2. Copy. If my current active chain was deleted somehow (storage failure, for example). But, I answered my own question. After playing around with it, if you delete your active chain, even though it resides in capacity tier, Veeam throws an error when you run the backup and says that you need to run a new active full since everything resides in capacity tier. If I were to re-download my backups, would that suffice, or would I need to actually run an active full? Also, is there even a way to only download a subset of your capacity tier backups, or is it an all-or-nothing approach?

Thanks!

[veremin]

If I were to re-download my backups, would that suffice

Correct.

Also, is there even a way to only download a subset of your capacity tier backups, or is it an all-or-nothing approach?

You can download subsets of backup, using backup properties dialog.

Thanks!

[YouGotServered]

Thank you! As you can tell, I really like to dig into something and understand all of the potential branching situations

[veremin]

Sure, and we're here to make this researching journey easier for you