v12 SOBR Tiering change?

[TheJourney]

Hello all, wondering if something changed in v12 but I cannot find any documentation etc about this change so here goes..

We are seeing the backup chains on performance tiers get broken by the offload jobs, they seem to not honor the chain on the performance tier and instead just use however many restore point etc you set for the offload on the SOBR. So we get left with incrementals on the performance tier that are not usable unless the full and other incrementals are downloaded from the capacity tier. We are seeing this on OBJ > OBJ SOBRs as well as our older NFS > OBJ SOBRs.

I don’t recall ever seeing the chains broken on the Performance tier on v11. I recall v11 honoring the performance tier chains and not removing restore points until they were able to be removed without breaking the Chain. Am I wrong? I have no way of validating on my end as we updated to v12 months ago.

If this is the way SOBR Tiering functions can someone point me to the documentation or change log for V12? I don’t see any logic for leaving incrementals in the performance tier that cannot be used without the other backups from the capacity tier. There is no point to leave those on the performance tier, as they only take up space.

I think we will need to re-architect our environment to use copy jobs to get the functionality that we are looking for. I’m looking for a solid response from Veeam experts on this before we move to copy jobs.

Case #06230600

Cheers.

[Mildur]

Hello Sam

That sounds wrong. The offload process shouldn't leave "orphaned incremental" on the performance tier.
Please continue to work with our customer support team. I can see in the case, that the engineer will soon analyze the provided log files. I will follow the case from our side.

I think we will need to re-architect our environment to use copy jobs to get the functionality that we are looking for. I’m looking for a solid response from Veeam experts on this before we move to copy jobs.

I suggest to wait a few days for a conclusion in your case before changing your architecture.

Best,
Fabian

[TheJourney]

Bump?

[Gostev]

Hello Sam, you have the answer from the product team above. This is the "solid response from Veeam experts" you were looking for. Not sure what else do you want to hear and why are you bumping this thread.

To reiterate: this is not normal so you should work with support to get to the bottom of the issue, which is likely environment-specific (otherwise I guess it would have been reported by many other users by now, after nearly half a million of V12 installs). We also do not recommend you to re-architect until the issue is understood, as the reason may be misbehaving storage etc.

Thanks!

[TheJourney]

Gostev, thank you for the reply. I'm trying to get folks to believe me. I have other supposed experts telling me that's how v11 worked, your own support staff wondering if this is expected behavior or not. The case is almost a month old, I'm on my 2nd or 3rd support person. I've done webex's to explain the situation, gathered logs a few times etc. I'm not sure how long im expected to wait. You replying to this thread will help, maybe those folks will believe me.

As far as storage goes, we have a Pure Flashblade's running NFS or S3 for performance tier, Wasabi S3 for capacity. I can engage either of those vendors if needed.

PS I cant find any documentation around how to setup OBJ > OBJ Sobr tiering? It seems unless you setup GFS (synthetic fulls) or weekly active fulls nothing gets removed from Performance tier at all, until you hit your job retention.

[TheJourney]

@Gostev

Support closed my case with this is expected behavior. Seems odd to leave unusable incremental backups on performance tier. What would be the reason for that? To take up space?

We switched to Copy Jobs so we could have more control over what stays on-prem and what does not.

Supports response:

Hi Sam,

Apologies for the slow response.
As mentioned in last email, the behavior from v11 to v12 did not change and it working as intended after testing:

Veeam Backup & Replication will transfer to the capacity extent only those restore points that belong to inactive backup chains. To ensure a backup chain is inactive, Veeam Backup & Replication verifies its state. This does not apply to the copy policy: all newly created restore points are copied immediately.

Here is the User Guide information about Backup Chain Detention on v11:
https://helpcenter.veeam.com/archive/ba ... +detection
AND
v12 - https://helpcenter.veeam.com/docs/backu ... on&ver=120

[Gostev]

Could you clarify where exactly did they say that "leaving unusable incremental backups on Performance Tier" is "an expected behavior"?

I only see two statements:
1. The statement that "the behavior from v11 to v12 did not change" which is true.
2. The correct explanations of how Move and Copy policies are supposed to work.

But where did they say that its normal for inactive chains that have been moved to the Capacity Tier to leave behind some of their increments on the Performance Tier?

If they did not actually say this, then please keep the support case open to ensure this is investigated properly (or escalated into R&D if needed). Currently it seems there's some basic misunderstanding between you and Support. Perhaps your support engineers did not actually see the issue you think you have, which makes them say the product is working as intended.

May be you could simply show them an incremental backup file on the Performance Tier that should not be there. Then it will be impossible for them to say this is "working as intended" if some old increment still sits on the Performance Tier after the entire inactive chain it belongs to has been moved to the Capacity Tier. Because the whole idea of the Move policy is to free up the Performance Tier from the oldest backups, leaving nothing behind.

Thanks

[TheJourney]

They closed the case, you can review everything in there. Should have screenshots, logs, everything else I was expected to provide. I did multiple webexes, trying to explain the issue, pretty sure they understood. I had calls with my account team and SE, which were him telling me that's what its supposed to do, and me arguing against that. That's when I posted here to begin with. I would have to tell support go read the post, see what your experts are saying. This was amazingly frustrating and a continuous waste of time. The case was open for more than a month with very little traction, we had to move to copy jobs in order to stay protected.

In order to keep the case going I would have to recreate the jobs and SOBRs.

[veremin]

We verified internally that there were no changes in Capacity Tier move policy between 11a and 12 releases.

As before, the move policy operates with inactive backup chains and as soon as a restore point falls out of a specified operational restore window, the policy starts moving it to capacity extent. There is no additional logic that keeps a full backup till all dependent increments also go beyond the operational restore window.

We were not able to confirm that these increments are staying forever on performance extents in your case. It seems that they were also moved to capacity tier at proper time.

An inactive backup chain part of which is located on performance tier and part on capacity tier is still useful, cause all restore operations work in these scenarios without any issues. Plus, you can download restore points from capacity tier back to performance one at any time.

Thanks!

[Gostev]

Thanks to further back and forth with Vladimir I finally understood the issue. Even if nothing changed in V12, I'm now convinced its a design bug in our offload functionality. Let me try to explain this to the team now

[TheJourney]

Thank you Gostev.