veeam does not always automatically offload fulls

[mcz]

Hi everyone,

I'm currenly using a BCJ in immediate copy mode and I've used the "other" option in the past, both were configured as GFS's. For some reason, veeam would suddenly start to NOT automatically offload sealed fulls, e.g. the weekly fulls (W is seen at the retention) and then my repository runs silently out of disk space. You can then manually trigger the offload when you select "move to capacity tier" in the backup properties but I don't get it why it wouldn't offload these vbk's because the offload job is offloading/copying all the time. When I manually force the offload, it would do it immediately and then it only would have to update the metadata because all the blocks were already offloaded.

Is it a known bug? I can't imagine that it's expected because I've set the correct thresholds on the SOBR side.

Thanks!

[jmmarton]

If you go to the history section, what's the status of the offload process? Are there errors?

Joe

[mcz]

No, no errors during/after offload. The funny thing is that if you trigger tiering job, it won't offload. You have to use the option I've described above...

[veremin]

What is operational restore window specified? And how old those GFS restore points are? Thanks!

[Gostev]

Considering that Michael is pretty good with the product and it worked for him before, let's not waste anyone's time guessing most obvious issues, and rather have support review the debug logs

[veremin]

We were trying to confirm the described issue, while asking the most obvious question, which is never a bad thing to do

Besides the questions above, by "setting correct threshold" don't you refer to "Move oldest backup files sooner if scale-out backup repository is reaching capacity" setting by any chance?

Thanks!

[mcz]

What is operational restore window specified? And how old those GFS restore points are? Thanks!

Here are my sobr settings, operational window is enabled all the time:



I'm currently waiting until BCJ has finished because I've opened a ticket because of a new bug in v10 (failure during Transform.Patch) and then I'll update you. But for now it looks like this:



In the past, when I had no issues with BCJ, I had the same picture. There was a regular full and beneath the vbk marked as W or Q and no offload has taken place - only if you forced it manually which is strange to me.

[veremin]

But the selected restore point (31.05.2020) has not fallen out of the specified operational restore window (60 days), so, the local copy still stays on-prem. Or you're saying that currently there is no copy of this restore point in object storage, even though the copy policy is configured? Thanks!

[mcz]

Yeah, correct it hasn't fallen out of operational restore, but what I forgot to mention is the fact that it should offload because our repository is used more than 10 % - see screenshot above. That's why I expect the offload. I have also decreased the figures in the past (e.g. set operational restore to 1) and it also didn't offload...

[veremin]

Didn't you wait for 4 hours or so? The move offload policy runs every 4 hours, so, might be that you simply haven't waited long enough.

The bug with override police (offload until used space is below) is confirmed and will be fixed in the upcoming product update (don't confuse with cumulative patches).

So, thank for raising this; much appreciated!

[mcz]

So to not get you wrong: There is a confirmed bug, which causes veeam to ignore the override policy when more space is used than specified? This would be a good explanation in my case...

[veremin]

When this thread started, I asked QA team to confirm the described behaviour.

During our investigation we found that if both copy and move policies are configured, then, override policy does not work correctly - it leaves local blocks intact, instead of removing them.

Once confirmed, the bug was scheduled for fixing in the next product update.

So, once again on the behalf of R&D team thank you for raising this issue.

That's said, please, monitor whether the decreased operational window results in offloaded restore points. If that does not happen, please, open a support ticket and post its number here, and we will follow the investigation.

Thanks!

[mcz]

Thanks Vladimir, fantastic collaboration on this forum! Looks like I'm kinda S3-bug-hunter
Will let you know the results the next time.

[veremin]

Looks like I'm kinda S3-bug-hunter

And we appreciate this quality of yours

[avs262]

When this thread started, I asked QA team to confirm the described behaviour.

During our investigation we found that if both copy and move policies are configured, then, override policy does not work correctly - it leaves local blocks intact, instead of removing them.

Once confirmed, the bug was scheduled for fixing in the next product update.

So, once again on the behalf of R&D team thank you for raising this issue.

That's said, please, monitor whether the decreased operational window results in offloaded restore points. If that does not happen, please, open a support ticket and post its number here, and we will follow the investigation.

Thanks!

Is there any way to get a patch early? I have the "Offload until used space is below" option checked (but do not have the "Copy backups to object storage" enabled) but the offload job will never move anything so it's completely run out of disk space on the scaleout repo.

[Gostev]

You should open a support case and request it.

[avs262]

I've got a case open and linked to this post but they said it was hard to tell which bug this post is referring to. Here's the case #: 04216148

[mcz]

Jeff,

are you able to force the offload manually as I've described above?

[veremin]

I've got a case open and linked to this post but they said it was hard to tell which bug this post is referring to. Here's the case #: 04216148

The bug ID is 236282, the fix will be added to next product update.

For now, kindly, decrease operational restore window, so, restore points are moved, when they fall out of the specified time frame, not when the override policy kicks in.

Thanks!

[avs262]

Michael,

I'm unable to forcefully offload backups - I don't have the option when viewing recovery points.

veremin, I've set the operational restore window to '0' to move backups immediately after they age out. When I setup this Veeam instance I knew that the scaleout repo had just enough storage to hold the required amount of recovery points and I was counting on the "offload until used space is below" feature to keep backups running successfully. Any idea when the next release is scheduled for?

[mcz]

Jeff,

in the backup properties, you see retentions like "Q", "M" and so on, not "R", correct? Veeam would only offload sealed backup chains, have you verified that?

[avs262]

I'm using forever incremental without synthetic fulls. It must be my misunderstanding that the "offload until used space is below" option would move any backup file to return the scaleout repository to the defined threshold?

Based on this doc - https://helpcenter.veeam.com/docs/backu ... ml?ver=100 I've adjusted one of the jobs to produce synthetic fulls and I'm running that now.

[avs262]

After adjusting jobs to generate periodic synthetic fulls and letting them run overnight has resulted in Veeam offloading backups as expected. Thanks for the help!

[veremin]

It must be my misunderstanding that the "offload until used space is below" option would move any backup file to return the scaleout repository to the defined threshold?

Override threshold (being an additional option within move policy) offloads only sealed restore points. So, it does not work for active restore points of forward forever incremental chain. Thanks!

[mcz]

Hey veeam-guys,

for a very long time everything was running fine until now: I'm running on version 10.0.1.4854 and suddenly veeam doesn't offload certain weekly fulls... Retention on SOBR-level is set to 0 days and offload if space is used more than 10 % (which would always be the case, btw).

Now the interesting part is that I can't even offlaod by forcing it manually. For instance, already offloaded weekly fulls have this options:



Now my VBK to offload doesn't even give me any option:



Now, is this the same bug or a new one? Thanks!

[Gostev]

You cannot offload backups from the active (latest) backup chain, so all looks good on the screenshot since what you have selected is the latest full backup.

[mcz]

Yeah Anton, you're right, but if you look again, I'm not talking about the active chain. This would be the "R" file, but I'm talking about the "W" and this is completely independent.

[Gostev]

OK, then you should investigate this with support. May be it is already placed in the Move policy processing queue and is just sitting there waiting for backup infrastructure resources, which is why you cannot do this again.

[veremin]

The restore points that are currently processed or placed in the processing queue do not have these buttons greyed out, so, the behaviour you're seeing is unexpected.

Kindly, open a support ticket and provide us with its number - QA team is interested in seeing the debug logs.

Thanks!

[mcz]

case # is 04490442. I'm still hunting bugs