Discussions related to using object storage as a backup target.
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

veeam does not always automatically offload fulls

Post by mcz »

Hi everyone,

I'm currenly using a BCJ in immediate copy mode and I've used the "other" option in the past, both were configured as GFS's. For some reason, veeam would suddenly start to NOT automatically offload sealed fulls, e.g. the weekly fulls (W is seen at the retention) and then my repository runs silently out of disk space. You can then manually trigger the offload when you select "move to capacity tier" in the backup properties but I don't get it why it wouldn't offload these vbk's because the offload job is offloading/copying all the time. When I manually force the offload, it would do it immediately and then it only would have to update the metadata because all the blocks were already offloaded.

Is it a known bug? I can't imagine that it's expected because I've set the correct thresholds on the SOBR side.

Thanks!
jmmarton
Veeam Software
Posts: 2092
Liked: 309 times
Joined: Nov 17, 2015 2:38 am
Full Name: Joe Marton
Location: Chicago, IL
Contact:

Re: veeam does not always automatically offload fulls

Post by jmmarton »

If you go to the history section, what's the status of the offload process? Are there errors?

Joe
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

No, no errors during/after offload. The funny thing is that if you trigger tiering job, it won't offload. You have to use the option I've described above...
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

What is operational restore window specified? And how old those GFS restore points are? Thanks!
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam does not always automatically offload fulls

Post by Gostev »

Considering that Michael is pretty good with the product and it worked for him before, let's not waste anyone's time guessing most obvious issues, and rather have support review the debug logs :D
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

We were trying to confirm the described issue, while asking the most obvious question, which is never a bad thing to do :)

Besides the questions above, by "setting correct threshold" don't you refer to "Move oldest backup files sooner if scale-out backup repository is reaching capacity" setting by any chance?

Thanks!
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

veremin wrote: May 27, 2020 4:21 pm What is operational restore window specified? And how old those GFS restore points are? Thanks!
Here are my sobr settings, operational window is enabled all the time:

Image

I'm currently waiting until BCJ has finished because I've opened a ticket because of a new bug in v10 (failure during Transform.Patch) and then I'll update you. But for now it looks like this:

Image

In the past, when I had no issues with BCJ, I had the same picture. There was a regular full and beneath the vbk marked as W or Q and no offload has taken place - only if you forced it manually which is strange to me.
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

But the selected restore point (31.05.2020) has not fallen out of the specified operational restore window (60 days), so, the local copy still stays on-prem. Or you're saying that currently there is no copy of this restore point in object storage, even though the copy policy is configured? Thanks!
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

Yeah, correct it hasn't fallen out of operational restore, but what I forgot to mention is the fact that it should offload because our repository is used more than 10 % - see screenshot above. That's why I expect the offload. I have also decreased the figures in the past (e.g. set operational restore to 1) and it also didn't offload...
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

Didn't you wait for 4 hours or so? The move offload policy runs every 4 hours, so, might be that you simply haven't waited long enough.

The bug with override police (offload until used space is below) is confirmed and will be fixed in the upcoming product update (don't confuse with cumulative patches).

So, thank for raising this; much appreciated!
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

So to not get you wrong: There is a confirmed bug, which causes veeam to ignore the override policy when more space is used than specified? This would be a good explanation in my case...
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

When this thread started, I asked QA team to confirm the described behaviour.

During our investigation we found that if both copy and move policies are configured, then, override policy does not work correctly - it leaves local blocks intact, instead of removing them.

Once confirmed, the bug was scheduled for fixing in the next product update.

So, once again on the behalf of R&D team thank you for raising this issue.

That's said, please, monitor whether the decreased operational window results in offloaded restore points. If that does not happen, please, open a support ticket and post its number here, and we will follow the investigation.

Thanks!
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz » 1 person likes this post

Thanks Vladimir, fantastic collaboration on this forum! Looks like I'm kinda S3-bug-hunter ;)
Will let you know the results the next time.
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

Looks like I'm kinda S3-bug-hunter
And we appreciate this quality of yours :)
avs262
Novice
Posts: 8
Liked: 1 time
Joined: May 26, 2020 7:44 pm
Full Name: Jeff
Contact:

Re: veeam does not always automatically offload fulls

Post by avs262 »

veremin wrote: Jun 04, 2020 4:17 pm When this thread started, I asked QA team to confirm the described behaviour.

During our investigation we found that if both copy and move policies are configured, then, override policy does not work correctly - it leaves local blocks intact, instead of removing them.

Once confirmed, the bug was scheduled for fixing in the next product update.

So, once again on the behalf of R&D team thank you for raising this issue.

That's said, please, monitor whether the decreased operational window results in offloaded restore points. If that does not happen, please, open a support ticket and post its number here, and we will follow the investigation.

Thanks!
Is there any way to get a patch early? I have the "Offload until used space is below" option checked (but do not have the "Copy backups to object storage" enabled) but the offload job will never move anything so it's completely run out of disk space on the scaleout repo.
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam does not always automatically offload fulls

Post by Gostev »

You should open a support case and request it.
avs262
Novice
Posts: 8
Liked: 1 time
Joined: May 26, 2020 7:44 pm
Full Name: Jeff
Contact:

Re: veeam does not always automatically offload fulls

Post by avs262 » 1 person likes this post

I've got a case open and linked to this post but they said it was hard to tell which bug this post is referring to. Here's the case #: 04216148
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

Jeff,

are you able to force the offload manually as I've described above?
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

avs262 wrote: Jun 07, 2020 10:49 pm I've got a case open and linked to this post but they said it was hard to tell which bug this post is referring to. Here's the case #: 04216148
The bug ID is 236282, the fix will be added to next product update.

For now, kindly, decrease operational restore window, so, restore points are moved, when they fall out of the specified time frame, not when the override policy kicks in.

Thanks!
avs262
Novice
Posts: 8
Liked: 1 time
Joined: May 26, 2020 7:44 pm
Full Name: Jeff
Contact:

Re: veeam does not always automatically offload fulls

Post by avs262 »

Michael,

I'm unable to forcefully offload backups - I don't have the option when viewing recovery points.

veremin, I've set the operational restore window to '0' to move backups immediately after they age out. When I setup this Veeam instance I knew that the scaleout repo had just enough storage to hold the required amount of recovery points and I was counting on the "offload until used space is below" feature to keep backups running successfully. Any idea when the next release is scheduled for?
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

Jeff,

in the backup properties, you see retentions like "Q", "M" and so on, not "R", correct? Veeam would only offload sealed backup chains, have you verified that?
avs262
Novice
Posts: 8
Liked: 1 time
Joined: May 26, 2020 7:44 pm
Full Name: Jeff
Contact:

Re: veeam does not always automatically offload fulls

Post by avs262 »

I'm using forever incremental without synthetic fulls. It must be my misunderstanding that the "offload until used space is below" option would move any backup file to return the scaleout repository to the defined threshold?

Based on this doc - https://helpcenter.veeam.com/docs/backu ... ml?ver=100 I've adjusted one of the jobs to produce synthetic fulls and I'm running that now.
avs262
Novice
Posts: 8
Liked: 1 time
Joined: May 26, 2020 7:44 pm
Full Name: Jeff
Contact:

Re: veeam does not always automatically offload fulls

Post by avs262 »

After adjusting jobs to generate periodic synthetic fulls and letting them run overnight has resulted in Veeam offloading backups as expected. Thanks for the help!
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

It must be my misunderstanding that the "offload until used space is below" option would move any backup file to return the scaleout repository to the defined threshold?
Override threshold (being an additional option within move policy) offloads only sealed restore points. So, it does not work for active restore points of forward forever incremental chain. Thanks!
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

Hey veeam-guys,

for a very long time everything was running fine until now: I'm running on version 10.0.1.4854 and suddenly veeam doesn't offload certain weekly fulls... Retention on SOBR-level is set to 0 days and offload if space is used more than 10 % (which would always be the case, btw).

Now the interesting part is that I can't even offlaod by forcing it manually. For instance, already offloaded weekly fulls have this options:

Image

Now my VBK to offload doesn't even give me any option:

Image

Now, is this the same bug or a new one? Thanks!
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam does not always automatically offload fulls

Post by Gostev »

You cannot offload backups from the active (latest) backup chain, so all looks good on the screenshot since what you have selected is the latest full backup.
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz »

Yeah Anton, you're right, but if you look again, I'm not talking about the active chain. This would be the "R" file, but I'm talking about the "W" and this is completely independent.
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: veeam does not always automatically offload fulls

Post by Gostev »

OK, then you should investigate this with support. May be it is already placed in the Move policy processing queue and is just sitting there waiting for backup infrastructure resources, which is why you cannot do this again.
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: veeam does not always automatically offload fulls

Post by veremin »

The restore points that are currently processed or placed in the processing queue do not have these buttons greyed out, so, the behaviour you're seeing is unexpected.

Kindly, open a support ticket and provide us with its number - QA team is interested in seeing the debug logs.

Thanks!
mcz
Veeam Legend
Posts: 824
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: veeam does not always automatically offload fulls

Post by mcz » 1 person likes this post

case # is 04490442. I'm still hunting bugs ;)
Post Reply

Who is online

Users browsing this forum: rhys.hammond and 5 guests