Using object storage as a backup target
Post Reply
mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

offload. incomplete restore points

Post by mcz »

...and there seems to be another odd offload-issue: Since yesterday there are very often incomplete restore points on the object storage. Now when I take a look at the offload job, it was successful, but only offloaded some few MB's for each vm. When you manually restart the offload job (or wait for the next run), it will offload the missing blocks for those restore points. But why would it ever be successful when it was or is incomplete? Very strange.

PS: Rescan of the SOBR has been done yesterday, just in case.

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

How did you understand the restore points are incomplete? Was it only offload session statistic (and its tiny number of transferred data) that made you believe so? Were there any new restore points created between automatic and manual offload session execution? Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

First, the amount of offloaded data was too less but the main fact is that in the backup chain properties there was the 'incomplete' status and also icon with the red exclamation mark was present. There were no new restore points being created and also the BCJ did not run during the offload process...

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

...furthermore, I do see those '403 forbidden' very often during offlaods which also lead to incomplete restore points (no surprise):

Image

Now first I thought that this error was thrown by the object storage - but the wasabi-guys checked the bucket and told me everything was allright. They suppose that this 403 was thrown by an veeam-internal API.

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Is this issue also discussed within your existing ticket (04793305)? Or has it been reported as a separate case? Just need to some point to direct Q&A team. Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

I haven't discussed it yet, to be honest, but I can upload the logs to the existing ticket. I wasn't sure if we need an investigation here, it could have been that you guys already know that bug. Vladimir please let me know how you'd like to continue regarding the 403 - thanks!

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

We would appreciate, if you provide us with the debug logs, as incomplete restore points along with 403 error do not look like expected behaviour. Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

I'm uploading the logs, should be on the ftp soon. Within that bundle you should see the offload operation that started on the 7th at 08:23 am. It is displayed successful but the restore points were incomplete and the transferred amount of data is ridiculous.

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Got it, passing the information to QA team, will keep you updated on the results of our findings.

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

I'm uploading a second bundle (last week), so please attach that one too. Should be uploaded in 10-15 minutes. Thanks Vladimir!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

Hi Vladimir,

did QA find anything regarding this 403 forbidden message? Thanks!

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Apologize for a delay, we have been occupied with different activities past few weeks, so haven't had a chance to review the case thoroughly. However, I just got a response from QA team that they will investigate the ticket today or tomorrow - I will update the thread with the results of their findings, as soon as I have it. Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

Hi Vladimir,

still no news from QA? Thanks!

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Hi, Michael, I have been on vacation past few weeks, so will need some time to check the current issue state :) After that I will provide you with the recent information. Thanks!

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

The responsible QA engineer is getting back from vacation on Monday, so we should have some update next week. Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

Vladimir, any update? I've uploaded a new log bundle last week, it contains logs about invalid certificates from the s3 provider, but that can't really be the case...

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Yes, we did proceed with our investigation. This is what happening, based on the debug logs:

- We query Wasabi to get encryption keys (specified in the settings of Capacity Tier)
- Wasabi returns success and then immediately 403 forbidden error
- Offload fails

So we kindly ask you to open a ticket with Wassabi and share its number here, so we can work on the resolution from both sides.

Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

Thanks Vladimir, what an interesting situation! Can you please share one last thought about those strange certificate issues, I mean could those 403's be related to an invalid certificate? After that I'll open the ticket on wasabi-side.

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Sure, we will request new debug logs from support team and check them as well. After that we will reach you again. Thanks!

AlexL
Service Provider
Posts: 70
Liked: 1 time
Joined: Aug 24, 2010 8:55 am
Full Name: Alex
Contact:

Re: offload. incomplete restore points

Post by AlexL »

Has this issue been resolved? We're starting with Wasabi and the initial uploads are working, multiple vbr servers and buckets, 1000+ vm's and few hundred TB of data.
I'm also seeing frequent 403 errors but retries 4 hours later seem to resolve the issue.

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

Hi Alex, the ticket is still open, will share the outcome here once we know what the issue is.

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz » 2 people like this post

Hi Alex, hi Vladimir,

our current situation:

- 403 never appeared again and we didn't change anything (at least I was not aware of it)
- the situation with the incomplete restore points is being handled in case #04980568
- issue with a wrong certificate during offload is being investigated in case #04909110, but probably it's an internal problem

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

Looks like the 403's and the 'storage was not uploaded' issue re-appeared, so I will continue to work on the issues with my engineers. Will post the results here.

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz »

okay, regarding the incomplete restore points: looks like the offload job kicks in before the copy job has written everything to the target file. I guess this is a bug, but it's a hidden one because there a no errors thrown. At the moment this is my assumption, I'm waiting for the response from case #04980568.

In any case: This is for sure not the expected behaviour when you expect your restore points to be copied to the (immutable) object storage as soon as they are created... This is not a criticism, it's just the hint what you 'loose' if you faith the same bug.

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

Let me bring this issue internally again and see if we can find something. Thanks!

veremin
Product Manager
Posts: 18837
Liked: 1904 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: offload. incomplete restore points

Post by veremin »

R&D team is currently heavily occupied working on 11a release, but we will try to find time this or next week and investigate the case. We will keep you updated. Thanks!

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz » 1 person likes this post

Thank you Vladimir, this is a great teamwork

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz » 2 people like this post

Just a quick update on the certificate issue: Looks like that the firewall caused this issue, so it has nothing to do with veeam components.

mcz
Veeam Legend
Posts: 513
Liked: 87 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: offload. incomplete restore points

Post by mcz » 1 person likes this post

Just wannted to update this thread regarding the incomplete restore points. Now we've got an answer: The BCJ kicks in before the backup job started or completed the file merge (forever incremental) and so you've got some files created by the BCJ. The BCJ then gets stopped by the merge (triggered by the backup job) and then the offload kicks in and starts to offload the incomplete BCJ files. Of course that can only lead to incomplete restore points on the object storage. The next time when everything has already finished (merge, bcj), it would of course offload the deltas and that's why all these restore points are complete on the second run.

Took a while to figure that one out, allthough it looks quite simple in the end... Anyways, support told me that there should be some improvements in the upcoming update 11a and that it should be fixed there.

Post Reply

Who is online

Users browsing this forum: No registered users and 16 guests