Discussions related to using object storage as a backup target.
Post Reply
mikeely
Expert
Posts: 224
Liked: 69 times
Joined: Nov 07, 2016 7:39 pm
Full Name: Mike Ely
Contact:

Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by mikeely »

Was just attempting a SOBR offload to a B2 repo after resyncing (V10 VBR) and was greeted with the following error message:
Index has been recently resynchronized and can only be accessed in 19 hours 39 minutes
My understanding is that Backblaze uses strong consistency throughout so a message like the above is not necessary for their object storage. Is this not correct?
'If you truly love Veeam, then you should not let us do this :D' --Gostev, in a particularly Blazing Saddles moment
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by dalbertson »

Can you open a support case and upload logs?

You are correct that BB does state they are strongly consistent. However this is the opposite of that. It would be helpful to review the logs to see what is happening.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
mikeely
Expert
Posts: 224
Liked: 69 times
Joined: Nov 07, 2016 7:39 pm
Full Name: Mike Ely
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by mikeely »

Gathering logs now, will upload to existing case 04714385 - thanks!
'If you truly love Veeam, then you should not let us do this :D' --Gostev, in a particularly Blazing Saddles moment
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by dalbertson »

Thank you. I’ll take a glance and reply back.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
mikeely
Expert
Posts: 224
Liked: 69 times
Joined: Nov 07, 2016 7:39 pm
Full Name: Mike Ely
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by mikeely »

Largish file, took a minute but it's uploaded now. Thanks again.
'If you truly love Veeam, then you should not let us do this :D' --Gostev, in a particularly Blazing Saddles moment
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by Gostev »

I don't believe logs are needed. We cannot know is the particular S3 storage the user is registering is strong or eventually consistent, so to ensure reliability we can only work with S3-compatible storage assuming eventual consistency.
mikeely
Expert
Posts: 224
Liked: 69 times
Joined: Nov 07, 2016 7:39 pm
Full Name: Mike Ely
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by mikeely »

@Gostev even when the URL contains clear strings like "backblaze" or similar?
'If you truly love Veeam, then you should not let us do this :D' --Gostev, in a particularly Blazing Saddles moment
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by dalbertson »

Thanks for the details Mike.

As Gostev mention - we treat s3 compatible as eventual consistency just due to the wide array of s3 compatible vendors.

But I do see from your case that there were some other issues going on and it looks like you are in good hands with a rebuild of the archive index.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by Gostev »

mikeely wrote: Apr 01, 2021 12:59 am@Gostev even when the URL contains clear strings like "backblaze" or similar?
Correct. Our S3-compable object storage support is "generic" and so it had to be designed to work with any S3 storage reliably. It does not currently have "Backblaze awareness" so to speak, or a logic to behave differently depending on the S3 endpoint substring.
ma7u
Novice
Posts: 7
Liked: 4 times
Joined: Mar 30, 2022 10:59 am
Full Name: ma7u
Contact:

[MERGED] swap between NAS object storage index corruption

Post by ma7u »

Good morning,
I have had a technical problem in the last few days on veeam object storage backblaze.
I also write to veeam forum, but I suppose is a mix of responsibility between Veeam pipeline and b2 cloud storage.

Last Friday we swap between to NAS because of a hardware upgrade.
I Copied all the backup sets from the old NAS to the new one and then disconnect the old NAS from the scale-out repository, and then connect it to the new one. Rescan the whole repository with no errors.

But since this moment I got an error message on veeam saying that :

The index has been recently resynchronized and can only be accessed in 21 hours and xx minutes

The SOBR offload so stops with warnings or sometimes errors not writing the pieces of information on the cloud.

I tried to sync again the repository and the procedure ends with no errors.

Could you please help me to understand this issue?

Thanks.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by veremin »

Looks like specific actions led to backup server assuming object storage is eventual consistent and further activation of offload timeout mechanism. You can open a ticket with our support team to confirm that there is nothing else going on and that after timeout the offload will work as expected.

Thanks!
ma7u
Novice
Posts: 7
Liked: 4 times
Joined: Mar 30, 2022 10:59 am
Full Name: ma7u
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by ma7u »

i confirm that after waiting the period reported in the warning the storage start again to work fine.

The index has been recently resynchronized and can only be accessed in 21 hours and 18 minutes

no idea of what happened, but the situation is now restored ad before.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Do we need to be treating Backblaze S3-compat repos as eventually consistent?

Post by veremin »

The backup server seems to have identified that local version of index differs from the version of index stored in object storage and triggered the said timeout mechanism. The reason why backup server decided to do so might be found in the debug logs, so if you are still interested in digging deeper into the root cause, kindly reach our support team.

However, it's not required, as after the timeout passes, everything should start working correctly, which looks to have happened in your case.

Thanks!
Post Reply

Who is online

Users browsing this forum: kavaa and 11 guests