Do we need to be treating Backblaze S3-compat repos as eventually consistent?

[mikeely]

Was just attempting a SOBR offload to a B2 repo after resyncing (V10 VBR) and was greeted with the following error message:

Index has been recently resynchronized and can only be accessed in 19 hours 39 minutes

My understanding is that Backblaze uses strong consistency throughout so a message like the above is not necessary for their object storage. Is this not correct?

[dalbertson]

Can you open a support case and upload logs?

You are correct that BB does state they are strongly consistent. However this is the opposite of that. It would be helpful to review the logs to see what is happening.

[mikeely]

Gathering logs now, will upload to existing case 04714385 - thanks!

[dalbertson]

Thank you. I’ll take a glance and reply back.

[mikeely]

Largish file, took a minute but it's uploaded now. Thanks again.

[Gostev]

I don't believe logs are needed. We cannot know is the particular S3 storage the user is registering is strong or eventually consistent, so to ensure reliability we can only work with S3-compatible storage assuming eventual consistency.

[mikeely]

@Gostev even when the URL contains clear strings like "backblaze" or similar?

[dalbertson]

Thanks for the details Mike.

As Gostev mention - we treat s3 compatible as eventual consistency just due to the wide array of s3 compatible vendors.

But I do see from your case that there were some other issues going on and it looks like you are in good hands with a rebuild of the archive index.

[Gostev]

@Gostev even when the URL contains clear strings like "backblaze" or similar?

Correct. Our S3-compable object storage support is "generic" and so it had to be designed to work with any S3 storage reliably. It does not currently have "Backblaze awareness" so to speak, or a logic to behave differently depending on the S3 endpoint substring.

[ma7u]

Good morning,
I have had a technical problem in the last few days on veeam object storage backblaze.
I also write to veeam forum, but I suppose is a mix of responsibility between Veeam pipeline and b2 cloud storage.

Last Friday we swap between to NAS because of a hardware upgrade.
I Copied all the backup sets from the old NAS to the new one and then disconnect the old NAS from the scale-out repository, and then connect it to the new one. Rescan the whole repository with no errors.

But since this moment I got an error message on veeam saying that :

The index has been recently resynchronized and can only be accessed in 21 hours and xx minutes

The SOBR offload so stops with warnings or sometimes errors not writing the pieces of information on the cloud.

I tried to sync again the repository and the procedure ends with no errors.

Could you please help me to understand this issue?

Thanks.

[veremin]

Looks like specific actions led to backup server assuming object storage is eventual consistent and further activation of offload timeout mechanism. You can open a ticket with our support team to confirm that there is nothing else going on and that after timeout the offload will work as expected.

Thanks!

[ma7u]

i confirm that after waiting the period reported in the warning the storage start again to work fine.

The index has been recently resynchronized and can only be accessed in 21 hours and 18 minutes

no idea of what happened, but the situation is now restored ad before.

[veremin]

The backup server seems to have identified that local version of index differs from the version of index stored in object storage and triggered the said timeout mechanism. The reason why backup server decided to do so might be found in the debug logs, so if you are still interested in digging deeper into the root cause, kindly reach our support team.

However, it's not required, as after the timeout passes, everything should start working correctly, which looks to have happened in your case.

Thanks!