[IBM COS] Local index is not synchronized with object storage

[markhensler]

Support Case 04173377

We have two SOBRs that have been giving "Local index is not synchronized with object storage" for quite a while. I've been working with Support to resolve these issues. I'm not sure how our local index came to be out of sync, but to be honest, the struggle to resolve the issue has caused me to reconsider the use of COS in our environment.

One of our two SOBRs took over 197 hours to complete a rescan after deleting the ArchiveIndex directory from every extent in the SOBR (per Supports direction). After completing the rescan, I am still getting out-of-sync errors.


What is the proper way to recover from these errors?

How long should it take to rescan ~100 TB in COS?

Does a long rescan time indicate that there may be issues with the COS? network? other?

Does a high change rate create a challenge for the rescan process?

Can a high change rate defeat the rescan process? Do backups need to be disabled during the rescan process?

[dalbertson]

Just curious, do you have Antivirus running on the Veeam or repo server?

[markhensler]

We do. An audit of the AV system revealed no activity under any of the Veeam directories. We even disabled AV for a period of time, but the issue persisted.

At one point, Support suspected something running as PID 4 (system) may be locking files. This is when my ticket was first escalated.

[dalbertson]

Understood. Let me look at the case and see. It may be a known issue that has a workaround. Have you tried recreating the job and trying a fresh test?

[markhensler]

I'm sorry. I don't understand what you mean by recreating the job. This issue is affecting a repository.

I have a second Veeam environment, which is only a few weeks old and is configured to use COS. It is currently working; however, this SOBR is significantly smaller (in the neighborhood of 2% of the size) than the two SOBRs in my troubled environment. This was partly the reason for the questions pertaining to performance impact.

[dalbertson]

What COS are you using?

[markhensler]

Both SOBRs are using the Flex Tier in IBM Cloud.

Our Dallas datacenter uses COS in the US East region.
Our Washington D.C. datacenter uses COS in the US South region.

All networking is alleged to be at least 10Gbps.

[PetrM]

Hello,

I'm not sure that we will be able to find the root cause because our possibility to troubleshoot technical issues over forum posts is limited.

I would suggest to continue working with support team and to request an escalation of the support case because the detailed analysis of debug logs is required to understand the source of the problem:
what exactly slows down the rescan process and how to get rid of the error messages.

Thanks!

[markhensler]

In the last Veeam Community Forums Digest, Anton shared some stats for COS usage by Veeam customers. He said that the largest IBM COS client used 290 TB. We have 106.51 TB in US-East and 187.97 TB in US-South. I suppose that makes us the largest IBM COS consumer and 3rd largest COS consumer overall - from the clients who have opened support tickets so far.

[Gostev]

You quite possible are! Thing is, our big data is anonymized, so we cannot know if some entries belong to the same customer.

The only exception is when you use your own Internet domain name in the S3 service endpoint's DNS address for an on-prem S3-compatible storage.

This actually made me come back to look at the data again, and I did find one such case. For that 305TB ECS deployment from the top 5, I can see two additional Veeam installations consuming 223TB and 120TB respectively from the same ECS clusters, for the total of 648TB of data in object storage.

[Gostev]

On an unrelated note, I just remembered there were scalability complaints on the on-prem version of IBM COS from another customer recently. This may not apply to your case, as you're using cloud storage, and with so much data in it you clearly did not have any performance issues with offloading but I figured I'd mention it here just in case. I don't recommend you make any tweaks discussed there, and instead wait until our support engineers review the rescan logs.

[markhensler]

Thanks for that link. There is some good information in there. I've PM'd the OP for any updates since his last on May 19th.

When we first turned on COS, it took about 6 weeks for the seed to complete. Our retention policy is only 30-days. So, data was aging out of COS before our seed was even finished. To be honest, I was a bit anxious about whether the seed would ever finish, but it did.

I think we were stable for a few weeks before our local index came to be out-of-sync, and now I cannot seem to resolve these pesky "Local index is not synchronized with object storage" errors.

It sounds like my director has managed to arrange a call with both IBM Cloud and Veeam resources. Knock on wood for me.

[nvdwansem]

I'm facing the same issue, I even have more TB in total stored in IBM COS. So keeping an eye on this thread.

Rescan takes about 24 hours, only to get the same error message again:

Code: Select all

15-7-2020 07:34:45 :: Local index is not synchronized with object storage, please rescan the scale-out backup repository  

Out of curiosity are you using IBM COS Classic or IBM COS? Prolly there's no difference in the backend but still interesting to know.

[veremin]

@nvdwansem can you please tell us the support case ID where this issue is reported, if you opened one? We would like to track this. Thanks!

[nvdwansem]

@veremin I haven't created a case for it but I will do so and will update this post as soon as I have a case ID.

[UPDATE] 04284412 is the Case ID, uploaded all my SOBR offload logs and waiting for a response.

[markhensler]

We are using two Flex Tier buckets in IBM COS Classic.

I have a second Veeam environment which is using the Smart Tier in IBM COS. This environment is having no issues; however, it is only about 2% of the size.

[markhensler]

I just got off our 3rd call with both IBM Cloud and Veeam resources. Someone from Veeam's Engineering team was on the call, today.

We had given Veeam a copy of our database, on Friday. I was told that they had stood up our database and had identified issues with it. They are still working on a remedy.

I did mention that there was another client who was experiencing a similar issue and had posted their Veeam case to this thread. If you have not yet given Support a copy of your database, you might ask them if they want it.

[markhensler]

I have an encouraging update, today. I just got off a call with Veeam Support and someone from the engineering team. It seems that we have stumbled upon some bugs.

Bug #1: It sounds like the rescan process assumes all extent paths in a SOBR are unique from each other and it uses these paths when indexing data in SQL. Note that unlike NFS/SMB, Direct Attached Storage paths omit the hostname. Because we have multiple repository servers with identical direct attached storage paths ("D:\Veeam_Backups") in our SOBRs, this caused data corruption.

Bug #2: I didn't catch when this bug comes up, but under some circumstances, the following paths are mistakenly considered to be the same, which compounds the issue from Bug #1.

• D:\Veeam_Backups
• D:\Veeam Backups

Unrelated to the above bugs, Veeam found an unknown error code (2053205313) thrown by IBM Cloud COS. We'll be kicking that one back to IBM Cloud.

[nvdwansem]

Well the suggestion I've got was to update to the 10a RTM which was provided, haven't had time yet to install it because the environment is constantly busy.

Interesting bug, we don't use sub dirs in our environment but the backup job names are folders as well of course and those are definitely the same.

[Gostev]

I must say I am a bit skeptical the bug with matching folder names is real, because it's got to be the reality of our QC labs too, with most things there provisioned and refreshed automatically. So there must be some other dependency, besides just matching folder names.

[markhensler]

Yesterday, I worked with Support to perform the following steps on one of our two erring SOBRs:

1. Stop Veeam Backup Service
2. Stop Veeam Broker Service
3. Backup SQL database
4. Pseudo-SQL: UPDATE BackupRepositories SET Path="D:\VeeamBackups1" WHERE Name="Proxy-3 Share-3a"
5. Pseudo-SQL: UPDATE BackupRepositories SET Path="D:\VeeamBackups2" WHERE Name="Proxy-6 Share-6a"
6. Rename folder on Proxy-3 from "D:\Veeam Backups" to "D:\VeeamBackups1"
7. Rename folder on Proxy-6 from "D:\Veeam Backups" to "D:\VeeamBackups2"
8. Start Veeam Backup Service
9. Start Veeam Broker Service
10. Rescan SOBR

The rescan completed after 13h 43m. The only warnings were due to "locked due to running session".

The rescan finished about 8 hours ago. Right now, there are still 9 SOBR Tiering jobs running. Usually, they have all completed by this time. Maybe it is wishful thinking, but I think there are fewer occurrences of the error, now.

I also see successful offloads of restore points that were created several weeks ago, judging by the datestamp in the VIB filename. So, I'm hopeful that this is evidence that we are now able to offload restore points that we were previously unable to offload.

[Gostev]

Thanks for the update! Glad it's working fine for you now. Unfortunately, so far we could not reproduce this issue internally in an identical environment. It's hard to argue that you somehow ended up in the situation that required to make your backup folder names unique to resolve the issue, but it looks like getting into such situation indeed requires some special sauce that our QC is still trying to find.

[markhensler]

Your engineer did ask how old our SOBRs were and if we had migrated data from previous SOBRs. I think we even re-used extents that were in previous SOBRs. So, we could very well have found some highly unusual circumstance that landed us in this weird state.

[markhensler]

Yesterday, Veeam Support provided me a hotfix for v10 P2. I informed them that I had already upgraded to v10a. They confirmed that the fix had already been included in v10a.

At this point, we have far fewer occurrences of the "local index is not synchronized" error. It seems that with every rescan of our SOBRs, the fequency of this reduces. I believe this is because each rescan produces a number of "locked by running session" warnings, and it may take a few (well-timed) rescans to fully repair our SOBR indices.