Oracle RMAN Offload to AWS Errors

[dmtinklenb]

We suddenly started seeing these errors in early July with the offload to AWS for our Oracle RMAN Plugin backups. I've opened a separate case for each error because the severity is different for the two errors. I'm bringing these cases to the attention of the forums because I'm not getting anywhere with support, even though I've asked for escalation.

Case# 04953060
This is a severity 2 case because offloads are not working at all with this error:

Attempt to overwrite existing non-sparse block version. Block ID: [472490], new version: [XfYaRcXh.JVohKlLTvDlXSma9NiMWD1o], existing version: [xPO6LbEB5xRjJxzehRuD01KPn3nqMmxN]


This error is only happening on one of our Oracle databases servers and I've asked for escalation, but nothing is happening.

Case# 04918053

This is a severity 3 case because offloads are working but give this warning:

Checkpoint cleanup failed Details: Unable to decrease checkpoints counter for object [plv099imgdb.xxxx.com_Oracle_backup__xxxx_RMAN_1_.vacm:149]

This warning message is happening on 9 of our 20 Oracle database servers.


Has anyone else out there seen these errors and have a possible resolution?

[Andreas Neufert]

Hello, thanks for sharing. I have escalated your support case. I think support is the best to help here.

You can always use the "contact manager" link in the support case system when you think the support ticket is not progressing well to escalate the case.

[dmtinklenb]

Yes, I've done that, and I've been contacted by a manager and his response was to upload logs again, which I did, but have gotten no response and no escalation.

[PetrM]

Hello,

Just checked on our ticket tracking system: 04918053 has already been escalated and one of our senior engineers should contact you soon. We'll keep an eye on it to make sure that it's on the right track. However, 04953060 has been closed after moving backup to another repository. Do you encounter the same issue again?

Thanks!

[dmtinklenb]

Yes, moving the backup to a new repository resolved the issue for a few days, but then started again this week. At least now it is actually copying the files to AWS but then fails with this error:

Code: Select all

8/18/2021 1:35:59 PM :: Processing plv050ebsdb.xxxx.com Oracle backup (XXX RMAN 2) Error: REST API error: 'DeleteMultipleObjects request failed to delete object [Veeam/Archive/RMAN-2/ba04dbc4-6006-4197-8412-b2f13da95779/00000000-0000-0000-0000-000000000000/blocks/384e39644295b9b689d8bd1e1da8b16a/300.65fcac88b3e50472232bfa6fc986110e.00000000000000000000000000000000.blk] and [546633] others', error code: 500
Exception from server: REST API error: 'DeleteMultipleObjects request failed to delete object [Veeam/Archive/RMAN-2/

[PetrM]

Ok, thanks for information. Let's wait for the result of analysis from our support team, I believe they will be able to narrow down the issue.

Thanks!

[dmtinklenb]

This case has been open for 45 days and we are nowhere near a solution. The last response I received was 4 days ago saying they have "opened a thread with our internal resource team", which to me means they have no clue as to what is causing the problem and no plan on how to get this resolved.

Right now, the AWS offload of our largest and most critical database is failing, leaving our entire organization vulnerable and at risk for ransomware. This is completely unacceptable and extremely frustrating. My manager is breathing down my neck for answers and the only thing I can tell him each day is, "no response from Veeam."

The biggest frustration is that lack of communication. This case was escalated on Aug. 18, but I didn't receive any communication from an escalation engineer until Aug. 26. What's the point of escalating the case if it doesn't change the priority within Veeam Support?

[PetrM]

Hello,

Thanks for sharing this information with us, I fully realize the severity of this problem. The escalation literally means the transfer of the support case to the team of senior engineers who dive deeper into a problem and carry out technical analysis on the lower level. However, such sophisticated and complex issues require significant time for troubleshooting very often. As far as I see, our engineers have quite an active discussion of the issue internally but apparently they need more time to complete RCA and come up with the results.

Anyway, I've just asked support team leaders to look for more available resources in order to accelerate the research and assign the highest possible priority to it.

Thanks!

[dmtinklenb]

I received an email from support stating this:

The issue that you are experiencing seems to be with the Versions on the S3 side. Only Amazon S3 generates version IDs, and they cannot be edited.

Is this only an issue with AWS S3 storage? If we were to switch to Wasabi immutable S3 storage, would we run into the same issue or does Veeam handle Wasabi's S3 storage better?

[PetrM]

It's worth clarifying all details regarding this statement directly with your support engineer. So far, I'd prefer to avoid making any conclusions unless the root cause is crystal clear. The support case is being reviewed by our Tier-3 engineers, let's wait for what they can figure out.

Thanks!

[AlexL]

was this ever resolved? we're starting to see these errors also, using wasabi
Case #05051813 (Severity 2) — S3 storage issue: DeleteMultipleObjects request failed to delete object

also got the non-sparse error, have not opened a separate case for that yet

and it started with a range of issues that got escalated to Severity 1 Case #05028983, still open, which is mostly ok now but still some jobs with errors, mainly "Sequence contains more than one element"

[PetrM]

Hi Alex,

The problem is still being researched by our RnD, we're not ready to provide the permanent solution yet.

The case 05051813 has been created yesterday, I believe that our support team will contact you soon to start investigation. I'll ask our support folks to check whether you experience the same as the one reported above or not and to look for potential workarounds. The case 04918053 is under investigation already, let's wait for the conclusion from our engineers.

Thanks!

[dmtinklenb]

Until there is a fix for this issue, our process has been as soon as we start seeing the errors we create a new folder in our bucket or a new bucket, change the offload to use the new folder or bucket, resync, and then the offloads work great for 14-18 days and then we start getting the errors again.

We are currently backing up up 20 Oracle database servers using the RMAN plugin. We only get this offload error on one database server which contains our largest database (6+ TB).

[dmtinklenb]

Is there an update on this issue other than "RnD is researching the issue?" This case has been open for two months and I am at my wit's end with this problem. Our work around of creating a new bucket is only working for a few days before we start getting these errors and the offloads fail again.

[PetrM]

Hello,

I fully understand how painful is to stay without permanent resolution for such a long time but the problem is out of scope of our known issues and RCA requires a lot of engineering efforts. Basically, this is the main reason of delay. Our developers work on it and we'll be keeping you posted on the results of investigation and will inform you as soon as we have more information. Many thanks for understanding!

Thanks!

[gtelnet]

I started receiving the identical "Sequence contains more than one element" error during offloads and sobr rescans on Oct 20, the day I upgraded the bnr to 11a. Using Backblaze not AWS, so issue does not seem to be specific to any S3 provider. Case # 05100495

[PetrM]

Hi Greg,

It looks like one more new issue, so far I don't see any correlation with the ones discussed above. Anyway, let's wait for the results of debug logs analysis from our support engineers.

Thanks!