Backup Copy to Backblaze B2 storage hangs 'failed to process item is locked by running session'

[juanmachado]

Backup Copy to Backblaze B2 storage hangs 'failed to process item is locked by running session'

We are running Veeam 12.1.0.2131.

Is anybody having this issue? It started early last week. We have a case open with Veeam but no luck so far.

We can individually push a new backup to B2 with no issue.
We can restore from B2 with no issue.

But every time we try to run our Veeam Backup Copy jobs sending data to B2, it fails with the same error.

In reviewing the logs, Veeam encounters some issues when communicating with the Backblaze bucket, causing the copy job to get into a loop and fail. You can see this in the sanitized log snippets below:

Code: Select all

[05.01.2024 07:40:08.098]    <05>    Info (3)        [Storage] Deleting storage 'xxx.xxx.com.vm-434299D2024-01-05T040637_1E95.vib:4b0506-f829-4908-b317-8328xxxx8d31:1/5/2024 4:06:37 AM:1/5/2024 4:06:37 AM' from DB
[05.01.2024 07:40:08.176]    <05>    Info (3)    [DbKey] Clear db key and read caches
[05.01.2024 07:40:08.176]    <05>   Error (3)    S3 delete multiple objects request failed to delete object [Veeam/Backup/VeeamBackups/Clients/{54ca8df9-a680-4936-857c-fxxx1dfc610}/cxxx0c8ca-c929-4a53-bb4b-0exxx522064/CloudStg/Meta.Copy.0/StgRoots/{2dxxb84-1b71-4941-a6e9xxxxe0f0b7a2}.{412ab48f-4ecc-4802-8ece-931b3xxxx6a3}] and [3] others (Veeam.Backup.Common.CCppComponentException)

[05.01.2024 07:40:08.176]    <05>   Error (3)       in c++: UndeserializableResponse
[05.01.2024 07:40:08.176]    <05>   Error (3)       in c++: CS3VersioningUtils::DeleteFileVersionListAsync async task has failed, objects count [4]
[05.01.2024 07:40:08.176]    <05>   Error (3)       in c++: [CCloudParallelObjsDeleter] Failed to wait for finish
[05.01.2024 07:40:08.176]    <05>   Error (3)       in c++: Failed to delete checkpoints for cloud backup 'c970c8ca-c929-4a53-bb4b-0e4d3exxxx064'
[05.01.2024 07:40:08.176]    <05>   Error (3)    Agent failed to process method {Cloud.DeleteCheckpointsNoSession}. (Veeam.Backup.Common.CCppComponentException)

……………………………

[05.01.2024 07:10:05.372] < 26520> cstg     | Delete object: /Meta/Crypto/{2d493b84-1b71-4941-a6ea-9b5xxx0b7a2}.{dd834e1b-2b18-4d79-b6eb-0e36xxxxx874}, version: 4_z36b6f39xxxx8b4585640e1f_f101bc4839eef2e17_d20231225_m060111_c005_v0501013_t0031_u017xxxx4071415

[05.01.2024 07:10:05.387] < 32360>          | ERR |Failed to deserialize SMultiObjectDeleteResponse: [<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
[05.01.2024 07:10:05.387] < 32360>          | >>  |<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
[05.01.2024 07:10:05.387] < 32360>          | >>  |    <Error>
[05.01.2024 07:10:05.387] < 32360>          | >>  |        <Code>MalformedXML</Code>
[05.01.2024 07:10:05.387] < 32360>          | >>  |        <Message>The XML you provided was not well-formed or did not validate against our published schema</Message>
[05.01.2024 07:10:05.387] < 32360>          | >>  |    </Error>
[05.01.2024 07:10:05.387] < 32360>          | >>  |</DeleteResult>
[05.01.2024 07:10:05.387] < 32360>          | >>  |]
[05.01.2024 07:10:05.387] < 32360>          | ERR |There is no node with the specified name: [Key].
[05.01.2024 07:10:05.387] < 32360> aws      | WARN|Going to retry MultiObjectDeleteRequest, because undeserializable response could mean remote service overload
[05.01.2024 07:10:05.387] < 32360> aws      | >>  |There is no node with the specified name: [Key].
[05.01.2024 07:10:05.387] < 32360> aws      | >>  |--tr:Failed to deserialize SMultiObjectDeleteResponse
[05.01.2024 07:10:05.387] <  7688> aws      | Amazon REST exception with status=500, error code=InternalError.
[05.01.2024 07:10:05.387] <  7688> aws      | WARN|HTTP request failed, retry in [1] seconds, attempt number [2], total retry timeout left: [1798] seconds
[05.01.2024 07:10:05.387] <  7688> aws      | >>  |S3 delete multiple objects request failed to delete object [Veeam/Backup/VeeamBackups/Clients/{54ca8df9-a680-4936-857c-f2192xxx610}/cxxx8ca-c929-4a53-bb4b-0e4dxxx2064/CloudStg/Meta.Copy.0/StgRoots/{2dxxxb84-1b71-4941-a6eaxxxxxxb7a2}.{412ab48f-4ecc-xxxx-8ece-931xxxx76a3}] and [3] others
[05.01.2024 07:10:05.387] <  7688> aws      | >>  |--tr:UndeserializableResponse

As you can see, Veeam attempts to delete some objects from the bucket but isn’t able to, and we get some interesting errors back from the Backblaze bucket itself stating there is no node with the specified name, and a “500 internal error” which is a catch-all error that object storages use.


We opened a case with Backblaze and everything looks OK on their side.

[Mildur]

Hello Juan

Please do not post any logs to this forum as requested by our Forum policy. Our customer support has the tools and training to analyze log files and error messages. Can you please let me know the case number so I can check your support case?

Best,
Fabian

[juanmachado]

@Mildur,

Sorry, I didn't know we couldn't post a tiny sanitized log snippet for others to see if they are experiencing the same problems!

Case 07077746 .

[Mildur]

Thank you for the case number.
If other users have reported the same error message, our support team will see that in their case portal.
I have asked support management to check the state of the escalation, so we get it faster to the next tier.

Best,
Fabian

[Mildur]

A short update from support management. We are seeing now multiple cases with the same situation.
I added the logs snippet back (exception of our forum policy)

Best,
Fabian

[juanmachado]

Thank you for the case number.
If other users have reported the same error message, our support team will see that in their case portal.
I have asked support management to check the state of the escalation, so we get it faster to the next tier.

Best,
Fabian

Thanks. We just found a post on Reddit too
https://www.reddit.com/r/Veeam/comments ... s3_bucket/

[juanmachado]

More details..

I was able to escalate a case with Backblaze..

Their response:
"We are aware of this, and we found the issue. With that said, I don't have an ETA on when the change will be made. Hopefully, we can get it out by the end of the week(After our weekly Thursday maintenance). "

My response:
"This is a very critical issue. We have been out of compliance since Thursday last week when it is now clear that this was caused for a planned change on your end.
Every customer that uses Veeam ( and probably other backup platforms) to send their critical backups to B2 are broken and out of compliance.
Are you saying we have to wait 3 or more days for Backblaze to revert the change and fix this issue ? "

If other B2 customers here are facing the same problem, I recommend you to open a case and put some pressure on Backblaze from your side too.

[wocomike]

Same issue here. We've escalated a ticket with Backblaze.

[glecit]

We've been seeing the same thing since 1/4/23. Once the underlying issue is resolved, will B&R delete these objects automatically during the next run or are objects orphaned after some number of failed attempts to delete?

Code: Select all

Failed to delete checkpoint. Details: S3 delete multiple objects request failed to delete object [Veeam/Archive/Backups/GUID/checkpoints/checkpoint.1131] and [0] others

Code: Select all

Exception from server: S3 delete multiple objects request failed to delete object [Veeam/Archive/Backups/GUID/checkpoints/checkpoint.1131] and [0] others

Code: Select all

         | ERR |Failed to deserialize SMultiObjectDeleteResponse: [<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
         | >>  |<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
         | >>  |    <Error>
         | >>  |        <Code>MalformedXML</Code>
         | >>  |        <Message>The XML you provided was not well-formed or did not validate against our published schema</Message>
         | >>  |    </Error>
         | >>  |</DeleteResult>
         | >>  |]
         | ERR |There is no node with the specified name: [Key].
aws      | WARN|Going to retry MultiObjectDeleteRequest, because undeserializable response could mean remote service overload
aws      | >>  |There is no node with the specified name: [Key].
aws      | >>  |--tr:Failed to deserialize SMultiObjectDeleteResponse
aws      | Amazon REST exception with status=500, error code=InternalError.
aws      | WARN|HTTP request failed, retry in [8] seconds, attempt number [5], total retry timeout left: [1784] seconds
aws      | >>  |S3 delete multiple objects request failed to delete object [Veeam/Archive/Backups/GUID/GUID/checkpoints/checkpoint.1131] and [0] others
aws      | >>  |--tr:UndeserializableResponse

[rmeany]

+1 to glecit's post. I want to make sure that all of these failed deletions won't result in orphaned data that never gets deleted once Backblaze gets their act together.

[Mildur]

@juanmachado
Thank you for the update. Our customer support has an open support ticket as well with Backblaze.
Let's hope it can get solved more quickly.

Best,
Fabian

[juanmachado]

Response from BackBlaze:

"... the fix is now in place as of late last night.  Our support team will start sending correspondence at 9am PST to all affected and advise of the fix. "

[juanmachado]

@juanmachado
Thank you for the update. Our customer support has an open support ticket as well with Backblaze.
Let's hope it can get solved more quickly.

Best,
Fabian

Backblaze said they fixed last night 1/9/24, but it is still broken for us. We are working with Backblaze B2 escalation team and Veeam escalation team to see if all can sit in an online meeting to get this resolved ASAP!

[cableguy]

Hi,
my last backup to Backblaze B2 ran yesterday (Jan 10) at 9 pm GMT+1, and it worked fine after having failed for the last few days. Will submit that to my support case #07083135.
Cheers,
Andy

[Mildur]

Hi Juan, Hi Andreas

Our support team is waiting for Blackblaze to confirm that everything is solved. Your support engineer engineer will reach out to you if further investigations from us are required.

@cableguy
I replaced in your comment your contract number (#03) with the case number (#07).

Best,
Fabian

[juanmachado]

After working with Veeam for several days and after being able to escalate the case to an escalation level that we didn't know existed, Veeam developers were able to find the problem and issue a new Hotfix.

After the hotfix was installed, the problem is resolved.

[Mildur]

Hello Juan

Thanks for the feedback.

Best,
Fabian

[glecit]

@Mildur, do you know the answer to this?

Once the underlying issue is resolved, will B&R delete these objects automatically during the next run or are objects orphaned after some number of failed attempts to delete?

[Mildur]

Hi Glec IT

I will check with my colleagues.

Best,
Fabian

[Mildur]

Hi @glecit
The next session will retry the cleaning process. Previously "failed to delete objects" will be deleted in the next session.

Best,
Fabian