Using object storage as a backup target
Post Reply
wa15
Expert
Posts: 319
Liked: 24 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Capacity Tier Offload - Failure on some VMs

Post by wa15 »

A SOBR capacity tier offload job started for the first time, containing 19 VMs totaling 30TB, writing to Azure. After some time, the job gave two different error messages on two of the VMs and failed at the same time. One VM had been offloading for about 7 hours and the other for 10 hours. One is a 4TB VM and one is a 2TB VM. Larger VMs in the batch have been uploading fine thus far.

One error message reads:
1/14/2020 11:31:48 PM :: Failed to offload backup Error: WinHttpReceiveResponse: 12002: The operation timed out

The second error message reads:
1/14/2020 11:31:48 PM :: Failed to offload backup Error: WinHttpWriteData: 12030: The connection with the server was terminated abnormally

The two other VMs that were processing during this time did not give any errors (total 4 simultaneous VMs processing). And other VMs in the job (9) uploaded successfully.

There is no proxy for internet access, and there is no antivirus on the backup server. This is an all in one Veeam server (v9.5.4.2866) with low resource utilization.

I did open up case 03953992 but wanted to reach out in case any one has seen these errors before.

Thanks!

wishr
Veeam Software
Posts: 2808
Liked: 388 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Capacity Tier Offload - Failure on some VMs

Post by wishr » 1 person likes this post

Hi Wa15,

For me, it looks like a connectivity issue, but I would suggest letting our support engineers have a detailed look at all the debug logs.

Thanks

jdombeck
Novice
Posts: 3
Liked: 1 time
Joined: Feb 15, 2014 6:42 pm
Full Name: John Dombeck
Contact:

Re: Capacity Tier Offload - Failure on some VMs

Post by jdombeck »

I am having the same problem using Backblaze B2 as the capacity tier. Some offload jobs complete successfully, but others run for minutes to hours, then fail with a similar message to the above:

10/27/2020 6:54:39 PM :: WinHttpWriteData: 12152: The server returned an invalid or unrecognized response. Shared memory connection was closed.

There isn't much of anything in the forum about this issue. Does anyone have an idea about how to approach this? Thanks.

Gostev
SVP, Product Management
Posts: 29346
Liked: 5475 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Capacity Tier Offload - Failure on some VMs

Post by Gostev »

Start from opening a support case with Veeam to see if they can fetch the exact invalid response from debug logs. Then open a support case with Backblaze and have them troubleshoot why their servers return this invalid message sometimes.

pciccone
Service Provider
Posts: 2
Liked: 1 time
Joined: Aug 14, 2020 4:17 pm
Full Name: Phil Ciccone
Contact:

Re: Capacity Tier Offload - Failure on some VMs

Post by pciccone » 1 person likes this post

Avoid BackBlaze S3-compatible storage with Veeam. We had a nightmare of the exact same problems and ultimately we had to change providers. It really opened our eyes when we tried to pull data back and could not (which was scary).

maus
Lurker
Posts: 1
Liked: never
Joined: Jul 04, 2012 7:45 am
Full Name: Maurice van Beelen
Contact:

Re: Capacity Tier Offload - Failure on some VMs

Post by maus »

Hi,

I have kind of the the same problem. SOBR offload jobs were running fine for weeks. Then all of sudden (13-09) the offload stopped working the error code: 12002*

Veeam case: Case #05027499

I also have of case open at Wasabi which is my Object storage provider for the SOBR. The told me that at there end everything looks good. When I create a new backup job offload is working fine... I'm afraid that if we cant fix this issue, we're getting to far behind with uploading and the scale out (copy) will be useless and we forced start over (25 TB).

I cant fix this issue myself. I wonder what Veeam's next suggestion will be...

Wasabi support:
Thanks for getting back to me. I have just looked into logs on your account to eu-central-1 buckets over the past 7 days, and I see a total ~3.3Million PUT requests, all of which received a 200 OK response meaning successful. Only 1 OUT got a 500 error on our end, but that is retryable on the client end and would not cause your backup(s) to fail. All else looks good to me. No errors on DELETEs, or on GET or HEAD. Just a handful of 404s on those but that is normal and is just Veeam checking for the existence of objects on the S3 end. Let me know if you/Veeam needs more from my end!

Best,

Wasabi Technologies Inc

*

Code: Select all

[18.09.2021 03:57:35] <19> Error        HTTP exception: WinHttpQueryDataAvaliable: 12002: The operation timed out
[18.09.2021 03:57:35] <19> Error         (Veeam.Backup.Common.CCppComponentException)
[18.09.2021 03:57:35] <19> Error        , error code: 12002 (Veeam.Backup.Common.CCppComponentException)
[18.09.2021 03:57:35] <19> Error           in c++: [CloudArchDiskCleaner] Failed to wait for finish
[18.09.2021 03:57:35] <19> Error           in c++: Failed to remove unique blocks of item '/12193af0-ed91-4363-8c5b-8f5db835ce50 (1)/AD10-flat.vmdk'.
[18.09.2021 03:57:35] <19> Error           in c++: Failed to do a partial commit for storage 'ca1ddfdd-de5b-b1e1-4a8f-8d312cbabee1.94'.
[18.09.2021 03:57:35] <19> Error           in c++: Failed to do a partial commit for storage [ca1ddfdd-de5b-b1e1-4a8f-8d312cbabee1:94].
[18.09.2021 03:57:35] <19> Error           in c++: Failed to finalize pending archive task [DehydrateTask] for object [ca1ddfdd-de5b-b1e1-4a8f-8d312cbabee1:94].
[18.09.2021 03:57:35] <19> Error           in c++: Failed to process method 'ArchRepo.FinalizePendingTask'
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[18.09.2021 03:57:35] <19> Error        --- End of stack trace from previous location where exception was thrown ---
[18.09.2021 03:57:35] <19> Error           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.ArchRepoFinalizePendingTask(String archRepoId, String backupId, String indexId, CCapacityTierAgentTask task, DateTime currentDate)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveIndexAgent.FinalizePendingTask(CCapacityTierAgentTask task, DateTime currentDate)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupRepairPerformer.TryRollbackAgentTask(List`1 storagesToDehydrate, CCapacityTierAgentTask agentTask, CArchiveIndexAgent indexAgent)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupRepairPerformer.GetAndRollbackPendingTasks(List`1 storagesToDehydrate, CArchiveIndexAgent indexAgent, Boolean isCleanup)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupRepairPerformer.Repair(List`1 orderedStoragesToDehydrate, CBackupTaskSession taskSession)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, CBackupTaskSession taskSession, ILogRecordHandler log, IStopSessionSync sessionControl, CDisposableLock& indexLock, IDisposable& backupLock)
[18.09.2021 03:57:35] <19> Error        --- End of stack trace from previous location where exception was thrown ---
[18.09.2021 03:57:35] <19> Error           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.HanldeException(Exception e, CArchiveBackupChainTask task, CArchiveBackupChainResourceTask chainResourceTask, CTasksScheduler tasksScheduler, ILogRecordHandler log)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, CBackupTaskSession taskSession, ILogRecordHandler log, IStopSessionSync sessionControl, CDisposableLock& indexLock, IDisposable& backupLock)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.PerformCycle(CArchiveBackupChainTask task, CBackupTaskSession& taskSession, ILogRecordHandler log, IStopSessionSync sessionControl)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, IStopSessionSync sessionControl)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.PerformSafe(CArchiveBackupChainTask task, IStopSessionSync sessionControl)
The other issue is that since this offload jobs get sucked somehow. New offload jobs (of the same backups job) just keep piling up..... to eventually fail or get cancelt.

Regards,
Maurice

veremin
Product Manager
Posts: 18981
Liked: 1923 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Capacity Tier Offload - Failure on some VMs

Post by veremin »

Let our engineers analyze the debug logs and your environment and comes us with the resolution, as it's rather hard to investigate technical issue via forum correspondence. Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 13 guests