Capacity Tier Offload - Failure on some VMs

[wa15]

A SOBR capacity tier offload job started for the first time, containing 19 VMs totaling 30TB, writing to Azure. After some time, the job gave two different error messages on two of the VMs and failed at the same time. One VM had been offloading for about 7 hours and the other for 10 hours. One is a 4TB VM and one is a 2TB VM. Larger VMs in the batch have been uploading fine thus far.

One error message reads:
1/14/2020 11:31:48 PM :: Failed to offload backup Error: WinHttpReceiveResponse: 12002: The operation timed out

The second error message reads:
1/14/2020 11:31:48 PM :: Failed to offload backup Error: WinHttpWriteData: 12030: The connection with the server was terminated abnormally

The two other VMs that were processing during this time did not give any errors (total 4 simultaneous VMs processing). And other VMs in the job (9) uploaded successfully.

There is no proxy for internet access, and there is no antivirus on the backup server. This is an all in one Veeam server (v9.5.4.2866) with low resource utilization.

I did open up case 03953992 but wanted to reach out in case any one has seen these errors before.

Thanks!

[wishr]

Hi Wa15,

For me, it looks like a connectivity issue, but I would suggest letting our support engineers have a detailed look at all the debug logs.

Thanks

[jdombeck]

I am having the same problem using Backblaze B2 as the capacity tier. Some offload jobs complete successfully, but others run for minutes to hours, then fail with a similar message to the above:

10/27/2020 6:54:39 PM :: WinHttpWriteData: 12152: The server returned an invalid or unrecognized response. Shared memory connection was closed.

There isn't much of anything in the forum about this issue. Does anyone have an idea about how to approach this? Thanks.

[Gostev]

Start from opening a support case with Veeam to see if they can fetch the exact invalid response from debug logs. Then open a support case with Backblaze and have them troubleshoot why their servers return this invalid message sometimes.

[pciccone]

Avoid BackBlaze S3-compatible storage with Veeam. We had a nightmare of the exact same problems and ultimately we had to change providers. It really opened our eyes when we tried to pull data back and could not (which was scary).

[maus]

Hi,

I have kind of the the same problem. SOBR offload jobs were running fine for weeks. Then all of sudden (13-09) the offload stopped working the error code: 12002*

Veeam case: Case #05027499

I also have of case open at Wasabi which is my Object storage provider for the SOBR. The told me that at there end everything looks good. When I create a new backup job offload is working fine... I'm afraid that if we cant fix this issue, we're getting to far behind with uploading and the scale out (copy) will be useless and we forced start over (25 TB).

I cant fix this issue myself. I wonder what Veeam's next suggestion will be...

Wasabi support:
Thanks for getting back to me. I have just looked into logs on your account to eu-central-1 buckets over the past 7 days, and I see a total ~3.3Million PUT requests, all of which received a 200 OK response meaning successful. Only 1 OUT got a 500 error on our end, but that is retryable on the client end and would not cause your backup(s) to fail. All else looks good to me. No errors on DELETEs, or on GET or HEAD. Just a handful of 404s on those but that is normal and is just Veeam checking for the existence of objects on the S3 end. Let me know if you/Veeam needs more from my end!

Best,

Wasabi Technologies Inc

*

Code: Select all

[18.09.2021 03:57:35] <19> Error        HTTP exception: WinHttpQueryDataAvaliable: 12002: The operation timed out
[18.09.2021 03:57:35] <19> Error         (Veeam.Backup.Common.CCppComponentException)
[18.09.2021 03:57:35] <19> Error        , error code: 12002 (Veeam.Backup.Common.CCppComponentException)
[18.09.2021 03:57:35] <19> Error           in c++: [CloudArchDiskCleaner] Failed to wait for finish
[18.09.2021 03:57:35] <19> Error           in c++: Failed to remove unique blocks of item '/12193af0-ed91-4363-8c5b-8f5db835ce50 (1)/AD10-flat.vmdk'.
[18.09.2021 03:57:35] <19> Error           in c++: Failed to do a partial commit for storage 'ca1ddfdd-de5b-b1e1-4a8f-8d312cbabee1.94'.
[18.09.2021 03:57:35] <19> Error           in c++: Failed to do a partial commit for storage [ca1ddfdd-de5b-b1e1-4a8f-8d312cbabee1:94].
[18.09.2021 03:57:35] <19> Error           in c++: Failed to finalize pending archive task [DehydrateTask] for object [ca1ddfdd-de5b-b1e1-4a8f-8d312cbabee1:94].
[18.09.2021 03:57:35] <19> Error           in c++: Failed to process method 'ArchRepo.FinalizePendingTask'
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Common.CVcpInvoker.Invoke(CVcpStreamBase stream, String command, CVcpCommandArgs inArgs, Boolean noLog)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[18.09.2021 03:57:35] <19> Error        --- End of stack trace from previous location where exception was thrown ---
[18.09.2021 03:57:35] <19> Error           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.AgentProvider.CBackupClient.ArchRepoFinalizePendingTask(String archRepoId, String backupId, String indexId, CCapacityTierAgentTask task, DateTime currentDate)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveIndexAgent.FinalizePendingTask(CCapacityTierAgentTask task, DateTime currentDate)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupRepairPerformer.TryRollbackAgentTask(List`1 storagesToDehydrate, CCapacityTierAgentTask agentTask, CArchiveIndexAgent indexAgent)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupRepairPerformer.GetAndRollbackPendingTasks(List`1 storagesToDehydrate, CArchiveIndexAgent indexAgent, Boolean isCleanup)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupRepairPerformer.Repair(List`1 orderedStoragesToDehydrate, CBackupTaskSession taskSession)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, CBackupTaskSession taskSession, ILogRecordHandler log, IStopSessionSync sessionControl, CDisposableLock& indexLock, IDisposable& backupLock)
[18.09.2021 03:57:35] <19> Error        --- End of stack trace from previous location where exception was thrown ---
[18.09.2021 03:57:35] <19> Error           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.HanldeException(Exception e, CArchiveBackupChainTask task, CArchiveBackupChainResourceTask chainResourceTask, CTasksScheduler tasksScheduler, ILogRecordHandler log)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, CBackupTaskSession taskSession, ILogRecordHandler log, IStopSessionSync sessionControl, CDisposableLock& indexLock, IDisposable& backupLock)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.PerformCycle(CArchiveBackupChainTask task, CBackupTaskSession& taskSession, ILogRecordHandler log, IStopSessionSync sessionControl)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.Perform(CArchiveBackupChainTask task, IStopSessionSync sessionControl)
[18.09.2021 03:57:35] <19> Error           at Veeam.Backup.Core.Archive.CArchiveBackupTaskPerformer.PerformSafe(CArchiveBackupChainTask task, IStopSessionSync sessionControl)

The other issue is that since this offload jobs get sucked somehow. New offload jobs (of the same backups job) just keep piling up..... to eventually fail or get cancelt.

Regards,
Maurice

[veremin]

Let our engineers analyze the debug logs and your environment and comes us with the resolution, as it's rather hard to investigate technical issue via forum correspondence. Thanks!

[ColterT]

I will open a formal case for this tomorrow, but I'm researching this today. Was there a resolution on the mentioned case here? Thanks everyone.

[veremin]

Unfortunately, I was unable to find the ticket data or the final resolution.

I kindly request that you create a support ticket of your own and have the assigned engineers investigate the issues further.

Thanks!

[Gostev]

This was a few years ago, we had a different support ticketing system back then. Also it wouldn't be relevant anyway, as the engine has evolved too significantly.