[Cloud machines] Azure VM SQL Servers VEEAM DISASTER CLR exception type: System.TimeoutException

[patrikcze]

Dear all,

after 5 months of finding issue why Veeam is not possible to backup couple of Azure VMs running Microsoft SQL Server. I've decided to post something to forum.
We are facing major issues with Veeam which is intermittently failing into state from which only "Reboot of whole VBR" server can partially help. Whole Veeam Infrastructure is built in Azure and is taking care about Azure VMs running Microsoft SQL Servers.
First our database team colleagues found out that there are problems with transactional log backups, then with backups in general and then it is required to reboot server because no jobs can be started, stopped, nothing is running database is full of information that jobs are already running and there is now way around this. This problem usually comes in times where many changes to our Azure environment is happening. Now it occurs from time to time but at least once or twice a week. Currently about 80+ individual SQL Servers located in different Azure Subscriptions using dedicated Storage Account for backups with copy to storage account sitting in different azure region.
After 4 months and exchange of 0.5TB of VEEAM LOGS we came to nothing. Completly nothing, we do not know what is root cause of whole problem, we do not know how to fix it, we do not know how to continue. Not mentioning that it is causing troubles to production SQL Systems which are growing in TR Logs and failing down.

Complete disaster is crash in Veeam process :

Code: Select all

CLR exception type: System.TimeoutException
    "Cloud instance is unresponsive."

Call stack snippet:
Veeam.Backup.ServiceLib.CPublicCloudQueueHubService.ReceiveResponse
 → Veeam.Backup.Core.CCloudMessageServiceSendQueueClient.ReceiveResponse
 → ... 
 → System.Threading.ThreadHelper.ThreadStart
 

Cloud Machines (regular Agents) in azure do not communicate with Backed-Up VMs directly, rather using indirect communication over Azure Storage Account QUEUEs, where they send their API Requests from Agents and from VBR server, I did not program that so I can only guess how it works in details. However whole problem seems to be lying somewhere here. Unfortunately we do not know, because Veeam was not able to find anything and our DB Colleauges are complaining that Veeam Cannot use Try&Catch possibly to handle unexpected and unhandled exceptions like TimeOutExceptions (just a sidenote). However it seems that due to that issue which might occur because no-one knows how any cloud works it's simply just someone's else computer Veeam instead of gracefully closing threads and backups and trying to handle this somehow will send VBR to complete death, totally unresponsive non working app which is doing something but backups.

My internal review (I was not fully involved in investigation) is following :

• Where the Timeout Happens

The exception arises in Veeam.Backup.ServiceLib.CPublicCloudQueueHubService.ReceiveResponse, which is part of Veeam’s logic for talking to cloud services (e.g., Azure Blob, Amazon S3, or a Veeam Cloud Connect provider). In other words, the “hub service” is attempting to send or receive data and has not gotten a response in time.

• “Cloud instance is unresponsive.”

This is Veeam’s message stating that the remote endpoint (storage, queue, or cloud connect server) is not replying before the configured .NET/Veeam timeout.

• In Azure scenarios, this can be triggered by high latency or an unavailable storage service, network interruption, or throttling on the Azure side.

• If you run large or frequent backups, or have concurrency set high, you might hit ephemeral network issues that cause the Veeam call to stall until it times out.

• Likely Root Causes

• Azure transient network or storage latency. Even minor network blips can cause timeouts if Veeam’s default limit is exceeded.

• Azure resource throttling (e.g., exceeding storage IOPS or egress limits).

• Firewall / NSG misconfiguration that intermittently blocks or slows traffic.

• DNS / name resolution delays.

• Timeout / Retry settings in Veeam are too low to handle sporadic high-latency calls.

Anyway none of above should cause troubles to Veeam at all, its architecture should be somewhat resilient and durable, because we know how it is in Cloud. Sometimes Too Many requests, simetimes something else. However it is clear that someone should properly handle how "Cloud Machines" are handled. We know that previously Veeam was using "Service Bus" unfortunately that option missed our implementation by 2 months. So we are now using "Azure Storage QUEUES" I do not know if this was really good approach, in my eyes it seems like complete disaster. There must be something wrong with Veeam, .NET Framework or something and it has to be fixed. Otherwise it is not possible to wait for crash of complete backup system just becusase timeout occurred somewhere.

Did anyone experienced this issue? Can someone help? We do appreciate any help here.

Backups running daily with 1h trlog backups (Where FULL RECOVERY DBs are), there are invidual copy jobs which do secondary copy to different azure region. We do have Storage QUEUEs in each subscription where resources are sitting. Storage Accounts are accessible to both backed up VMs and also to VBR server. It is almost 80 Individual Subscriptions for 80+ Individual Azure VMs. Veeam B&R Server is quite beefy Azure VM and should handle even more, it has individual dedicated remote MSSQL Server Also Azure VM which has no performance issues. Azure Cloud Machines (Agents) do have sizing required for running Microsoft SQL Servers. Usually there are no more then 1 backups running to one Storage Account.

[PetrM]

Hi Patrik,

Indeed, the last method in the stack (Veeam.Backup.ServiceLib.CPublicCloudQueueHubService.ReceiveResponse) likely indicates that the timeout while waiting for a response from the cloud has been exceeded. I fully agree with the list of potential root causes, but I think it's too early to draw any conclusions about possible ways to handle it. We simply do not have enough information. We need to answer two main questions first:
1) What is the real root cause in your case?
2) Is it possible to work around the problem on our side?

Could you please provide me with a support case number? I doubt that we can effectively troubleshoot it by exchanging messages on this topic, our support team must review the logs and probably carry out some additional testing.

Thanks!

[patrikcze]

Sent via private message.

[patrikcze]

Happened again. Today. “Cloud instance is unreaponsive”

[PetrM]

Hi Patrik,

The case has been escalated to the QA/Dev team, and they are researching the issue. Our engineer will share the results of the research once it is completed.

Thanks!

[patrikcze]

Many thanks

[MPECSInc]

Any update?

[PetrM]

Hi Philip,

Our development team is working on the issue and has made good progress: there was a command hanging that impacted jobs on Veeam B&R. They had to run a couple of tests with additional logging to gather more details. However, if you encounter a similar issue in your environment, please open a support case and share its ID as well. Sometimes, technical issues can have identical symptoms while the root causes are entirely different.

Thanks!

[PetrM]

Hello,

@patrikcze As far as I can see, our support team has sent you a private fix. I think you should continue working with them to resolve any remaining questions you have.

Thanks!

[MPECSInc]

--- edited by moderator ---
We build highly available solutions for a living and have been doing so since 2007.

These two:
1: Azure transient network or storage latency. Even minor network blips can cause timeouts if Veeam’s default limit is exceeded.
2: Azure resource throttling (e.g., exceeding storage IOPS or egress limits).

When the issues are happening be logged in to the desktop of a couple or three of the SQL Servers being backed up.

Open Resource Monitor.
Disk Tab
Processes with Disk Activity: Sort Highest: Total (B/sec)
Disk Activity: Sort Highest: Response Time
Right Column: Disk Queue Length (Normally this is 2x the number of disks in a RAID array)

The two key metrics are latency and disk queue length. Are they spiking or consistently high?

Is the Veeam instance in a Windows OS and have the desktop available? If it does, do the same thing.

This is cloud. In cloud there is no consistency since _we_ don't build the end to end solution structure.

You are completely reliant upon Microsoft here and that is where the problem lies.

[PetrM]

Our support team has already sent a fix for this. Let's not interfere, I strongly recommend following the instructions provided by our support team only. This forum topic is not the best communication channel for technical troubleshooting.

Thanks!

[MPECSInc]

Apologies PetreM.

Please delete my post since I am not able to either redact it or delete it.

[patrikcze]

Our support team has already sent a fix for this. Let's not interfere, I strongly recommend following the instructions provided by our support team only. This forum topic is not the best communication channel for technical troubleshooting.

Thanks!

My colleague has already notified me. Many thanks

[PetrM]

Hello,

@MPECSInc No worries, I edited your post to shorten it. In general, your point of view is valid, so if you don't mind, let's keep it for future readers so they can perform some basic troubleshooting while waiting for a reply from our support team.

Thanks!