Discussions specific to the VMware vSphere hypervisor
Post Reply
Andrew1344
Novice
Posts: 5
Liked: never
Joined: Feb 24, 2012 3:03 pm
Full Name: andrew
Contact:

Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by Andrew1344 » Apr 08, 2019 2:21 pm

Hope someone can help. I am receiving this error intermittently on multiple jobs. I opened a case with veeam but haven't gotten much help.
Is my case number 03491145

The error is:

Code: Select all

Error: Unstable connection: unable to transmit data.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: read: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [0].
Failed to download disk.
I have multiple replication jobs that are functioning well using the same proxy and connection. This error is appearing on 2 of those jobs.
Steps Taken:
Iperf'd the connection no issues
Updated to latest build on all components from veeam
Rebooted Proxy
Called Support

I still have the issue.
Thanks!!!!!

foggy
Veeam Software
Posts: 19065
Liked: 1697 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by foggy » Apr 08, 2019 5:31 pm

Hi Andrew, as far as I can see from the case notes, you've been requested to provide the logs that cover the time the error occurred, let's see what our engineers can suggest after reviewing them.

csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by csydas » Apr 10, 2019 9:02 pm 1 person likes this post

Hiya Andrew,

We had this once and support pushed us here: https://www.veeam.com/kb2140

I know already what you're thinking; no fabulous way it's the firewall, 'cause we had the same thought in my shop. But sure enough, I dug though the IDS logs and we saw intrusion prevention entries that matched exactly when the job ran.

Check it out on your firewall, I bet you it's this.

Andrew1344
Novice
Posts: 5
Liked: never
Joined: Feb 24, 2012 3:03 pm
Full Name: andrew
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by Andrew1344 » Apr 11, 2019 6:03 pm

Thanks csydas! I just escalated the case as I we were getting no where. I do believe that is not fabulously possible its the firewall. But I will take a look again. Thanks appreciate the effort and suggestion.

unsichtbarre
Expert
Posts: 138
Liked: 28 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by unsichtbarre » Apr 12, 2019 4:59 pm

Hi Andrew,

After Update 4a we are seeing more jobs stop/hang after several hours. Since there have been no firewall/av updates, this seems to be a 4a issue as opposed to a network one. Good luck.
-The Invisible Admin-
http://www.johnborhek.com

Gostev
SVP, Product Management
Posts: 26122
Liked: 4067 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by Gostev » Apr 12, 2019 10:15 pm

Guys, the issue should be super easy to troubleshoot by creating a temporary repository with direct network connection (no firewalls, routers) and see if the issue persists. If not, then clearly the issue is with the networking equipment. This is not really an uncommon issue: for most environments, no other production workload generates as much network traffic as backup application, and this high load uncovers bugs in the network equipment.

lussierd16
Service Provider
Posts: 3
Liked: 3 times
Joined: Mar 14, 2016 2:48 pm
Full Name: Dustin Lussier
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by lussierd16 » Sep 18, 2019 2:47 pm

Has anyone found a resolution to this? I'm on 9.5u4 and I've been having this issue for months. I have a case open and at this point we're going to start a packet capture to figure out what is causing the issue. we're also going to upgrade to 9.5u4b tomorrow.

Gostev
SVP, Product Management
Posts: 26122
Liked: 4067 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by Gostev » Sep 18, 2019 3:21 pm

Yep, WireShark dumps is usually the way to go to troubleshoot these kind of issues.

unsichtbarre
Expert
Posts: 138
Liked: 28 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by unsichtbarre » Sep 23, 2019 1:56 pm

Any specific conversation, search or pattern I should identify?
-The Invisible Admin-
http://www.johnborhek.com

Gostev
SVP, Product Management
Posts: 26122
Liked: 4067 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by Gostev » Sep 23, 2019 5:24 pm

I would guess it depends on the specific issue, but no idea on details to be honest - I just know this is the process, and that it works well in determining the offending networking equipment. Also, just to be clear - customers are not expected to analyse those dumps themselves, this is done by our T3 support engineers who specialize on networking issues. Thanks!

soncscy
Expert
Posts: 134
Liked: 54 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey Carel
Contact:

Re: Error: Unstable connection: unable to transmit data. Failed to upload disk

Post by soncscy » Sep 24, 2019 4:52 am 1 person likes this post

Responding from my newer account since I previously commented in this thread.

@John, let support help I guess, but really, just check with whoever is managing your firewall(s) and look for IDS/IPS activity during your backup times. You can hunt this down with Wireshark sure, but I think it's overkill. Since Veeam will just repeatedly try the same block over and over, you should get a huge chunk in the logging about blocked activity. For clients of ours we catch this all the time just by checking the firewall logs.

And to be clear, my understanding of the issue is that there's no need for any changes to have happened to the firewall/av. The data going into a backup file ends up with more or less "random" looking blocks, but each time the same block that triggers IDS/IPS is resent until Veeam is instructed to send a different one (which is why Active Fulls fix this, since it's different data blocks then). Since signature based threat detection is always a ratio of speed:uniqueness, algorithms that favor speed inevitably have hash collisions between legitimate data and malicious data. And basic probability tells us that the likelihood of a collision happening multiple times is independent of the number of times in a period; you can get hot streaks of "luck" like this.

@Gostev; I suppose the way they probably do it is via monitoring TTL on the RSTs, as shown in this article: https://medium.com/@pjperez/the-ttl-trick-c78f1279817d

tldr; - TTL value will tell you what sent the RST packet; windows typically uses 128, Linux typically 64, networking gear will have their own preferences. If you see RST with TTL of 128, it's local firewall. 127 would be some server in between, something like 255 is some network gear 1 hop away.

Post Reply

Who is online

Users browsing this forum: Edage, gavrun, torri.sigl and 40 guests