Discussions specific to the VMware vSphere hypervisor
Mark Muyskens
Lurker
Posts: 1
Liked: never
Joined: Aug 07, 2012 7:33 pm
Full Name: Mark Muyskens
Contact:

Unable to retrieve next block transmission command

Post by Mark Muyskens » Aug 10, 2012 7:29 am

Hello,

I am trying to setup Veeam to use an Amazon EC2 instance as a backup repository and I am running into the following type of errors;

Error: Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [210419].
Error: Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [316108].

Is anyone familiar with this type of setup? Does this seem like a network type issue? Would moving from a micro instance to a larger instance at Amazon provide me with more network resources and resolve this issue?

Anyone care to take a stab?

foggy
Veeam Software
Posts: 18366
Liked: 1575 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backing up to Amazon EC2

Post by foggy » Aug 10, 2012 9:37 am

Yes, usually this type of errors is caused by the unstable network connection.

Please include your support case ID number as explained when you click New Topic.

garufo
Novice
Posts: 3
Liked: never
Joined: Sep 14, 2010 6:34 pm
Full Name: Garufo
Contact:

[MERGED] Deduplication Storage: "Connection reset by peer"

Post by garufo » Oct 16, 2012 8:50 am

Hello,

We are having several backup fails in our environment (Case # 00134265). We have a Veeam Backup 6.1 patch 1 (virtual machine), and we use a Storage Deduplication Appliance (Fujitsu CS800) as a repository, attached by as a NFS resource to a Red Hat Linux virtual machine in the same VMware 4.1 environment.

The CS800 NFS resource is with the deduplication active. 2 weeks ago we had errors in the backups that we assumed that were about the Storage Array where the virtual machines are running because we had an incident in the storage. For this reason we had to reconfigure all the backup tasks in a no deduplication resource, all the jobs where running in the right way. Last week we where "moving" the backup files to a new repositoy in the CS800, another NFS resource in the Red Hat virtual machine, but with deduplication active. This weeked had started again the errors:

Error: Cliente error: Connection reset by peer. Unable to retrieve next block transmission command. Number of already processed blocks: [9811]
Busy: Source 13%> Proxy> 6%> Target 89%

(and similiar errors always, changing the number of processed blocks). Sometimes it stopped in a virtual disk, sometimes in another. There are 2 tasks at least with that error.

We had do the following troubleshooting:

- Disable multiple TCP/IP in global traffic throttling.
- Backup task options: Inline deduplication: disabled. Compression: NONE. Backup type: LAN

We are now again with Veeam support trying to find out what is happening.

Any suggest??

thanks in advanced

Gostev
SVP, Product Management
Posts: 24936
Liked: 3621 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Unable to retrieve next block transmission command

Post by Gostev » Oct 16, 2012 10:43 am 1 person likes this post

Most likely, this is caused by network connection drops.

garufo
Novice
Posts: 3
Liked: never
Joined: Sep 14, 2010 6:34 pm
Full Name: Garufo
Contact:

Re: Unable to retrieve next block transmission command

Post by garufo » Oct 16, 2012 11:24 am

Thank you, we are trying a troubleshooting about it,

thanks

George
Influencer
Posts: 22
Liked: never
Joined: Jan 27, 2012 12:25 pm
Full Name: George
Contact:

[MERGED] End of file Unable to retrieve next block transmiss

Post by George » Oct 25, 2012 5:27 pm

Hi Guys,

I have seen other posts regarding this error but was wondering if you could help me out.

I have a Veeam Server running on Windows with a Linux repository, I'm running on 6.1 and at the moment i get this error during the middle of the job. Its running just fine for a while then stops. Not sure what could be causing it everything on the network looks fine.

The errors message i get is: End of file: Unable to retrieve next block transmission command. Number of already processed blocks:

Any thoughts would be much appreciated!

Thanks!

mgd5
Enthusiast
Posts: 25
Liked: never
Joined: Oct 05, 2012 6:36 am
Full Name: Christer Sundqvist
Contact:

Re: Unable to retrieve next block transmission command

Post by mgd5 » Nov 16, 2012 9:14 am

Hi,

We experience the exact same error. have you found any resolution to the issue

Code: Select all

Error: Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [723354]
MGD

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Unable to retrieve next block transmission command

Post by Vitaliy S. » Nov 16, 2012 9:16 am

Hi Christer, what is the network connection to your repository? Are you using a WAN link?

mgd5
Enthusiast
Posts: 25
Liked: never
Joined: Oct 05, 2012 6:36 am
Full Name: Christer Sundqvist
Contact:

Re: Unable to retrieve next block transmission command

Post by mgd5 » Nov 19, 2012 10:00 am

Hi

No it´s set to LAN Target.

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Unable to retrieve next block transmission command

Post by Vitaliy S. » Nov 19, 2012 10:02 am

Yes, but what is your backup repository? Where is it located? According to the error message, the connection doesn't seem to be very stable.

mgd5
Enthusiast
Posts: 25
Liked: never
Joined: Oct 05, 2012 6:36 am
Full Name: Christer Sundqvist
Contact:

Re: Unable to retrieve next block transmission command

Post by mgd5 » Nov 19, 2012 1:05 pm

Hi

Our Backup Rep is a Linux gateway witch has mounted NFS share from our Datadomain box. I have seen the same error when we had the linux gateway had a NFS mount to a physical linux box. The backuprepositorys we use are virtual.

Christer

jpeake
Enthusiast
Posts: 88
Liked: 25 times
Joined: Sep 25, 2012 7:57 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by jpeake » Nov 19, 2012 2:18 pm

I am having this same error in 6.5 when backing up to our remote site over WAN. It happens on one or two servers per job. If I retry, sometimes it works, sometimes it doesn't.

Working with support, they advised to use network mode on the proxy for that job (i set up a new proxy in network mode, and pointed that job at it). I tried this over the weekend, and sure enough the job completed without error.

Curious to see if this actually fixes it. Will report back after a few more jobs run

topry
Enthusiast
Posts: 49
Liked: 1 time
Joined: Jan 07, 2011 9:30 pm
Full Name: Tim O'Pry
Contact:

Re: Unable to retrieve next block transmission command

Post by topry » Nov 20, 2012 8:33 pm

We have started seeing these errors since the upgrade to 6.5 (not before) and no other upgrades/changes other than the upgrade.
We receive this error primarily while backing up one VM, yet it shares the same SAN/iSCSI network as all of our 3 hosts and the frequency is never more than once per day, 4-5 days out of 7.

All backup sets are using SAN and other than this periodic issue, everything else works normally and test restores before/after this error show no problems. Veeam is installed withing a VM and the host and SAN are connected via dedicated gigabit switch for all iSCSI traffic with 6 ports on the SAN and 4 each per host. No other indications of network issues.

The specific error we receive is:

Code: Select all

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]
If anyone has any suggestions on what else to check/how to track this further.

jpeake
Enthusiast
Posts: 88
Liked: 25 times
Joined: Sep 25, 2012 7:57 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by jpeake » Nov 20, 2012 8:43 pm

I tried running in network mode, as suggested by support. It worked over the weekend, but last night's inc failed with the same error. So i am back to the drawing board.

This has only ever happened on my "offiste" job, which is across a WAN. The local job has been solid

Gostev
SVP, Product Management
Posts: 24936
Liked: 3621 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Unable to retrieve next block transmission command

Post by Gostev » Nov 20, 2012 8:51 pm

If you are always gettin [0] at the end of the error, then I'd say this is definitely completely different issue (not what was being discussed in this thread above). WAN issues would cause this number to be random every time the job fails, because connection drop is random.

jpeake
Enthusiast
Posts: 88
Liked: 25 times
Joined: Sep 25, 2012 7:57 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by jpeake » Nov 20, 2012 8:56 pm

Gostev wrote:If you are always gettin [0] at the end of the error, then I'd say this is definitely completely different issue (not what was being discussed in this thread above). WAN issues would cause this number to be random every time the job fails, because connection drop is random.
makes sense. my number is always some random block

Gostev
SVP, Product Management
Posts: 24936
Liked: 3621 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Unable to retrieve next block transmission command

Post by Gostev » Nov 20, 2012 9:04 pm

In that case, this is clearly WAN link issue that did not exist before. In fact, 6.5 can sustain much worse packet loss than all previous B&R versions, so the issues must be real bad comparing to how this one link used to work when you had previous B&R versions installed. To be precise, this engine's enhancement was added in B&R 6.1 Patch 1.

jpeake
Enthusiast
Posts: 88
Liked: 25 times
Joined: Sep 25, 2012 7:57 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by jpeake » Nov 20, 2012 9:17 pm

Gostev wrote:In that case, this is clearly WAN link issue that did not exist before. In fact, 6.5 can sustain much worse packet loss than all previous B&R versions, so the issues must be real bad comparing to how this one link used to work when you had previous B&R versions installed. To be precise, this engine's enhancement was added in B&R 6.1 Patch 1.
we just bought Veeam, I wasn't using it for offsite backups before. I had a demo of 6.1, but didn't start doing WAN backups until after I installed 6.5

The connection is a 25Mb cable modem at a branch office. Otherwise rock solid. Connected to our HQ via Cisco ASA site-to-site VPN. The VPN connection is rock stable (currently shows it has been established for 53d : 5h). The users at the remote site use several servers that reside in the HQ, as well as Cisco IP phones, where Call Manager servers are in the HQ. No problems at all with anything else.

I know cable internet can be spotty, but seems strange that only Veeam is having issue. I would expect to see the VPN connection flap if there were serious issues.

aeccles
Enthusiast
Posts: 80
Liked: 6 times
Joined: May 01, 2012 3:00 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by aeccles » Feb 19, 2013 3:05 pm

Gostev wrote:If you are always gettin [0] at the end of the error, then I'd say this is definitely completely different issue (not what was being discussed in this thread above). WAN issues would cause this number to be random every time the job fails, because connection drop is random.
Hi Gostev,

I just got a few of these messages that ended with [0] for the same job over the weekend. My jobs are set to auto-retry, and this error does not come up the second time the job runs. Any ideas what might be causing it? My repository is a windows server and it's going over a WAN link via VPN. I monitor that link and don't show any packet loss or high latency.

*edit - forgot to mention I am using 6.5
thanks

Thanks,
Aaron

foggy
Veeam Software
Posts: 18366
Liked: 1575 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to retrieve next block transmission command

Post by foggy » Feb 19, 2013 3:18 pm

Aaron, have you already contacted support with this issue?

aeccles
Enthusiast
Posts: 80
Liked: 6 times
Joined: May 01, 2012 3:00 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by aeccles » Feb 19, 2013 4:43 pm

foggy wrote:Aaron, have you already contacted support with this issue?
Hi foggy,

No I have not done that. I was hoping someone had a some background on this and I could save a little time :)

If that's what I should do I'll go ahead and open a case.

Thanks,
Aaron

jpeake
Enthusiast
Posts: 88
Liked: 25 times
Joined: Sep 25, 2012 7:57 pm
Contact:

Re: Unable to retrieve next block transmission command

Post by jpeake » Feb 19, 2013 4:50 pm

I worked with support re: the issue and never got it resolved. WAN connection is the scape-goat. It usually completes during the retry, so i just live with it.

RLPITSOttawa
Lurker
Posts: 1
Liked: never
Joined: Mar 19, 2013 1:33 pm
Full Name: RLPITS Ottawa
Contact:

[MERGED] : Failed backups

Post by RLPITSOttawa » Mar 19, 2013 2:01 pm

Hi,

We've been having an issue with one of our backups. We have one backup for extreemly vital information that is set to run every hour that usually works but sporadically it fails with the error, Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. On occasion the houerly backups will fail consequtively 2 or three times in a row but usually the failiure occurs once a day.

This is on a LAN connection backing up a virtual SERVER2008 machine with the veeam backup software SERVER2008 running on the same ESX host backing up to another non-virtualized backup server running SERVER2008 on the same LAN

Need some insight on this matter

veremin
Product Manager
Posts: 17005
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Unable to retrieve next block transmission command

Post by veremin » Mar 19, 2013 2:19 pm

As it’s been already mentioned, this issue is almost certainly related to the network drops. So, if I were you, I would doublecheck it first.

Furthermore, feel free to open a ticket with our support team and let them take a closer look at your backup infrastructure.

Hope this helps.
Thanks.

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Unable to retrieve next block transmission command

Post by zak2011 » May 07, 2013 11:35 am 1 person likes this post

I am getting the same error for the past two days for my exchange VM. However I have not observed any network drops. A case has been also opened for the same.

foggy
Veeam Software
Posts: 18366
Liked: 1575 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to retrieve next block transmission command

Post by foggy » May 07, 2013 11:43 am

Arun, kindly please update this thread with the results of the investigation (as you usually do ;)), as we are very interested to hear the outcome. Thanks!

Gostev
SVP, Product Management
Posts: 24936
Liked: 3621 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Unable to retrieve next block transmission command

Post by Gostev » May 07, 2013 7:22 pm

...or just include the support case ID so we could follow your support case.

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Case # 00203616

Post by zak2011 » May 07, 2013 8:44 pm

Thanks Gostev..mentioned the case id above... I am sure support will definitely help in resolving this as always :)

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Unable to retrieve next block transmission command

Post by zak2011 » May 23, 2013 12:54 pm 1 person likes this post

Okay, support had asked me to use Wireshark to capture network packets from the source proxy. I am not sure if they could find anything from this.
However the job using a Linux proxy as a backup repository. This VM was using the VMXnet3 driver. Network drops were observed between the source proxy and the Linux proxy.
After changing the drivers from VMXnet3 to E1000, the job seems to be running proerly.
However it seems unknown the reason as to why the job used to work previously with the VMXnet3 driver and had to be changed.

Thanks!

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Unable to retrieve next block transmission command

Post by zak2011 » May 24, 2013 12:28 pm 3 people like this post

Here the facts laid out by support which resolved the issue.

a) As per the logs sent the connection broke between the source and the target agent.
b) based on the setup we figured out, that the problem is most likely in the target.
c) based on the fact that I saw more issues with vmxnet3 NICs than E1000 I recommended to try switching the NIC type which has helped.
d) Now based on the Fact that changin the NIC type help, we can guess that most likely it is either the driver or the OS issue.
e) My next idea was to redeploy the linux VM from other distributive let's say CentOS and assign it a E1000 NIC. And see how it handles the incoming network IO.

Hope this helps,

Thanks

Post Reply

Who is online

Users browsing this forum: backupquestions, roterdan and 20 guests