Unable to retrieve next block transmission command

[Mark Muyskens]

Hello,

I am trying to setup Veeam to use an Amazon EC2 instance as a backup repository and I am running into the following type of errors;

Error: Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [210419].
Error: Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [316108].

Is anyone familiar with this type of setup? Does this seem like a network type issue? Would moving from a micro instance to a larger instance at Amazon provide me with more network resources and resolve this issue?

Anyone care to take a stab?

[foggy]

Yes, usually this type of errors is caused by the unstable network connection.

Please include your support case ID number as explained when you click New Topic.

[garufo]

Hello,

We are having several backup fails in our environment (Case # 00134265). We have a Veeam Backup 6.1 patch 1 (virtual machine), and we use a Storage Deduplication Appliance (Fujitsu CS800) as a repository, attached by as a NFS resource to a Red Hat Linux virtual machine in the same VMware 4.1 environment.

The CS800 NFS resource is with the deduplication active. 2 weeks ago we had errors in the backups that we assumed that were about the Storage Array where the virtual machines are running because we had an incident in the storage. For this reason we had to reconfigure all the backup tasks in a no deduplication resource, all the jobs where running in the right way. Last week we where "moving" the backup files to a new repositoy in the CS800, another NFS resource in the Red Hat virtual machine, but with deduplication active. This weeked had started again the errors:

Error: Cliente error: Connection reset by peer. Unable to retrieve next block transmission command. Number of already processed blocks: [9811]
Busy: Source 13%> Proxy> 6%> Target 89%

(and similiar errors always, changing the number of processed blocks). Sometimes it stopped in a virtual disk, sometimes in another. There are 2 tasks at least with that error.

We had do the following troubleshooting:

- Disable multiple TCP/IP in global traffic throttling.
- Backup task options: Inline deduplication: disabled. Compression: NONE. Backup type: LAN

We are now again with Veeam support trying to find out what is happening.

Any suggest??

thanks in advanced

[Gostev]

Most likely, this is caused by network connection drops.

[garufo]

Thank you, we are trying a troubleshooting about it,

thanks

[George]

Hi Guys,

I have seen other posts regarding this error but was wondering if you could help me out.

I have a Veeam Server running on Windows with a Linux repository, I'm running on 6.1 and at the moment i get this error during the middle of the job. Its running just fine for a while then stops. Not sure what could be causing it everything on the network looks fine.

The errors message i get is: End of file: Unable to retrieve next block transmission command. Number of already processed blocks:

Any thoughts would be much appreciated!

Thanks!

[mgd5]

Hi,

We experience the exact same error. have you found any resolution to the issue

Code: Select all

Error: Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [723354]

MGD

[Vitaliy S.]

Hi Christer, what is the network connection to your repository? Are you using a WAN link?

[mgd5]

Hi

No it´s set to LAN Target.

[Vitaliy S.]

Yes, but what is your backup repository? Where is it located? According to the error message, the connection doesn't seem to be very stable.

[mgd5]

Hi

Our Backup Rep is a Linux gateway witch has mounted NFS share from our Datadomain box. I have seen the same error when we had the linux gateway had a NFS mount to a physical linux box. The backuprepositorys we use are virtual.

Christer

[jpeake]

I am having this same error in 6.5 when backing up to our remote site over WAN. It happens on one or two servers per job. If I retry, sometimes it works, sometimes it doesn't.

Working with support, they advised to use network mode on the proxy for that job (i set up a new proxy in network mode, and pointed that job at it). I tried this over the weekend, and sure enough the job completed without error.

Curious to see if this actually fixes it. Will report back after a few more jobs run

[topry]

We have started seeing these errors since the upgrade to 6.5 (not before) and no other upgrades/changes other than the upgrade.
We receive this error primarily while backing up one VM, yet it shares the same SAN/iSCSI network as all of our 3 hosts and the frequency is never more than once per day, 4-5 days out of 7.

All backup sets are using SAN and other than this periodic issue, everything else works normally and test restores before/after this error show no problems. Veeam is installed withing a VM and the host and SAN are connected via dedicated gigabit switch for all iSCSI traffic with 6 ports on the SAN and 4 each per host. No other indications of network issues.

The specific error we receive is:

Code: Select all

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]

If anyone has any suggestions on what else to check/how to track this further.

[jpeake]

I tried running in network mode, as suggested by support. It worked over the weekend, but last night's inc failed with the same error. So i am back to the drawing board.

This has only ever happened on my "offiste" job, which is across a WAN. The local job has been solid

[Gostev]

If you are always gettin [0] at the end of the error, then I'd say this is definitely completely different issue (not what was being discussed in this thread above). WAN issues would cause this number to be random every time the job fails, because connection drop is random.

[jpeake]

If you are always gettin [0] at the end of the error, then I'd say this is definitely completely different issue (not what was being discussed in this thread above). WAN issues would cause this number to be random every time the job fails, because connection drop is random.

makes sense. my number is always some random block

[Gostev]

In that case, this is clearly WAN link issue that did not exist before. In fact, 6.5 can sustain much worse packet loss than all previous B&R versions, so the issues must be real bad comparing to how this one link used to work when you had previous B&R versions installed. To be precise, this engine's enhancement was added in B&R 6.1 Patch 1.

[jpeake]

In that case, this is clearly WAN link issue that did not exist before. In fact, 6.5 can sustain much worse packet loss than all previous B&R versions, so the issues must be real bad comparing to how this one link used to work when you had previous B&R versions installed. To be precise, this engine's enhancement was added in B&R 6.1 Patch 1.

we just bought Veeam, I wasn't using it for offsite backups before. I had a demo of 6.1, but didn't start doing WAN backups until after I installed 6.5

The connection is a 25Mb cable modem at a branch office. Otherwise rock solid. Connected to our HQ via Cisco ASA site-to-site VPN. The VPN connection is rock stable (currently shows it has been established for 53d : 5h). The users at the remote site use several servers that reside in the HQ, as well as Cisco IP phones, where Call Manager servers are in the HQ. No problems at all with anything else.

I know cable internet can be spotty, but seems strange that only Veeam is having issue. I would expect to see the VPN connection flap if there were serious issues.

[aeccles]

If you are always gettin [0] at the end of the error, then I'd say this is definitely completely different issue (not what was being discussed in this thread above). WAN issues would cause this number to be random every time the job fails, because connection drop is random.

Hi Gostev,

I just got a few of these messages that ended with [0] for the same job over the weekend. My jobs are set to auto-retry, and this error does not come up the second time the job runs. Any ideas what might be causing it? My repository is a windows server and it's going over a WAN link via VPN. I monitor that link and don't show any packet loss or high latency.

*edit - forgot to mention I am using 6.5
thanks

Thanks,
Aaron

[foggy]

Aaron, have you already contacted support with this issue?

[aeccles]

Aaron, have you already contacted support with this issue?

Hi foggy,

No I have not done that. I was hoping someone had a some background on this and I could save a little time

If that's what I should do I'll go ahead and open a case.

Thanks,
Aaron

[jpeake]

I worked with support re: the issue and never got it resolved. WAN connection is the scape-goat. It usually completes during the retry, so i just live with it.

[RLPITSOttawa]

Hi,

We've been having an issue with one of our backups. We have one backup for extreemly vital information that is set to run every hour that usually works but sporadically it fails with the error, Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. On occasion the houerly backups will fail consequtively 2 or three times in a row but usually the failiure occurs once a day.

This is on a LAN connection backing up a virtual SERVER2008 machine with the veeam backup software SERVER2008 running on the same ESX host backing up to another non-virtualized backup server running SERVER2008 on the same LAN

Need some insight on this matter

[veremin]

As it’s been already mentioned, this issue is almost certainly related to the network drops. So, if I were you, I would doublecheck it first.

Furthermore, feel free to open a ticket with our support team and let them take a closer look at your backup infrastructure.

Hope this helps.
Thanks.

[zak2011]

I am getting the same error for the past two days for my exchange VM. However I have not observed any network drops. A case has been also opened for the same.

[foggy]

Arun, kindly please update this thread with the results of the investigation (as you usually do ), as we are very interested to hear the outcome. Thanks!

[Gostev]

...or just include the support case ID so we could follow your support case.

[zak2011]

Thanks Gostev..mentioned the case id above... I am sure support will definitely help in resolving this as always

[zak2011]

Okay, support had asked me to use Wireshark to capture network packets from the source proxy. I am not sure if they could find anything from this.
However the job using a Linux proxy as a backup repository. This VM was using the VMXnet3 driver. Network drops were observed between the source proxy and the Linux proxy.
After changing the drivers from VMXnet3 to E1000, the job seems to be running proerly.
However it seems unknown the reason as to why the job used to work previously with the VMXnet3 driver and had to be changed.

Thanks!

[zak2011]

Here the facts laid out by support which resolved the issue.

a) As per the logs sent the connection broke between the source and the target agent.
b) based on the setup we figured out, that the problem is most likely in the target.
c) based on the fact that I saw more issues with vmxnet3 NICs than E1000 I recommended to try switching the NIC type which has helped.
d) Now based on the Fact that changin the NIC type help, we can guess that most likely it is either the driver or the OS issue.
e) My next idea was to redeploy the linux VM from other distributive let's say CentOS and assign it a E1000 NIC. And see how it handles the incoming network IO.

Hope this helps,

Thanks