Unable to retrieve next block transmission command

[bdoellefeld]

My case # is 00248715

Curious if anyone has seen his repetitive error before. I have been seeing more and more errors like this for the past few months and it seems to be increasing in frequency. They appear to be random failures but always the same reason. End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Job retry's generally succeed.

I've been looking at the network side (WAN, 45Mb MPLS source / 30Mb MPLS target) for quite a while and although when replications are occurring there is some increase in latency the network is stable.

Curiously however is this right here: Number of already processed blocks: [0]. It’s always zero. If it was network and random, I would think this number would be different every time it happens. Also, with these and other failed jobs for other VMs, one vmdk will replicate but additional vmdk’s attached on these VMs will fail with this same error.

The last word I got from support was that I need to make sure "Antivirus is not locking the drive"(?) or other whole disk operations are not interfering.

All logs have been uploaded to support.

[foggy]

Curiously however is this right here: Number of already processed blocks: [0]. It’s always zero. If it was network and random, I would think this number would be different every time it happens.

This is definitely a correct assumption.

Bill, please review this topic for some hints and continue working with support on this issue.

[zak2011]

My case #00203616

The error seems to be very intermittent. I was asked to run tshark and capture the packets so that support can investigate this further.

[bdoellefeld]

Not much progress made here. I haven't been asked to do any packet captures yet. The latest is I'm being asked to disable the multiple TCP/IP connections per job setting. I have this error maybe 1 out of 15 replications and am pretty sure I am going to regret changing that.

[eric.siedentopf]

I have the same issue after a upgrade to 6.5 last week.

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. it was always [0].

I have never seen this error before with Veeam 5.02 running Backups for a few years and I also have Veeam 6.5 pulling Replication to my DR site without this error for at least 9 months.

All of the hardware is identical all I did was upgrade to Veeam 6.5. I am not using Proxies at this time it is all being ran from the Physical Veeam Server (Proxy, Repo).

I have had two different VMs fail with this error and succeeded on the retry and they are in separate jobs at separate times I have 3-7 VMs per job and no other VMs got this error.

VMware 4.1
NBD Mode (always has been)
Name Status Start time End time Size Read Transferred Duration
Friday VS-IMG1 Success 8:05:50 PM 8:07:56 PM 530.0 GB 717.0 MB 263.2 MB 0:02:06
Yesterday VS-IMG1 Error 8:06:13 PM 8:07:45 PM 530.0 GB 0.0 KB 0.0 KB 0:01:31
Retry VS-IMG1 Success 8:18:46 PM 8:20:31 PM 530.0 GB 1.2 GB 443.8 MB 0:01:44

[Vitaliy S.]

Hi Eric, do you have this error confirming for the same VMs all the time? I wonder if there is anything in common between them?

[zak2011]

Okay..the latest and quite an interesting update I have now about this problem is the 'Network Traffic Control' feature triggers some false positives due to which these errors come up. Support has disabled this feature by adding registry keys 'EnableTrafficControl' to zero and also 'VddkLogLevel' to zero ( not to be done without supervision of support). According to the support experts, this functionality has been improved much better in v7. Hopefully this should help.

Thanks!

[eric.siedentopf]

Hi Eric, do you have this error confirming for the same VMs all the time? I wonder if there is anything in common between them?

It has only happened once to each VM. They are on Diff LUNs, Diff OS, Diff Jobs and Diff Times. They also failed on diff days. They are on the same Host but there are also 8-10 other VM on there as well.

[Vitaliy S.]

Can you please contact our technical guys and try to implement the workaround mentioned by Arun in the post above?

[eric.siedentopf]

I will.

Thank you

[yizhar]

Hi.

Just want to add that I'm experiencing same issue randomly.

I have seen it on 2 totally different systems, in Backup and also in Replication jobs.

Details about one instance of the problem:

Error message:
Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0].
After retry it works, but every several days one different VM might have this or similar problem (I don't always have time to investigate further).

VBR server = physical Dell R610 (acting as the proxy and repository).
VBR version = 6.5.0.144
Production storage = HP 3PAR 7200 with 4 FC connections.
Production environment = 3 hosts, VMware esxi 5.1 update 1.
Backup job in SAN mode (using FC card on backup server), using reverse incremental.
Backup Repository = IBM DS3300 iscsi storage, mounted directly by VBR server.

I have also received report from another client of us, he had the same error but with replication job to remote site.
He has an HP P4300G2 (lefthand) ISCSI SAN on production site.
So I assume the problem is related to reading data from source storage.

Maybe the problem is related to SAN mode?

Bye
Yizhar

[yizhar]

Hi.

Just to make it clear regarding my previous post:
I am experiencing the problem with no WAN connection involved, for example:
HP 3PAR storage -- FC connection -- FC HBA on physical backup server (proxy and repository) -- NIC -- iscsi switch -- IBM DS3300 NTFS LUN as target.

My client also seen it with replication over WAN on totally different environment.

Yizhar

[foggy]

Yizhar, thank you for the valuable feedback. Have you got a chance to provide this information to our support guys? I assume they are extremely interested in getting data about occurrences of this error in different environments. Would be much appreciated!

[Virtualizer]

Hi everyone,

I'm having a strange problem with my backup (Using Backup & Replicate 6.5.0.109)
Been looking at similar prompts floating around here but found different situation than mine.

The backup log is:
6/26/2013 9:20:58 PM :: Queued for processing at 6/26/2013 9:20:58 PM
6/26/2013 9:20:58 PM :: Required backup infrastructure resources have been assigned
6/26/2013 9:21:00 PM :: VM processing started at 6/26/2013 9:21:00 PM
6/26/2013 9:21:00 PM :: VM size: 1.8 TB (169.3 GB used)
6/26/2013 9:21:01 PM :: Using source proxy 10.200.120.118 [hotadd]
6/26/2013 9:21:13 PM :: Indexing guest file system
6/26/2013 9:22:56 PM :: Preparing guest for hot backup
6/26/2013 9:23:03 PM :: Creating snapshot
6/26/2013 9:23:11 PM :: Releasing guest
6/26/2013 9:23:28 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.vmx'
6/26/2013 9:23:32 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.vmxf'
6/26/2013 9:23:34 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.nvram'
6/26/2013 9:23:37 PM :: Hard Disk 1 (80.0 GB) 1.3 GB read at 24 MB/s [CBT]
6/26/2013 9:24:35 PM :: Hard Disk 2 (800.0 GB) 7.1 GB read at 106 MB/s [CBT]
6/26/2013 9:25:43 PM :: Hard Disk 3 (100.0 GB) 1.0 MB read at 106 KB/s [CBT]
6/26/2013 9:25:53 PM :: Hard Disk 4 (200.0 GB) 960.0 MB read at 68 MB/s [CBT]
6/26/2013 9:26:07 PM :: Hard Disk 5 (600.0 GB) 1.1 GB read at 61 MB/s [CBT]
6/26/2013 9:26:25 PM :: Hard Disk 6 (20.0 GB) 57.0 MB read at 6 MB/s [CBT]
6/26/2013 9:26:35 PM :: Hard Disk 7 (0.0 KB) 0.0 KB read at 0 KB/s [CBT]
6/26/2013 9:26:41 PM :: Removing VM snapshot
6/26/2013 9:33:07 PM :: Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0].
6/26/2013 9:33:07 PM :: Network traffic verification detected no corrupted blocks
6/26/2013 9:33:07 PM :: Busy: Source 79% > Proxy 28% > Network 10% > Target 6%
6/26/2013 9:33:07 PM :: Primary bottleneck: Source
6/26/2013 9:33:08 PM :: Processing finished with errors at 6/26/2013 9:33:07 PM

What the hell is wrong all of a sudden with Disk 7?

I've had more VMs backed up succesfuly right after this one and prioer to this one at the same night.

Can't find anything wrong in there...

Usualy this backup always ends without any errors.

What can you advice me to check or do ?

Kind reagards,

-Gary

[veremin]

Hi, Gary.

You have been merged to the existing discussion regarding similar issues. Typically you might get the aforesaid error due to unstable network connection. Additional information, as well as, other potential causes can be found above; so, please take a look.

Thanks.

[wagnerj]

Hi,
I have searched the forums and have not found an answer. I also called support, and what they have asked me to do has not worked.

Here is the situation, I have two Veeam environments. I am backing up a clustered Hyper-V environment with 3 nodes. The nodes are connected to an IBM Storwize v7000 SAN. One Veeam server is the test environment which I still get perfect backups on every night (Old Gen 3 HP DL360). Keep in mind these are backing up to 4 external 1TB drives like a champ! The other environment is a high powered IBM server I just purchased just for Veeam with 20TB of Internal local storage on a RAID 5. I am able to take a full backup of my 4TB exchange environment on the new server, but reverse incremental fail with the following error:

Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [27987]

It fails each time at 25% on the second disk. Keep in mind it works perfectly on the old system. The backup is being performed over a LAN, on the same VLAN, connected to the same Cisco 3750 X. The port configs are the same as the test server. I did what Veeam support asked and throttle the connection by 60% and that did not work either. I also tried to re-seed the intital backup, I get the same results.
Can Anyone Help?
Jim

[Vitaliy S.]

Hi Jim,

I also tried to re-seed the intital backup, I get the same results.

Does it also fail on the same 25%? Is this the only VM affected by this behavior?

Thanks!

[wagnerj]

It's 23% now, to reiderate the old Veeam server just completed an incremental with no issue. Its a different change block reported each time. No all VM incrementals are affected.

[wagnerj]

I am running a ping to the VM being backed up, I would like to see if any network latency or timeouts occur. I am running and monitoring the job now.

Update:
I just re-ran the job that was failing, I had a ping running to the VM being backed up, the Veeam server and the hyper-v node. All pings stayed at <1ms the entire time.

[Vitaliy S.]

Hmm...I believe deeper logs investigation is required, as there might be something wrong with a Windows host where new Veeam B&R server is installed.

[DaFresh]

FYI, removing traffic throttling and disable 'multiple TCP connection per job' (which I don't know if it is used when traffic throttling is disabled) seams to solve my problem.
Hope it helps.

[veremin]

Thanks, Cedric, for sharing the information with the community; much appreciated. And yes, “use multiple TCP/IP connection” can be enabled/disabled regardless of throttling rules. Thanks.

[DaFresh]

As a complement, this error was encountered on a replication job, not a backup, I would think that traffic throttling and multiple TCP/IP connection are only used by replication jobs, which means most of the time remote site/different ip range, except maybe when proxy are used for backup ?

[Vitaliy S.]

TCP/IP connection configuration applies to both backup and replication jobs. Looks like there were issues between your proxy servers used on the target and source sites.

[veremin]

I would think that traffic throttling and multiple TCP/IP connection are only used by replication jobs

The situation is slightly different .Veeam B&R is using two-agent based architecture. And all of features like throttling rules, compression, etc, are implemented between them. In case of remote replication, these features would be implemented between two proxy servers; in case of backup, between a backup proxy and a repository; in case of backup to an SMB share between a backup proxy and a proxying Windows server.

Thanks.

[bnssite]

I kept getting "Unable to retrieve next block transmission command. Number of already processed blocks: [0]." I then built a FreeNAS box so I could have a different repository to send my Veeam 6.5 data backups to, and it worked.

[Jamie Pert]

If you get the following error:

Error: Client error: An existing connection was forcibly closed by the remote host
Exception from server: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Unable to retrieve next block transmission command. Number of already processed blocks: [0].
Failed to perform data backup.

Chances are you are using the "direct" option on a Veeam copy job to an offsite repository. Immediately switching to WAN accelerators worked fine. It's because the direct option is less resilient to factors often encountered with offsite storage aka delays / slow speeds.

What's weird is this only affected a few VMs in our infrastructure. On the same source storage to the same destination storage roughly 80% of our copy jobs were fine, but certain ones just failed, often very quickly. We tried different times of day, new jobs and much more and it just wouldn't work.

[leebtish]

Hi all,

Since upgrading to version 7 (patch 1) we have received the following error for many VM's: "Error: Client error: An existing connection was forcibly closed by the remote host Exception from server: An existing connection was forcibly closed by the remote host Unable to retrieve next block transmission command. Number of already processed blocks: [359]. Failed to perform data backup." Note - the processed blocks number is different for each VM.

The backups are successful on subsequent retries but we never received this error on version 6.5. We always had successful backups. We're seeing the errors on a daily basis now. I've raised a case with support and their initial response was to throttle the bandwidth as they state this error can occur in unstable networks. Thing is, this is on a gigabit LAN and we never had any problems before the upgrade so I find it hard to accept that it's a network issue.

Just wondering if anyone else come across this problem since upgrading to version 7 ? At the moment I'm considering rolling back to 6.5 if possible.

Thanks in advance

Lee

[foggy]

Lee, looks like you're the first one to report this error for v7 backup jobs here on forums. This error indeed typically relates to network connection issues, you may want to review this topic for other occurrences of the same error and probably find some hints that helped other users in similar situations. There is also a dedicated KB article regarding this kind of errors, worth reviewing: http://www.veeam.com/kb1781.

[Jamie Pert]

We had that on offsite copy jobs lee. Once again opting to use WAN accelerators fixed the issue. Obviously this doesn't massively help you as yours is backing up to a location on the same LAN.

However, I'm starting to wonder whether standard backup jobs are more sensitive to connection imperfections (delayed packets etc). We used to use standard copy jobs to backup to offsite repositories, but since Veeam 7 we have had to opt for backup copy jobs with the use of WAN accelerators - this has ran like a dream.