Unable to retrieve next block transmission command

VMware specific discussions

[MERGED] Replication errors: Number of already processed blo

Veeam Logoby bdoellefeld » Mon Jun 10, 2013 3:00 pm

My case # is 00248715

Curious if anyone has seen his repetitive error before. I have been seeing more and more errors like this for the past few months and it seems to be increasing in frequency. They appear to be random failures but always the same reason. End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Job retry's generally succeed.

I've been looking at the network side (WAN, 45Mb MPLS source / 30Mb MPLS target) for quite a while and although when replications are occurring there is some increase in latency the network is stable.

Curiously however is this right here: Number of already processed blocks: [0]. It’s always zero. If it was network and random, I would think this number would be different every time it happens. Also, with these and other failed jobs for other VMs, one vmdk will replicate but additional vmdk’s attached on these VMs will fail with this same error.

The last word I got from support was that I need to make sure "Antivirus is not locking the drive"(?) or other whole disk operations are not interfering.

All logs have been uploaded to support.
bdoellefeld
Influencer
 
Posts: 22
Liked: never
Joined: Wed Nov 02, 2011 5:19 pm
Location: Colorado, USA
Full Name: Bill Doellefeld

Re: [MERGED] Replication errors: Number of already processed

Veeam Logoby foggy » Mon Jun 10, 2013 3:26 pm

bdoellefeld wrote:Curiously however is this right here: Number of already processed blocks: [0]. It’s always zero. If it was network and random, I would think this number would be different every time it happens.

This is definitely a correct assumption.

Bill, please review this topic for some hints and continue working with support on this issue.
foggy
Veeam Software
 
Posts: 15282
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to retrieve next block transmission command

Veeam Logoby zak2011 » Tue Jun 11, 2013 6:52 am

My case #00203616

The error seems to be very intermittent. I was asked to run tshark and capture the packets so that support can investigate this further.
zak2011
Expert
 
Posts: 367
Liked: 41 times
Joined: Tue May 15, 2012 2:21 pm
Full Name: Arun

Re: Unable to retrieve next block transmission command

Veeam Logoby bdoellefeld » Wed Jun 12, 2013 7:48 pm

Not much progress made here. I haven't been asked to do any packet captures yet. The latest is I'm being asked to disable the multiple TCP/IP connections per job setting. I have this error maybe 1 out of 15 replications and am pretty sure I am going to regret changing that.
bdoellefeld
Influencer
 
Posts: 22
Liked: never
Joined: Wed Nov 02, 2011 5:19 pm
Location: Colorado, USA
Full Name: Bill Doellefeld

Re: Unable to retrieve next block transmission command

Veeam Logoby eric.siedentopf » Wed Jun 12, 2013 10:27 pm 1 person likes this post

I have the same issue after a upgrade to 6.5 last week.

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. it was always [0].

I have never seen this error before with Veeam 5.02 running Backups for a few years and I also have Veeam 6.5 pulling Replication to my DR site without this error for at least 9 months.

All of the hardware is identical all I did was upgrade to Veeam 6.5. I am not using Proxies at this time it is all being ran from the Physical Veeam Server (Proxy, Repo).

I have had two different VMs fail with this error and succeeded on the retry and they are in separate jobs at separate times I have 3-7 VMs per job and no other VMs got this error.

VMware 4.1
NBD Mode (always has been)
NameStatusStart time End time Size Read Transferred Duration
FridayVS-IMG1Success8:05:50 PM8:07:56 PM530.0 GB717.0 MB263.2 MB0:02:06
YesterdayVS-IMG1Error8:06:13 PM8:07:45 PM530.0 GB0.0 KB0.0 KB0:01:31
RetryVS-IMG1Success8:18:46 PM8:20:31 PM530.0 GB1.2 GB443.8 MB0:01:44
eric.siedentopf
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon May 02, 2011 10:40 pm
Full Name: Eric Siedentopf

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Sun Jun 16, 2013 7:37 pm

Hi Eric, do you have this error confirming for the same VMs all the time? I wonder if there is anything in common between them?
Vitaliy S.
Veeam Software
 
Posts: 19954
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Unable to retrieve next block transmission command

Veeam Logoby zak2011 » Mon Jun 17, 2013 2:17 pm 1 person likes this post

Okay..the latest and quite an interesting update I have now about this problem is the 'Network Traffic Control' feature triggers some false positives due to which these errors come up. Support has disabled this feature by adding registry keys 'EnableTrafficControl' to zero and also 'VddkLogLevel' to zero ( not to be done without supervision of support). According to the support experts, this functionality has been improved much better in v7. Hopefully this should help.

Thanks!
zak2011
Expert
 
Posts: 367
Liked: 41 times
Joined: Tue May 15, 2012 2:21 pm
Full Name: Arun

Re: Unable to retrieve next block transmission command

Veeam Logoby eric.siedentopf » Mon Jun 17, 2013 3:24 pm

Vitaliy S. wrote:Hi Eric, do you have this error confirming for the same VMs all the time? I wonder if there is anything in common between them?


It has only happened once to each VM. They are on Diff LUNs, Diff OS, Diff Jobs and Diff Times. They also failed on diff days. They are on the same Host but there are also 8-10 other VM on there as well.
eric.siedentopf
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon May 02, 2011 10:40 pm
Full Name: Eric Siedentopf

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Mon Jun 17, 2013 3:50 pm

Can you please contact our technical guys and try to implement the workaround mentioned by Arun in the post above?
Vitaliy S.
Veeam Software
 
Posts: 19954
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Unable to retrieve next block transmission command

Veeam Logoby eric.siedentopf » Mon Jun 17, 2013 3:54 pm

I will.

Thank you
eric.siedentopf
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon May 02, 2011 10:40 pm
Full Name: Eric Siedentopf

Re: Unable to retrieve next block transmission command

Veeam Logoby yizhar » Mon Jun 17, 2013 11:16 pm

Hi.

Just want to add that I'm experiencing same issue randomly.

I have seen it on 2 totally different systems, in Backup and also in Replication jobs.

Details about one instance of the problem:

Error message:
Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0].
After retry it works, but every several days one different VM might have this or similar problem (I don't always have time to investigate further).

VBR server = physical Dell R610 (acting as the proxy and repository).
VBR version = 6.5.0.144
Production storage = HP 3PAR 7200 with 4 FC connections.
Production environment = 3 hosts, VMware esxi 5.1 update 1.
Backup job in SAN mode (using FC card on backup server), using reverse incremental.
Backup Repository = IBM DS3300 iscsi storage, mounted directly by VBR server.

I have also received report from another client of us, he had the same error but with replication job to remote site.
He has an HP P4300G2 (lefthand) ISCSI SAN on production site.
So I assume the problem is related to reading data from source storage.

Maybe the problem is related to SAN mode?

Bye
Yizhar
yizhar
Expert
 
Posts: 179
Liked: 48 times
Joined: Mon Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz

Re: Unable to retrieve next block transmission command

Veeam Logoby yizhar » Mon Jun 17, 2013 11:23 pm

Hi.

Just to make it clear regarding my previous post:
I am experiencing the problem with no WAN connection involved, for example:
HP 3PAR storage -- FC connection -- FC HBA on physical backup server (proxy and repository) -- NIC -- iscsi switch -- IBM DS3300 NTFS LUN as target.

My client also seen it with replication over WAN on totally different environment.

Yizhar
yizhar
Expert
 
Posts: 179
Liked: 48 times
Joined: Mon Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz

Re: Unable to retrieve next block transmission command

Veeam Logoby foggy » Tue Jun 18, 2013 10:09 am

Yizhar, thank you for the valuable feedback. Have you got a chance to provide this information to our support guys? I assume they are extremely interested in getting data about occurrences of this error in different environments. Would be much appreciated!
foggy
Veeam Software
 
Posts: 15282
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

[MERGED] : Last Hard Disk on a VM reads 0.0 KB at 0 KB/s [CB

Veeam Logoby Virtualizer » Thu Jun 27, 2013 9:45 am

Hi everyone,

I'm having a strange problem with my backup (Using Backup & Replicate 6.5.0.109)
Been looking at similar prompts floating around here but found different situation than mine.

The backup log is:
6/26/2013 9:20:58 PM :: Queued for processing at 6/26/2013 9:20:58 PM
6/26/2013 9:20:58 PM :: Required backup infrastructure resources have been assigned
6/26/2013 9:21:00 PM :: VM processing started at 6/26/2013 9:21:00 PM
6/26/2013 9:21:00 PM :: VM size: 1.8 TB (169.3 GB used)
6/26/2013 9:21:01 PM :: Using source proxy 10.200.120.118 [hotadd]
6/26/2013 9:21:13 PM :: Indexing guest file system
6/26/2013 9:22:56 PM :: Preparing guest for hot backup
6/26/2013 9:23:03 PM :: Creating snapshot
6/26/2013 9:23:11 PM :: Releasing guest
6/26/2013 9:23:28 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.vmx'
6/26/2013 9:23:32 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.vmxf'
6/26/2013 9:23:34 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.nvram'
6/26/2013 9:23:37 PM :: Hard Disk 1 (80.0 GB) 1.3 GB read at 24 MB/s [CBT]
6/26/2013 9:24:35 PM :: Hard Disk 2 (800.0 GB) 7.1 GB read at 106 MB/s [CBT]
6/26/2013 9:25:43 PM :: Hard Disk 3 (100.0 GB) 1.0 MB read at 106 KB/s [CBT]
6/26/2013 9:25:53 PM :: Hard Disk 4 (200.0 GB) 960.0 MB read at 68 MB/s [CBT]
6/26/2013 9:26:07 PM :: Hard Disk 5 (600.0 GB) 1.1 GB read at 61 MB/s [CBT]
6/26/2013 9:26:25 PM :: Hard Disk 6 (20.0 GB) 57.0 MB read at 6 MB/s [CBT]
6/26/2013 9:26:35 PM :: Hard Disk 7 (0.0 KB) 0.0 KB read at 0 KB/s [CBT]
6/26/2013 9:26:41 PM :: Removing VM snapshot
6/26/2013 9:33:07 PM :: Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0].
6/26/2013 9:33:07 PM :: Network traffic verification detected no corrupted blocks
6/26/2013 9:33:07 PM :: Busy: Source 79% > Proxy 28% > Network 10% > Target 6%
6/26/2013 9:33:07 PM :: Primary bottleneck: Source
6/26/2013 9:33:08 PM :: Processing finished with errors at 6/26/2013 9:33:07 PM

What the hell is wrong all of a sudden with Disk 7?

I've had more VMs backed up succesfuly right after this one and prioer to this one at the same night.

Can't find anything wrong in there...

Usualy this backup always ends without any errors.

What can you advice me to check or do ?

Kind reagards,

-Gary
Virtualizer
Influencer
 
Posts: 10
Liked: never
Joined: Thu Jun 13, 2013 6:47 am
Full Name: Gary Golbraich

Re: Unable to retrieve next block transmission command

Veeam Logoby v.Eremin » Thu Jun 27, 2013 9:50 am

Hi, Gary.

You have been merged to the existing discussion regarding similar issues. Typically you might get the aforesaid error due to unstable network connection. Additional information, as well as, other potential causes can be found above; so, please take a look.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13708
Liked: 1024 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 1 guest