Discussions specific to the VMware vSphere hypervisor
Post Reply
bdoellefeld
Influencer
Posts: 22
Liked: never
Joined: Nov 02, 2011 5:19 pm
Full Name: Bill Doellefeld
Location: Colorado, USA
Contact:

[MERGED] Replication errors: Number of already processed blo

Post by bdoellefeld » Jun 10, 2013 3:00 pm

My case # is 00248715

Curious if anyone has seen his repetitive error before. I have been seeing more and more errors like this for the past few months and it seems to be increasing in frequency. They appear to be random failures but always the same reason. End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Job retry's generally succeed.

I've been looking at the network side (WAN, 45Mb MPLS source / 30Mb MPLS target) for quite a while and although when replications are occurring there is some increase in latency the network is stable.

Curiously however is this right here: Number of already processed blocks: [0]. It’s always zero. If it was network and random, I would think this number would be different every time it happens. Also, with these and other failed jobs for other VMs, one vmdk will replicate but additional vmdk’s attached on these VMs will fail with this same error.

The last word I got from support was that I need to make sure "Antivirus is not locking the drive"(?) or other whole disk operations are not interfering.

All logs have been uploaded to support.

foggy
Veeam Software
Posts: 16702
Liked: 1343 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] Replication errors: Number of already processed

Post by foggy » Jun 10, 2013 3:26 pm

bdoellefeld wrote:Curiously however is this right here: Number of already processed blocks: [0]. It’s always zero. If it was network and random, I would think this number would be different every time it happens.
This is definitely a correct assumption.

Bill, please review this topic for some hints and continue working with support on this issue.

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Unable to retrieve next block transmission command

Post by zak2011 » Jun 11, 2013 6:52 am

My case #00203616

The error seems to be very intermittent. I was asked to run tshark and capture the packets so that support can investigate this further.

bdoellefeld
Influencer
Posts: 22
Liked: never
Joined: Nov 02, 2011 5:19 pm
Full Name: Bill Doellefeld
Location: Colorado, USA
Contact:

Re: Unable to retrieve next block transmission command

Post by bdoellefeld » Jun 12, 2013 7:48 pm

Not much progress made here. I haven't been asked to do any packet captures yet. The latest is I'm being asked to disable the multiple TCP/IP connections per job setting. I have this error maybe 1 out of 15 replications and am pretty sure I am going to regret changing that.

eric.siedentopf
Influencer
Posts: 10
Liked: 1 time
Joined: May 02, 2011 10:40 pm
Full Name: Eric Siedentopf
Contact:

Re: Unable to retrieve next block transmission command

Post by eric.siedentopf » Jun 12, 2013 10:27 pm 1 person likes this post

I have the same issue after a upgrade to 6.5 last week.

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0]. it was always [0].

I have never seen this error before with Veeam 5.02 running Backups for a few years and I also have Veeam 6.5 pulling Replication to my DR site without this error for at least 9 months.

All of the hardware is identical all I did was upgrade to Veeam 6.5. I am not using Proxies at this time it is all being ran from the Physical Veeam Server (Proxy, Repo).

I have had two different VMs fail with this error and succeeded on the retry and they are in separate jobs at separate times I have 3-7 VMs per job and no other VMs got this error.

VMware 4.1
NBD Mode (always has been)
Name Status Start time End time Size Read Transferred Duration
Friday VS-IMG1 Success 8:05:50 PM 8:07:56 PM 530.0 GB 717.0 MB 263.2 MB 0:02:06
Yesterday VS-IMG1 Error 8:06:13 PM 8:07:45 PM 530.0 GB 0.0 KB 0.0 KB 0:01:31
Retry VS-IMG1 Success 8:18:46 PM 8:20:31 PM 530.0 GB 1.2 GB 443.8 MB 0:01:44

Vitaliy S.
Veeam Software
Posts: 21431
Liked: 1274 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Unable to retrieve next block transmission command

Post by Vitaliy S. » Jun 16, 2013 7:37 pm

Hi Eric, do you have this error confirming for the same VMs all the time? I wonder if there is anything in common between them?

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Unable to retrieve next block transmission command

Post by zak2011 » Jun 17, 2013 2:17 pm 1 person likes this post

Okay..the latest and quite an interesting update I have now about this problem is the 'Network Traffic Control' feature triggers some false positives due to which these errors come up. Support has disabled this feature by adding registry keys 'EnableTrafficControl' to zero and also 'VddkLogLevel' to zero ( not to be done without supervision of support). According to the support experts, this functionality has been improved much better in v7. Hopefully this should help.

Thanks!

eric.siedentopf
Influencer
Posts: 10
Liked: 1 time
Joined: May 02, 2011 10:40 pm
Full Name: Eric Siedentopf
Contact:

Re: Unable to retrieve next block transmission command

Post by eric.siedentopf » Jun 17, 2013 3:24 pm

Vitaliy S. wrote:Hi Eric, do you have this error confirming for the same VMs all the time? I wonder if there is anything in common between them?
It has only happened once to each VM. They are on Diff LUNs, Diff OS, Diff Jobs and Diff Times. They also failed on diff days. They are on the same Host but there are also 8-10 other VM on there as well.

Vitaliy S.
Veeam Software
Posts: 21431
Liked: 1274 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Unable to retrieve next block transmission command

Post by Vitaliy S. » Jun 17, 2013 3:50 pm

Can you please contact our technical guys and try to implement the workaround mentioned by Arun in the post above?

eric.siedentopf
Influencer
Posts: 10
Liked: 1 time
Joined: May 02, 2011 10:40 pm
Full Name: Eric Siedentopf
Contact:

Re: Unable to retrieve next block transmission command

Post by eric.siedentopf » Jun 17, 2013 3:54 pm

I will.

Thank you

yizhar
Expert
Posts: 179
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Unable to retrieve next block transmission command

Post by yizhar » Jun 17, 2013 11:16 pm

Hi.

Just want to add that I'm experiencing same issue randomly.

I have seen it on 2 totally different systems, in Backup and also in Replication jobs.

Details about one instance of the problem:

Error message:
Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0].
After retry it works, but every several days one different VM might have this or similar problem (I don't always have time to investigate further).

VBR server = physical Dell R610 (acting as the proxy and repository).
VBR version = 6.5.0.144
Production storage = HP 3PAR 7200 with 4 FC connections.
Production environment = 3 hosts, VMware esxi 5.1 update 1.
Backup job in SAN mode (using FC card on backup server), using reverse incremental.
Backup Repository = IBM DS3300 iscsi storage, mounted directly by VBR server.

I have also received report from another client of us, he had the same error but with replication job to remote site.
He has an HP P4300G2 (lefthand) ISCSI SAN on production site.
So I assume the problem is related to reading data from source storage.

Maybe the problem is related to SAN mode?

Bye
Yizhar

yizhar
Expert
Posts: 179
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Unable to retrieve next block transmission command

Post by yizhar » Jun 17, 2013 11:23 pm

Hi.

Just to make it clear regarding my previous post:
I am experiencing the problem with no WAN connection involved, for example:
HP 3PAR storage -- FC connection -- FC HBA on physical backup server (proxy and repository) -- NIC -- iscsi switch -- IBM DS3300 NTFS LUN as target.

My client also seen it with replication over WAN on totally different environment.

Yizhar

foggy
Veeam Software
Posts: 16702
Liked: 1343 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Unable to retrieve next block transmission command

Post by foggy » Jun 18, 2013 10:09 am

Yizhar, thank you for the valuable feedback. Have you got a chance to provide this information to our support guys? I assume they are extremely interested in getting data about occurrences of this error in different environments. Would be much appreciated!

Virtualizer
Influencer
Posts: 10
Liked: never
Joined: Jun 13, 2013 6:47 am
Full Name: Gary Golbraich
Contact:

[MERGED] : Last Hard Disk on a VM reads 0.0 KB at 0 KB/s [CB

Post by Virtualizer » Jun 27, 2013 9:45 am

Hi everyone,

I'm having a strange problem with my backup (Using Backup & Replicate 6.5.0.109)
Been looking at similar prompts floating around here but found different situation than mine.

The backup log is:
6/26/2013 9:20:58 PM :: Queued for processing at 6/26/2013 9:20:58 PM
6/26/2013 9:20:58 PM :: Required backup infrastructure resources have been assigned
6/26/2013 9:21:00 PM :: VM processing started at 6/26/2013 9:21:00 PM
6/26/2013 9:21:00 PM :: VM size: 1.8 TB (169.3 GB used)
6/26/2013 9:21:01 PM :: Using source proxy 10.200.120.118 [hotadd]
6/26/2013 9:21:13 PM :: Indexing guest file system
6/26/2013 9:22:56 PM :: Preparing guest for hot backup
6/26/2013 9:23:03 PM :: Creating snapshot
6/26/2013 9:23:11 PM :: Releasing guest
6/26/2013 9:23:28 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.vmx'
6/26/2013 9:23:32 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.vmxf'
6/26/2013 9:23:34 PM :: Saving '[4Snap] mgt-fs-2008/mgt-fs-2008.nvram'
6/26/2013 9:23:37 PM :: Hard Disk 1 (80.0 GB) 1.3 GB read at 24 MB/s [CBT]
6/26/2013 9:24:35 PM :: Hard Disk 2 (800.0 GB) 7.1 GB read at 106 MB/s [CBT]
6/26/2013 9:25:43 PM :: Hard Disk 3 (100.0 GB) 1.0 MB read at 106 KB/s [CBT]
6/26/2013 9:25:53 PM :: Hard Disk 4 (200.0 GB) 960.0 MB read at 68 MB/s [CBT]
6/26/2013 9:26:07 PM :: Hard Disk 5 (600.0 GB) 1.1 GB read at 61 MB/s [CBT]
6/26/2013 9:26:25 PM :: Hard Disk 6 (20.0 GB) 57.0 MB read at 6 MB/s [CBT]
6/26/2013 9:26:35 PM :: Hard Disk 7 (0.0 KB) 0.0 KB read at 0 KB/s [CBT]
6/26/2013 9:26:41 PM :: Removing VM snapshot
6/26/2013 9:33:07 PM :: Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [0].
6/26/2013 9:33:07 PM :: Network traffic verification detected no corrupted blocks
6/26/2013 9:33:07 PM :: Busy: Source 79% > Proxy 28% > Network 10% > Target 6%
6/26/2013 9:33:07 PM :: Primary bottleneck: Source
6/26/2013 9:33:08 PM :: Processing finished with errors at 6/26/2013 9:33:07 PM

What the hell is wrong all of a sudden with Disk 7?

I've had more VMs backed up succesfuly right after this one and prioer to this one at the same night.

Can't find anything wrong in there...

Usualy this backup always ends without any errors.

What can you advice me to check or do ?

Kind reagards,

-Gary

v.Eremin
Veeam Software
Posts: 15077
Liked: 1133 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Unable to retrieve next block transmission command

Post by v.Eremin » Jun 27, 2013 9:50 am

Hi, Gary.

You have been merged to the existing discussion regarding similar issues. Typically you might get the aforesaid error due to unstable network connection. Additional information, as well as, other potential causes can be found above; so, please take a look.

Thanks.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 32 guests