Unable to retrieve next block transmission command

VMware specific discussions

[MERGED] Error: Client error: An existing connection

Veeam Logoby wagnerj » Mon Jul 08, 2013 1:08 pm

Hi,
I have searched the forums and have not found an answer. I also called support, and what they have asked me to do has not worked.

Here is the situation, I have two Veeam environments. I am backing up a clustered Hyper-V environment with 3 nodes. The nodes are connected to an IBM Storwize v7000 SAN. One Veeam server is the test environment which I still get perfect backups on every night (Old Gen 3 HP DL360). Keep in mind these are backing up to 4 external 1TB drives like a champ! The other environment is a high powered IBM server I just purchased just for Veeam with 20TB of Internal local storage on a RAID 5. I am able to take a full backup of my 4TB exchange environment on the new server, but reverse incremental fail with the following error:

Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [27987]

It fails each time at 25% on the second disk. Keep in mind it works perfectly on the old system. The backup is being performed over a LAN, on the same VLAN, connected to the same Cisco 3750 X. The port configs are the same as the test server. I did what Veeam support asked and throttle the connection by 60% and that did not work either. I also tried to re-seed the intital backup, I get the same results.
Can Anyone Help?
Jim
wagnerj
Novice
 
Posts: 3
Liked: never
Joined: Mon Jul 08, 2013 12:45 pm
Full Name: JW

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Mon Jul 08, 2013 1:46 pm

Hi Jim,
wagnerj wrote:I also tried to re-seed the intital backup, I get the same results.

Does it also fail on the same 25%? Is this the only VM affected by this behavior?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19966
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Unable to retrieve next block transmission command

Veeam Logoby wagnerj » Mon Jul 08, 2013 2:14 pm

It's 23% now, to reiderate the old Veeam server just completed an incremental with no issue. Its a different change block reported each time. No all VM incrementals are affected.
wagnerj
Novice
 
Posts: 3
Liked: never
Joined: Mon Jul 08, 2013 12:45 pm
Full Name: JW

Re: Unable to retrieve next block transmission command

Veeam Logoby wagnerj » Mon Jul 08, 2013 2:16 pm

I am running a ping to the VM being backed up, I would like to see if any network latency or timeouts occur. I am running and monitoring the job now.

Update:
I just re-ran the job that was failing, I had a ping running to the VM being backed up, the Veeam server and the hyper-v node. All pings stayed at <1ms the entire time.
wagnerj
Novice
 
Posts: 3
Liked: never
Joined: Mon Jul 08, 2013 12:45 pm
Full Name: JW

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Tue Jul 09, 2013 11:18 am

Hmm...I believe deeper logs investigation is required, as there might be something wrong with a Windows host where new Veeam B&R server is installed.
Vitaliy S.
Veeam Software
 
Posts: 19966
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Unable to retrieve next block transmission command

Veeam Logoby DaFresh » Thu Jul 11, 2013 12:56 pm

FYI, removing traffic throttling and disable 'multiple TCP connection per job' (which I don't know if it is used when traffic throttling is disabled) seams to solve my problem.
Hope it helps.
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Unable to retrieve next block transmission command

Veeam Logoby v.Eremin » Thu Jul 11, 2013 1:06 pm

Thanks, Cedric, for sharing the information with the community; much appreciated. And yes, “use multiple TCP/IP connection” can be enabled/disabled regardless of throttling rules. Thanks.
v.Eremin
Veeam Software
 
Posts: 13728
Liked: 1027 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Unable to retrieve next block transmission command

Veeam Logoby DaFresh » Thu Jul 11, 2013 6:56 pm

As a complement, this error was encountered on a replication job, not a backup, I would think that traffic throttling and multiple TCP/IP connection are only used by replication jobs, which means most of the time remote site/different ip range, except maybe when proxy are used for backup ?
DaFresh
Enthusiast
 
Posts: 64
Liked: 1 time
Joined: Tue Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Thu Jul 11, 2013 10:07 pm

TCP/IP connection configuration applies to both backup and replication jobs. Looks like there were issues between your proxy servers used on the target and source sites.
Vitaliy S.
Veeam Software
 
Posts: 19966
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Unable to retrieve next block transmission command

Veeam Logoby v.Eremin » Fri Jul 12, 2013 8:59 am

I would think that traffic throttling and multiple TCP/IP connection are only used by replication jobs

The situation is slightly different .Veeam B&R is using two-agent based architecture. And all of features like throttling rules, compression, etc, are implemented between them. In case of remote replication, these features would be implemented between two proxy servers; in case of backup, between a backup proxy and a repository; in case of backup to an SMB share between a backup proxy and a proxying Windows server.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13728
Liked: 1027 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Unable to retrieve next block transmission command

Veeam Logoby bnssite » Thu Aug 01, 2013 12:51 pm

I kept getting "Unable to retrieve next block transmission command. Number of already processed blocks: [0]." I then built a FreeNAS box so I could have a different repository to send my Veeam 6.5 data backups to, and it worked.
bnssite
Influencer
 
Posts: 10
Liked: never
Joined: Wed May 16, 2012 10:11 pm
Full Name: James Monfregola

Re: Unable to retrieve next block transmission command

Veeam Logoby Jamie Pert » Thu Oct 10, 2013 2:26 pm 1 person likes this post

If you get the following error:

Error: Client error: An existing connection was forcibly closed by the remote host
Exception from server: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
Unable to retrieve next block transmission command. Number of already processed blocks: [0].
Failed to perform data backup.


Chances are you are using the "direct" option on a Veeam copy job to an offsite repository. Immediately switching to WAN accelerators worked fine. It's because the direct option is less resilient to factors often encountered with offsite storage aka delays / slow speeds.

What's weird is this only affected a few VMs in our infrastructure. On the same source storage to the same destination storage roughly 80% of our copy jobs were fine, but certain ones just failed, often very quickly. We tried different times of day, new jobs and much more and it just wouldn't work.
@jam1epert on Twitter
Jamie Pert
Enthusiast
 
Posts: 68
Liked: 2 times
Joined: Thu Jun 14, 2012 10:56 am
Location: twitter.com/jam1epert
Full Name: Jamie Pert

[MERGED] Veeam 7 - An existing connection was forcibly close

Veeam Logoby leebtish » Mon Oct 14, 2013 5:55 am

Hi all,

Since upgrading to version 7 (patch 1) we have received the following error for many VM's: "Error: Client error: An existing connection was forcibly closed by the remote host Exception from server: An existing connection was forcibly closed by the remote host Unable to retrieve next block transmission command. Number of already processed blocks: [359]. Failed to perform data backup." Note - the processed blocks number is different for each VM.

The backups are successful on subsequent retries but we never received this error on version 6.5. We always had successful backups. We're seeing the errors on a daily basis now. I've raised a case with support and their initial response was to throttle the bandwidth as they state this error can occur in unstable networks. Thing is, this is on a gigabit LAN and we never had any problems before the upgrade so I find it hard to accept that it's a network issue.

Just wondering if anyone else come across this problem since upgrading to version 7 ? At the moment I'm considering rolling back to 6.5 if possible.

Thanks in advance :)

Lee
leebtish
Influencer
 
Posts: 18
Liked: 3 times
Joined: Mon Nov 12, 2012 9:07 am
Full Name: Lee Russell

Re: Unable to retrieve next block transmission command

Veeam Logoby foggy » Mon Oct 14, 2013 8:50 am

Lee, looks like you're the first one to report this error for v7 backup jobs here on forums. This error indeed typically relates to network connection issues, you may want to review this topic for other occurrences of the same error and probably find some hints that helped other users in similar situations. There is also a dedicated KB article regarding this kind of errors, worth reviewing: http://www.veeam.com/kb1781.
foggy
Veeam Software
 
Posts: 15295
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to retrieve next block transmission command

Veeam Logoby Jamie Pert » Tue Oct 15, 2013 10:05 am

We had that on offsite copy jobs lee. Once again opting to use WAN accelerators fixed the issue. Obviously this doesn't massively help you as yours is backing up to a location on the same LAN.

However, I'm starting to wonder whether standard backup jobs are more sensitive to connection imperfections (delayed packets etc). We used to use standard copy jobs to backup to offsite repositories, but since Veeam 7 we have had to opt for backup copy jobs with the use of WAN accelerators - this has ran like a dream.
@jam1epert on Twitter
Jamie Pert
Enthusiast
 
Posts: 68
Liked: 2 times
Joined: Thu Jun 14, 2012 10:56 am
Location: twitter.com/jam1epert
Full Name: Jamie Pert

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 1 guest