Unable to retrieve next block transmission command

VMware specific discussions

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Fri Mar 21, 2014 12:51 pm

I doubt that VMware Tools version could be the issue here. I would suggest trying different backup modes, and if nothing helps, continue working with our technical team.
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

[MERGED] VM forcing connection to close, backup failes

Veeam Logoby JaquesC » Sat Apr 05, 2014 11:00 am

Hi there,

I have only 2 VMS that are doing this at the moment, stating that the connection was forcibly closes, however some data was written:

Is the assumption here that an application or something on that VM closes the Veeam connection when its running? What and where should I start troubleshooting?

question, I am grabbing a snap, so its not the live VM that has any input in terms of the snap and Veeam working together, what exactly is still closing off this job from running?
JaquesC
Influencer
 
Posts: 12
Liked: never
Joined: Wed Jan 22, 2014 5:46 pm

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Sat Apr 05, 2014 11:24 am

Hi Jaques,

Please review this thread for possible troubleshooting steps. If nothing helps, then please contact our technical team directly, as it is advised when you click New Topic.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

[MERGED] Some WAN jobs failing

Veeam Logoby lwarner » Mon May 05, 2014 8:20 pm

I currently have an open case, 00522320, but wanted some community input. Here's what's happening.

The setup - three sites, single AD, 100Mbps point-point switched ethernet over fiber connecting all sites. Connection is extremely stable, low latency. Veeam B&R v7.0.0.839, vSphere 5.1 at all sites. Site A is our primary data center, Site B is our contingency data center, Site C is our main office. Here are 3 relavent jobs:
1. Replication, Site A > B. Job contains single Win 2008R2 server with 2008R2 SQL. This SQL server contains the vCenter db.
2. Backup, Site C > B. Job contains a few Win7 workstations and a couple Win 2008R2 file servers.
3. Backup, Site C > B. Job contains several Windows app and file servers.

Initially with Veeam v6.5, all these jobs ran fine, no errors at all.

I then upgraded to v7, and job 1 immediately started to fail. I tried for weeks to get it to run, no luck, so I opened a support case.

Then a few weeks later, I had to restart the VMWare host at Site C. No config changes, just a restart (Dell OM plug in stopped responding). Immediately jobs 2 and 3 started to fail.

The symptoms for all of these 'failed' jobs is the same. The job runs for a good percentage, transferring lots of data (many GB), then bombs with "An existing connection was forcibly closed by the remote host..." or the job just stalls and stops sending data, but no error. Now I know what everyone's thinking, it's a network problem. But we monitored the network during the issue, and it showed no problems. Pluse we have other WAN jobs that run fine all day and night, and then this odd result: we can run an initial full replication of job 1 (400GB) no problem, but the incremental will fail, even if we run it right after the full backup.

We've tried network vs hot add, changing proxies, new replication seed, new fresh job, nothing works yet. We're still troubleshooting. It's been a couple months and I need to get this resolved, so any input is appreciated!
lwarner
Influencer
 
Posts: 22
Liked: 2 times
Joined: Sun May 29, 2011 6:05 pm
Full Name: Leigh Warner

Re: Unable to retrieve next block transmission command

Veeam Logoby foggy » Tue May 06, 2014 10:25 am

Leigh, please review this thread, probably some of the mentioned hints could help in your case. Otherwise, please continue to work with your support engineer.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Unable to retrieve next block transmission command

Veeam Logoby oz42 » Wed May 21, 2014 1:38 pm

lwarner, have you tried to disable and re-enable CBT?
http://www.lazywinadmin.com/2013/01/ena ... bt-on.html
At my site I got less errors since then. I assume that it is a CBT issue.
oz42
Influencer
 
Posts: 13
Liked: 1 time
Joined: Thu Jan 16, 2014 12:40 pm
Full Name: Olaf Zaplinski

Re: Unable to retrieve next block transmission command

Veeam Logoby davidkillingsworth » Tue Jan 20, 2015 7:27 pm

I have just upgraded from the latest, patched version 7 to Version 8 Patch 1 and I'm suddenly getting this problem.

The backups and replication jobs all fail. It has been working fine for months.

I do not have anti-virus on the servers, which is the first question support asked me.

I have one physical windows server where Veeam is installed, which is connected to the repository. The repository is connected to a Synology NAS via iSCSI.

I have 2 VMware hosts, production and failover/backup. Each VMware host has a Veeam proxy. The backup jobs are coming from the production VMware host using the veeam proxy on that server going to the repository. Proxy transport was automatic, but I've set it as Virtual Appliance and specified it as the proxy server for the job instead of automatic.

I did the same for the replication job, which is of course going to the backup/failover VMware host via the backup/failover veeam proxy.

I do not have any traffic throttling enabled or multiple TCP connection per job set.

I've tried running a full active backup and it doesn't help.

I've done pings from the veeam server where the respository is to the VMware host and Veeam proxy and there is no packet loss.

I have tried http://www.veeam.com/kb1781 and it does not help.

Here is one of the messages I'm getting:
Error: Serialized size of the SRawKeySetId is unexpected (-1). Expected size: '16'. Unable to retrieve next block transmission command. Number of already processed blocks: [295]. Failed to download disk. An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.



If I change the proxy transport mode to Network, the job seems to run, but I only get 10MB throughput, and at that rate, it will take a couple of days to complete a backup.
davidkillingsworth
Enthusiast
 
Posts: 29
Liked: 4 times
Joined: Mon Nov 21, 2011 5:41 am
Location: Hong Kong

Re: Unable to retrieve next block transmission command

Veeam Logoby Vitaliy S. » Tue Jan 20, 2015 8:32 pm

What's your case ID for this issue?
Vitaliy S.
Veeam Software
 
Posts: 19773
Liked: 1120 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Unable to retrieve next block transmission command

Veeam Logoby davidkillingsworth » Wed Jan 21, 2015 7:05 am

Case# 00731437

Oddly, after I left the replication job overnight, this morning the replication for 3 of the 4 VMs ran successfully, and then on the 2nd retry, the last of the 4 VMs, which also happens to be the largest also ran successfully.

The replication job is set to run at 7AM, 12PM, and 8PM.

The replication job at 12PM ran fine without a hitch. I just tried to do some file level restores (all are linux boxes) to make sure that the 7AM and 12PM replication snapshots were valid and I was able to mount the disk and do a file restore without any problem.

I did have a remote session support with a Veeam engineer and he suggested that I reboot the Veeam backup server connected to the repository and the proxies, which I did. This occured at about about 1:00PM today after the first 2 days replication jobs ran successfully.

I had already tried rebooting them, so I'm not sure that this would be the fix, but at least we know we are on a clean slate.

I'll see how the repliation job at 8PM and the backup job at 10PM go and report back.

Any idea or suggestions on why this was initially broken and then suddenly fixes itself?
davidkillingsworth
Enthusiast
 
Posts: 29
Liked: 4 times
Joined: Mon Nov 21, 2011 5:41 am
Location: Hong Kong

Re: Unable to retrieve next block transmission command

Veeam Logoby loelly » Wed Jan 21, 2015 10:23 am

I have had similar errors. I have four hosts each with a backup proxies and a Veeam B&R repository server. As I was also getting "Connection reset by peer" and "Connection was forcibly closed by remote host", I disabled the Windows Firewall on all Veeam backup proxies. Some guys point to a SYN_FLOOD detection problem on the proxies...

Support also advised to restart the VMware management daemons on all ESXi hosts. Change has been made last week and so far so good. Maybe give it a try?
loelly
Enthusiast
 
Posts: 49
Liked: 10 times
Joined: Thu Apr 17, 2014 8:25 am
Full Name: Jens Siegmann

Re: Unable to retrieve next block transmission command

Veeam Logoby davidkillingsworth » Thu Jan 22, 2015 9:13 am

loelly wrote:I have had similar errors. I have four hosts each with a backup proxies and a Veeam B&R repository server. As I was also getting "Connection reset by peer" and "Connection was forcibly closed by remote host", I disabled the Windows Firewall on all Veeam backup proxies. Some guys point to a SYN_FLOOD detection problem on the proxies...

Support also advised to restart the VMware management daemons on all ESXi hosts. Change has been made last week and so far so good. Maybe give it a try?


I just had a long remote support session for the 2nd time in 2 days with a level 1 support engineer and he escalated it to level 2. After a few minutes, I got an email that suggested that I reset the management deamons on the ESXi hosts also, as per this article. kb.vmware.com/kb/1003490.

After restarted management daemons and restarted all Veeam servers and proxies, I restarted the replication job and the first 2 VM guets processed in under 10 minutes, but a 3rd larger VM guests ran for over an hour calculating digest and then immediately failed after processing a few hundred MB.

This is the same error:
Code: Select all
Error: Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-394772', expected length: '524288'). Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
davidkillingsworth
Enthusiast
 
Posts: 29
Liked: 4 times
Joined: Mon Nov 21, 2011 5:41 am
Location: Hong Kong

Re: Unable to retrieve next block transmission command

Veeam Logoby davidkillingsworth » Fri Jan 23, 2015 9:09 am 3 people like this post

My problem is resolved.

It turned out to be this fatal combination:
1) Vmware 5.1 Update 1

2) Windows 2012 - Veeam Proxy Servers

3) VMware VM guest network card type of E1000 for the Veeam Proxy Server

4) Veeam 8

After we shut down the VM guests (Veeam proxy servers), deleted the E1000 network cards, added VMX3 network card, configured the new network cards with the appropirate IP address information, and restarted the replication and backup jobs, everything went back to normal and the jobs started completing.

I hope this helps someone else in a similar situation.

I want to say thanks to the Veeam support staff for working so long on this with me.
davidkillingsworth
Enthusiast
 
Posts: 29
Liked: 4 times
Joined: Mon Nov 21, 2011 5:41 am
Location: Hong Kong

Re: Unable to retrieve next block transmission command

Veeam Logoby SGalbincea » Mon Jan 26, 2015 4:27 pm

I am in a similar boat here. Ticket# 00713193

The setup:

VMware vSphere 5.5U2b Virtual Environment
2x Nimble CS460's via Brocade VDX 6740s
1x Veeam B&R Virtual Server
2x Physical Veeam Proxies, one 2008R2 and one 2012R2 (SAN Direct) each with 2x 10Gb LAN, 2x 10Gb SAN connectivity
1x DataDomain DD2500 w/DDBoost (2 repo's configured with each proxy as a GW for each repo)

I am having issues with larger servers. The backups will run for some time, and then fail with the following message:

"1/25/2015 10:25:28 PM :: Processing AMHOUGEO01 Error: Thread not finished within [1800000] milliseconds.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}."


I can't see any reason at all why the data flows would time out - the environment is very much over-spec'd for this application. I cannot find any clues in the logs anywhere as to what is happening. I have disabled parallel processing with some degree of success, but it is simply not practical to do so with our data amounts (30+TB nightly).

I have tried the suggestions here with regards to disabling IP4 offload and the reg key, so I have tried those and will report back what happens there.
Senior Network Engineer, VCP5-DCV, VMSP, VMTSP, NIOP
Senior Network Engineer
BEMA Information Technologies
Houston, TX
SGalbincea
Enthusiast
 
Posts: 55
Liked: 6 times
Joined: Fri May 25, 2012 2:09 pm
Location: Houston, TX
Full Name: Steve Galbincea

[MERGED] Unable to retrieve next block transmission command.

Veeam Logoby andnym » Wed Jan 28, 2015 9:37 am

Hi community!

In a job with around 40 vm:s we have suddenly got problems for 6 vm:s running Windows Server 2003. It started last week and I have done som testing myself. All vm:s are about the same in size (60 GB disk). There is no issues seen in vmware with snapshots for these vm:s.

I opened a case (Case #00732340) a week ago but so far they have come up with nothing that solves this problem.
Has anyone else seen this and got a solution for the problem? I'm currently running v7 with latest patch installed. Until I get okey from our managemet to upgrade to v8 I'm stuck with v7.

Here is the errors from the backups. Mostly it happeneds when the progress is around 30% of the vm:s
Code: Select all
2015-01-27 23:04:47 :: Processing 'no-nit-as1521' Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [4947].
Failed to perform data backup.
An existing connection was forcibly closed by the remote host

2015-01-27 23:04:49 :: Processing 'no-nit-as1536' Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [5587].
Failed to perform data backup.
An existing connection was forcibly closed by the remote host

2015-01-27 23:09:21 :: Processing 'no-nit-as1539' Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [14216].
Failed to perform data backup.
An existing connection was forcibly closed by the remote host

All other vm:s in the job works every night.

BR
Anders
andnym
Influencer
 
Posts: 10
Liked: 2 times
Joined: Wed Jul 13, 2011 8:04 am
Full Name: Anders Nyman

Re: Unable to retrieve next block transmission command

Veeam Logoby foggy » Wed Jan 28, 2015 8:47 pm

Anders, please review the thread above for possible hints and continue investigating with our support, if nothing else helps (most likely case escalation will be required). Thanks.
foggy
Veeam Software
 
Posts: 15087
Liked: 1110 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 25 guests