Unable to retrieve next block transmission command

[veremin]

However, I'm starting to wonder whether standard backup jobs are more sensitive to connection imperfections (delayed packets etc).

Yep. Backup jobs are less stable against network issues than WAN accelerated backup copy jobs are, as the latter were designed specifically for WAN type of connection that is notoriously known for delayed packets, drops, etc..

Thanks.

[Jamie Pert]

But also could it be said that in some ways that even on a LAN the standard jobs could benefit from WAN accelerators?

[foggy]

WAN acceleration is available for backup copy jobs only, not for ordinary backup jobs.

[leebtish]

Hi all,

I contacted support and asked or the case to be escalated as I wasn't happy with the initial response and I'm still convinced the upgrade has caused the problem. There seem to be a number of posts on here where the problem started after an upgrade. I then had a response from a Veeam engineer who advised that a number of similar cases had been reported and Veeam were investigating. I wouldn't mind betting that there will be a fix for this at some point.

That being said, I decided to try one of the options advised by support:
Select Traffic Throttling from the Veeam Console > Un-tick "Use multiple upload streams per job"

So far, since this change, the issue has not re-occurred. Backups are taking about the same time as well. I'm monitoring this for a few more days before confirming all OK with support. So far so good!

Hopefully this bit of information may help someone else

Cheers

Lee

[foggy]

Lee, thanks for sharing, much appreciated!

[zak2011]

For some strange reason my replication jobs only for Exchange servers are failing with the following error

Error: Client error: An existing connection was forcibly closed by the remote host
Exception from server: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [104684].
Failed to process [srcReplicateVddkDiskContent] command.

I have tried to explicitly select the source and the target proxies.
At the DR site, the proxies are using network mode.
Also tried removing the meta data of the replica at the backup repository.
Also the DNS name of the vcenter has been added to the host file of the target proxies.
Any other possible reasons why replication fails ?

Thanks

[veremin]

Hi, Arun. I’m wondering whether you’re using “Traffic throttling rules” or “Multiple TCP connections”. Thanks.

[llogan6906]

I too have had this issue pop up off and on while using version 6.5 and replicating to a secondary office. It would always retry and 90% of the time would complete on the retry.

Upgraded to version 7 yesterday and all my replication jobs failed with this error and did not complete on retries. I have change the "Select Traffic Throttling from the Veeam Console > Un-tick "Use multiple upload streams per job" setting and am going to see if that fixes the issue tonight.

My thought would be something has definitely changed in the code that is more sensitive to network latency or something. Also, there was a KB1781 about changing the registry to try to fix this issue, but it doesn't say on which server that setting needs to be changed. Is it on the server hosting Veeam, the Veeam Proxy, or the server that is having the error show up during processing?

Anyway. I will try to update tomorrow on the status of the change in upload streams per job.

[veremin]

Also, there was a KB1781 about changing the registry to try to fix this issue, but it doesn't say on which server that setting needs to be changed. Is it on the server hosting Veeam, the Veeam Proxy, or the server that is having the error show up during processing?

As far as I know, it should be implemented on the target side. However, your issue seems to be different from the one discussed there.

Thanks.

[llogan6906]

Update to post from the 21st...

The replication worked with no connection errors. It did, however, take about 8 hours longer than normal. I am chalking that up to the larger window of time that had passed since the last successful replication and therefore more changed blocks to process. I will find out when it runs again tonight. Another odd thing for me was that 98% of the bottleneck was the Target when in past replication jobs it has always been the Network holding things up (around 60%).

I'd still like any input possible on why that setting would need changed after the upgrade to version 7 as that baffles me.

Thanks for the help!

[veremin]

It did, however, take about 8 hours longer than normal. I am chalking that up to the larger window of time that had passed since the last successful replication and therefore more changed blocks to process.

I would also check whether CBT was used. Without CBT VB&R uses its proprietary mechanism, reading the whole VM image in order to find changed blocks. It, in its turn, results in longer than usual replication time.

Thanks.

[Shralok]

We had that on offsite copy jobs lee. Once again opting to use WAN accelerators fixed the issue. Obviously this doesn't massively help you as yours is backing up to a location on the same LAN.

However, I'm starting to wonder whether standard backup jobs are more sensitive to connection imperfections (delayed packets etc). We used to use standard copy jobs to backup to offsite repositories, but since Veeam 7 we have had to opt for backup copy jobs with the use of WAN accelerators - this has ran like a dream.

Jamie, would you mind elaborating on your backup setup?

What type of schedule do you have in terms of a Backup job, and then the Backup Copy job that uses the WAN accelerator?

Do you just do a straight Backup Copy job?

Thank you

[oz42]

Hello,

case no. 00534652

a backup of a single Windows 7 machine fails always:

Code: Select all

Processing 'win7' Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [2468]. Failed to perform data backup. An existing connection was forcibly closed by the remote host 

Any ideas? B&R version is 7.0.0.839, the machine is backed up via virtual Veeam Proxy on ESX A to the virtual backup machine on ESX B.

Strange is: when I remove the Win7 machine from the backup group and try to add it again, the machine ist displayed as running, but it is shut down.

Olaf

[Vitaliy S.]

Hi Olaf,

What backup mode are you using to backup this VM? Do you have only one VM with this behavior?

Thanks!

[oz42]

Hi,

this is one out of two Win7 machines in the same group with that behaviour. I am just checking if it is because the VMware tools version on this machine is newer than the ESXi version (5.5. vs. 5.1).

Olaf

[Vitaliy S.]

I doubt that VMware Tools version could be the issue here. I would suggest trying different backup modes, and if nothing helps, continue working with our technical team.

[JaquesC]

Hi there,

I have only 2 VMS that are doing this at the moment, stating that the connection was forcibly closes, however some data was written:

Is the assumption here that an application or something on that VM closes the Veeam connection when its running? What and where should I start troubleshooting?

question, I am grabbing a snap, so its not the live VM that has any input in terms of the snap and Veeam working together, what exactly is still closing off this job from running?

[Vitaliy S.]

Hi Jaques,

Please review this thread for possible troubleshooting steps. If nothing helps, then please contact our technical team directly, as it is advised when you click New Topic.

Thanks!

[lwarner]

I currently have an open case, 00522320, but wanted some community input. Here's what's happening.

The setup - three sites, single AD, 100Mbps point-point switched ethernet over fiber connecting all sites. Connection is extremely stable, low latency. Veeam B&R v7.0.0.839, vSphere 5.1 at all sites. Site A is our primary data center, Site B is our contingency data center, Site C is our main office. Here are 3 relavent jobs:
1. Replication, Site A > B. Job contains single Win 2008R2 server with 2008R2 SQL. This SQL server contains the vCenter db.
2. Backup, Site C > B. Job contains a few Win7 workstations and a couple Win 2008R2 file servers.
3. Backup, Site C > B. Job contains several Windows app and file servers.

Initially with Veeam v6.5, all these jobs ran fine, no errors at all.

I then upgraded to v7, and job 1 immediately started to fail. I tried for weeks to get it to run, no luck, so I opened a support case.

Then a few weeks later, I had to restart the VMWare host at Site C. No config changes, just a restart (Dell OM plug in stopped responding). Immediately jobs 2 and 3 started to fail.

The symptoms for all of these 'failed' jobs is the same. The job runs for a good percentage, transferring lots of data (many GB), then bombs with "An existing connection was forcibly closed by the remote host..." or the job just stalls and stops sending data, but no error. Now I know what everyone's thinking, it's a network problem. But we monitored the network during the issue, and it showed no problems. Pluse we have other WAN jobs that run fine all day and night, and then this odd result: we can run an initial full replication of job 1 (400GB) no problem, but the incremental will fail, even if we run it right after the full backup.

We've tried network vs hot add, changing proxies, new replication seed, new fresh job, nothing works yet. We're still troubleshooting. It's been a couple months and I need to get this resolved, so any input is appreciated!

[foggy]

Leigh, please review this thread, probably some of the mentioned hints could help in your case. Otherwise, please continue to work with your support engineer.

[oz42]

lwarner, have you tried to disable and re-enable CBT?
http://www.lazywinadmin.com/2013/01/ena ... bt-on.html
At my site I got less errors since then. I assume that it is a CBT issue.

[davidkillingsworth]

I have just upgraded from the latest, patched version 7 to Version 8 Patch 1 and I'm suddenly getting this problem.

The backups and replication jobs all fail. It has been working fine for months.

I do not have anti-virus on the servers, which is the first question support asked me.

I have one physical windows server where Veeam is installed, which is connected to the repository. The repository is connected to a Synology NAS via iSCSI.

I have 2 VMware hosts, production and failover/backup. Each VMware host has a Veeam proxy. The backup jobs are coming from the production VMware host using the veeam proxy on that server going to the repository. Proxy transport was automatic, but I've set it as Virtual Appliance and specified it as the proxy server for the job instead of automatic.

I did the same for the replication job, which is of course going to the backup/failover VMware host via the backup/failover veeam proxy.

I do not have any traffic throttling enabled or multiple TCP connection per job set.

I've tried running a full active backup and it doesn't help.

I've done pings from the veeam server where the respository is to the VMware host and Veeam proxy and there is no packet loss.

I have tried http://www.veeam.com/kb1781 and it does not help.

Here is one of the messages I'm getting:
Error: Serialized size of the SRawKeySetId is unexpected (-1). Expected size: '16'. Unable to retrieve next block transmission command. Number of already processed blocks: [295]. Failed to download disk. An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.



If I change the proxy transport mode to Network, the job seems to run, but I only get 10MB throughput, and at that rate, it will take a couple of days to complete a backup.

[Vitaliy S.]

What's your case ID for this issue?

[davidkillingsworth]

Case# 00731437

Oddly, after I left the replication job overnight, this morning the replication for 3 of the 4 VMs ran successfully, and then on the 2nd retry, the last of the 4 VMs, which also happens to be the largest also ran successfully.

The replication job is set to run at 7AM, 12PM, and 8PM.

The replication job at 12PM ran fine without a hitch. I just tried to do some file level restores (all are linux boxes) to make sure that the 7AM and 12PM replication snapshots were valid and I was able to mount the disk and do a file restore without any problem.

I did have a remote session support with a Veeam engineer and he suggested that I reboot the Veeam backup server connected to the repository and the proxies, which I did. This occured at about about 1:00PM today after the first 2 days replication jobs ran successfully.

I had already tried rebooting them, so I'm not sure that this would be the fix, but at least we know we are on a clean slate.

I'll see how the repliation job at 8PM and the backup job at 10PM go and report back.

Any idea or suggestions on why this was initially broken and then suddenly fixes itself?

[loelly]

I have had similar errors. I have four hosts each with a backup proxies and a Veeam B&R repository server. As I was also getting "Connection reset by peer" and "Connection was forcibly closed by remote host", I disabled the Windows Firewall on all Veeam backup proxies. Some guys point to a SYN_FLOOD detection problem on the proxies...

Support also advised to restart the VMware management daemons on all ESXi hosts. Change has been made last week and so far so good. Maybe give it a try?

[davidkillingsworth]

I have had similar errors. I have four hosts each with a backup proxies and a Veeam B&R repository server. As I was also getting "Connection reset by peer" and "Connection was forcibly closed by remote host", I disabled the Windows Firewall on all Veeam backup proxies. Some guys point to a SYN_FLOOD detection problem on the proxies...

Support also advised to restart the VMware management daemons on all ESXi hosts. Change has been made last week and so far so good. Maybe give it a try?

I just had a long remote support session for the 2nd time in 2 days with a level 1 support engineer and he escalated it to level 2. After a few minutes, I got an email that suggested that I reset the management deamons on the ESXi hosts also, as per this article. kb.vmware.com/kb/1003490.

After restarted management daemons and restarted all Veeam servers and proxies, I restarted the replication job and the first 2 VM guets processed in under 10 minutes, but a 3rd larger VM guests ran for over an hour calculating digest and then immediately failed after processing a few hundred MB.

This is the same error:

Code: Select all

Error: Failed to decompress LZ4 block: Incorrect decompression result or length (result: '-394772', expected length: '524288'). Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

[davidkillingsworth]

My problem is resolved.

It turned out to be this fatal combination:
1) Vmware 5.1 Update 1

2) Windows 2012 - Veeam Proxy Servers

3) VMware VM guest network card type of E1000 for the Veeam Proxy Server

4) Veeam 8

After we shut down the VM guests (Veeam proxy servers), deleted the E1000 network cards, added VMX3 network card, configured the new network cards with the appropirate IP address information, and restarted the replication and backup jobs, everything went back to normal and the jobs started completing.

I hope this helps someone else in a similar situation.

I want to say thanks to the Veeam support staff for working so long on this with me.

[SGalbincea]

I am in a similar boat here. Ticket# 00713193

The setup:

VMware vSphere 5.5U2b Virtual Environment
2x Nimble CS460's via Brocade VDX 6740s
1x Veeam B&R Virtual Server
2x Physical Veeam Proxies, one 2008R2 and one 2012R2 (SAN Direct) each with 2x 10Gb LAN, 2x 10Gb SAN connectivity
1x DataDomain DD2500 w/DDBoost (2 repo's configured with each proxy as a GW for each repo)

I am having issues with larger servers. The backups will run for some time, and then fail with the following message:

"1/25/2015 10:25:28 PM :: Processing AMHOUGEO01 Error: Thread not finished within [1800000] milliseconds.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}."

I can't see any reason at all why the data flows would time out - the environment is very much over-spec'd for this application. I cannot find any clues in the logs anywhere as to what is happening. I have disabled parallel processing with some degree of success, but it is simply not practical to do so with our data amounts (30+TB nightly).

I have tried the suggestions here with regards to disabling IP4 offload and the reg key, so I have tried those and will report back what happens there.

[andnym]

Hi community!

In a job with around 40 vm:s we have suddenly got problems for 6 vm:s running Windows Server 2003. It started last week and I have done som testing myself. All vm:s are about the same in size (60 GB disk). There is no issues seen in vmware with snapshots for these vm:s.

I opened a case (Case #00732340) a week ago but so far they have come up with nothing that solves this problem.
Has anyone else seen this and got a solution for the problem? I'm currently running v7 with latest patch installed. Until I get okey from our managemet to upgrade to v8 I'm stuck with v7.

Here is the errors from the backups. Mostly it happeneds when the progress is around 30% of the vm:s

Code: Select all

2015-01-27 23:04:47 :: Processing 'no-nit-as1521' Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [4947].
Failed to perform data backup.
An existing connection was forcibly closed by the remote host

2015-01-27 23:04:49 :: Processing 'no-nit-as1536' Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [5587].
Failed to perform data backup.
An existing connection was forcibly closed by the remote host

2015-01-27 23:09:21 :: Processing 'no-nit-as1539' Error: Client error: An existing connection was forcibly closed by the remote host
Unable to retrieve next block transmission command. Number of already processed blocks: [14216].
Failed to perform data backup.
An existing connection was forcibly closed by the remote host

All other vm:s in the job works every night.

BR
Anders

[foggy]

Anders, please review the thread above for possible hints and continue investigating with our support, if nothing else helps (most likely case escalation will be required). Thanks.