Unable to retrieve next block transmission command

[SGalbincea]

Disabling IPv4 Offload on all of our 10Gb NICs in both physical proxy servers (SAN and LAN sides) seems to have resolved this issue for us - 30TB server backup complete!

[andnym]

I have managed to solve the issue for my problem. After requesting a FW opening in order to have the possibility to use other Veeam proxies on a different network for troubleshooting I found out that IPS in our firewall suddenly started to reject the traffic. Checkpoint IPS watches the entire communication, not just the start of the transmission. Then probably somewhere into the job the IPS found a code that was suspect to one if the IPS detections and blocked the traffic.

All my servers affected was identical (cloned from one) and that is why only those was rejected.

In my case it happened around the same time for each vm, around 28-30% into the disk.

BR
Anders

[VenoX]

Hello,

I have a production critical Problem - Case 00812606 - Backup of one of my Servers in the job is not working.

Since customer support seems to be overstrained, i ask the community instead:

I tried ctb reset, readd the server to the job, but nothing helps.
Error Message:
24.02.2015 20:45:11 :: Processing FILESERVER Error: Thread not finished within [1800000] milliseconds.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Thread not finished within [1800000] milliseconds.
Unable to retrieve next block transmission command. Number of already processed blocks: [76327].
Failed to download disk.


Any ideas?

BR
VenoX

[veremin]

Hi,

Your post has been merged into existing discussion. While waiting for an answer from support team, it might be worth checking posts provided above for possible solutions.

Thanks.

[Andreas Neufert]

Hi,

I had random job failures with the message

Code: Select all

"Error: Application is shutting down. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Application is shutting down. Unable to retrieve next block transmission command. Number of already processed blocks: [34664]. Failed to download disk."

Thanks to support 00985273 it was fixed by updating the correct time at ESXi host.

In this case the ESXi host had a windows NTP server configured but ESXi expect NTP v4 while windows work on NTP v3.
After updating /etc/ntp.conf on the ESXi hosts
server <ip>
to
server <ip> version 3
the ESXi worked with correct time and Job run now without random failures.

Thank you Veeam Support.

[loelly]

Andreas,

that's new to me. My ESXi (5.5) hosts are all configured to poll NTP at Windows domain controllers. Time is OK, no issues.

Is this something specific to ESXi 6?

Thanks!

Maik.

[Andreas Neufert]

I don´t know if this is "new". I used another NTP Service before. I use a lot of nested ESXi servers and suspend them from time to time. Without NTP the ESXi host will run out of sync and backup is not possible.

In this case it was a 5.5 with latest patch level. All ESXi time diagnostic command showed a successful test with NTP. But the time never updated. After I changed it to "version 3" in the ntp.conf it worked immediatelly flawless since a month.

You can try to change the time manually (change date) (no VMs running on the ESXi host) and test if after a reboot the time will update automatically.

[Andreas Neufert]

At least this NTP setting fixed my random problem with "Unable to retrieve next block transmission command"

[manasar]

I have the save problem and follow to edit ntp.conf. (Before edit ntp.conf that my ntp service on esxi is ok)

It'work.Error has gone.

Thank you very much

[haslund]

Andreas,

that's new to me. My ESXi (5.5) hosts are all configured to poll NTP at Windows domain controllers. Time is OK, no issues.

Is this something specific to ESXi 6?

Thanks!

Maik.

Nothing new, please see VMware KB 1035833

[schmad27]

I have the same error and it only happens on the same server out of 10 servers. It's over a vpn WAN connection and the connection is stable, so I don't know why Veeam is reporting that it is not. We have users at that site who connect to our terminal servers and they never complain about network performance.

[Andreas Neufert]

Please check if B&R, Proxy and Repositiry Server have same time and timezone setting, just for a test.

[schmad27]

Yes, they both have the same time zone settings. Still the same 1 server out of 10 servers is failing. Here is the full error:

Processing MyServerName Error: Unstable connection: unable to transmit data. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: read: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [11377]. Failed to download disk.

[foggy]

David, please contact technical support for logs investigation.

[kingsize]

Hi Team,

A little bit information about my environment:
*There are 2 servers, both running vmware esxi 6.0 licensed fully patched update 2 HPE
*Rebooted the host
*Fully patched firmware on the HPE host
*This is not a domain controller.
*I do have VEEAM installed on this VM.

I am getting the below error on 1 VM.

VDDK error: 13 (You do not have access rights to this file). Value: 0x000000000000000d
Failed to read from a virtual disk
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 13 (You do not have access rights to this file). Value: 0x000000000000000d
Failed to read from a virtual disk
Unable to retrieve next block transmission command. Number of already processed blocks: [3053].
Failed to download disk

ticket id: #02328800


Support is telling me to update to the latest VEEAM version, I am 100% sure that this is not the issue. Anyone here can maybe help to fix this issue?
I searched the net and the forum section, tried every possible solution. I am stuck!

Thanks!

[foggy]

If nothing from this thread helps, I'd follow the advice of you engineer. What Veeam B&R version are you running, btw?

[Andreas Neufert]

Hi, during my 6+ years with Veeam, this error popped up once in a while at my customers and was in most of the cases fixed by updating ESXi and VDDK Kit to newest update/version.
VDDK newest version => Install latest Version and Update of Veeam Backup & Replication.

A typical indicator that this can be fixed with this approach is, that the Veeam retry was successful always and produced valid backups.

If you can not backup at all with this error, there are several other things to check. See above:
Check if DNS is working with FQDN names for ESXi hosts correctly
and check the following Veeam KB entry that list all kind of things that could lead into the error: https://www.veeam.com/kb2008

[kingsize]

If nothing from this thread helps, I'd follow the advice of you engineer. What Veeam B&R version are you running, btw?

I am using veeam 9.5, I am sure when I am going to update to 9.5 Update 2, it will not fix this issue.

[kingsize]

Hi, during my 6+ years with Veeam, this error popped up once in a while at my customers and was in most of the cases fixed by updating ESXi and VDDK Kit to newest update/version.
VDDK newest version => Install latest Version and Update of Veeam Backup & Replication.

ESXI is patched to the latest 6.0 HPE version.
DNS is working correctly
I already tried all of them written here: https://www.veeam.com/kb2008

It seems that the disk is corrupt. Engineer is telling me to check the vmdk file with vmkfstools.
http://www.vi-toolkit.com/wiki/index.ph ... air_a_disk

Alternativelly, I would suggest deploying vCenter and try to clone the VM. It's the bulletproof test which shoud show us what is wrong with the disk.

The problem is that I don't have storage left. Any recommendations? I need 300GB storage for the clone.

[Andreas Neufert]

Maybe. vddk error 13 is a infrstructure error of a code we get from vmware (vddk kit).

If the software update of ESXi host or version/update of veeam that include new vddk kit do not fix the issue you need to work on your infrastructure (maybe with vmware). You can check the above mentioned KB entry for tips and starting points.


Our Support can help here (this is not a support forum).

[Andreas Neufert]

thx for the feedback. we posted at same time.

[kingsize]

Andreas, I have to check the disk. I have never done that before. Is there any manual on how to do it? Does the VM needs to be offline? Only thing I know is that I have to connect trough Putty with my esxi hosts and thats it...

[Andreas Neufert]

Here is an example I found at google http://blog.asiantuntijakaveri.fi/2012/ ... k.html?m=1

And here an exaple for console access by ssh
https://pubs.vmware.com/vsphere-51/inde ... t.3.6.html

[Andreas Neufert]

And if you have trouble with backing up the vm at all, you can download our Agent for Linux/Win with 30 days trial to create an image level backup on Operating System level.

[kingsize]

Ok so I will boot up with the recovery veeam agent iso on the effected VM. I will make a backup to the NAS.
After this has finished, I make a new vm with disk on the NAS. I boot up from the recovery veeam agent and restore from backup.

It should work?

Or do I have to install the veeam agent on the server and make a backup?

[kingsize]

Got if fixed by myself. Anyone that is facing this issue, let me know. I can help you out.

[davidkillingsworth]

For anybody that has this issue again, you should check what virtual NIC card your Veeam backup server and your proxy's are using.

I just had this issue bite us again. We had this about 3 or 4 years ago and I even found my old comment about how I fixed it before in this or another thread by changing the VM guest virtual nic card to VMX3.

We bought a new server and setup a new VMware esxi 6.7 host. We tried replicating all of our VM guests onto it from two other older hosts (vmware esxi 5.1), one of which is going to be retired because the server hardware has run out of warranty.

From one VM host, all of the VM guests replicated fine. However, on another VM host, we had one very large VM guest that would fail every time we tried the replication.

Error was:

Error: Invalid packet size. Payload size: [1167126733]. Allocation size: [1048832] Unable to retrieve next block transmission command. Number of already processed blocks: [29574]. Failed to download disk. write: An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

I searched through this thread and decided to shut down both my existing Veeam server and the one proxy that I had setup on the new VM host. I changed the virtual NIC cards from E1000 or E1000E to VMX3. After re-configuring the IP address inside the Vm guest and rebooting, I was able to successfully run the replication job without any problem.

E1000 strikes again!!!!

[sambrown]

I just had this happen after I changed the VMXNET3 receive buffers from 2048 to 128 to allow receive offload to be re-enabled:

before: (this problem randomly occurring)
2018-11-12T13:24:00.990Z cpu4:35857)Vmxnet3: 15680: Using default queue delivery for vmxnet3 for port 0x2000005
After: (no problems)
2018-11-12T13:24:01.429Z cpu4:35860)Vmxnet3: 15430: Disable Rx queuing; queue size 2048 is larger than Vmxnet3RxQueueLimit limit of 128.

not sure how to change Vmxnet3RXqueuelimit higher (to 2048)..

The backup destination in question is a virtual windows 2016 server!

Error are:

Code: Select all

Processing ax Error: Unable to resend traffic block '1546' because traffic history doesn't contain this block. History low block id '1853'. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
Processing ax Error: Unexpected packet type: [%1]. Unable to retrieve next block transmission command. Number of already processed blocks: [3620]. Failed to download disk. read: An existing connection was forcibly closed by the remote host Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. 
Processing ax Error: Invalid packet size. Payload size: [1845549230]. Allocation size: [1048832] Unable to retrieve next block transmission command. Number of already processed blocks: [59997]. Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

just FYI! not sure if it is vmxnet3 "default queue delivery" with 128 receive buffers in windows 2016 network configuration or if the Disable RX queueing (2048 size buffers - all receive buffers) caused this!

[Yippeekayaye]

Can you share how you fixed this or contact me. I've been wrestling with this for over a month. It happens about once a year and Veeam support can't seem to provide a fix.