Client Error: Failed to send command

[vbr4pppsgv]

Case #: 00485039

3 Hyper-V Hosts getting errors with multiple VMs (Client error: Failed to send command)
Jobs all worked for about a month and then suddenly they started dropping.
Veeam B&R 7.0 Patch 2
Host 1: Hyper-V Server 2012 w/ local storage (2 of 9 VMs fail)
Host 2: Windows Server 2008 R2 in Microsoft Cluster w/ ISCSI SAN CSV's. (7 of 8 VMs fail)
Host 3: Windows Server 2008 R2 w/ local storage (1 of 2 VMs fail) This server had no problems until it rebooted yesterday.
Repository: Windows Server 2008 R2 with 7TB local storage (eSATA RAID 10 enclosure)
VEEAM server: Physical machine, Windows Server 2012

Tried numerous fixes:
- Rebooted all hosts, repository & veeam servers
- Changing from Incremental to Reverse Incremental
- Deleting jobs and recreating
- Splitting entire server jobs into individual VM jobs
- Convert VHDs from dynamic to fixed
- Converting VHDs to VHDXs
- Syncing NUMA nodes on 2012 hosts & disabling old processor migration
- Disabling Parallel Processing
- Exporting VMs and re-importing

The only thing I haven't done is reformat the repository.

Any help would be most appreciated.
Thanks.
-Chris

[foggy]

Chris, can you check whether the VeeamVssSupport service is running on the affected guests? And if yes, try to stop and delete it.

[vbr4pppsgv]

Hi Foggy,

It was running on 2 of my VMs. One on Host 1 and One on Host 2.
I stopped and disabled the service. Will try a VeeamZip to test.

Thanks,
Chris

[vbr4pppsgv]

Nope. Jobs still failing.
I have a second site with a single Hyper-V host (Windows Server 2012) and a similar repository. The Veeam server is actually the repository for this site. This site has no issues at all. I even exported all VMs from an older host (Windows Server 2008 R2) to the new one and didn't upgrade the Hyper-V integration components, and it still completed all jobs successfully.

As another test, I connected a cheapo WD MyCloud single hard drive "NAS" and added it as a CIFS repository. VMs that fail normally, still fail. VMs that succeed, succeed here too.

Does anyone know what the actual error means? Could it be a networking issue, independent of Veeam? If so, why aren't all jobs failing? I can reliably backup 2 VMs with no problems, but all the others are failing.

Frustrating. . .

[foggy]

What causes this error should be identified from the job debug logs, so please continue working with your support engineer.

[vbr4pppsgv]

Just wanted to post a follow-up:

The problem appears to have been related to a pair of SonicWALL WXA WAN Accelerator appliances causing the final copying of job info from the Hyper-V host at Site 1 to the Veeam server at Site 2 to fail. TCP connections were being "accelerated" and causing connection drops. So even after all the data blocks had been successfully transferred, in some cases 1TB+ worth, it would still report as a failure and the restore point wouldn't be available.

Disabling the WXAs has allowed all my jobs to complete successfully.

[veremin]

Thanks for coming back and updating the topic with the found solution; much appreciated.