Client Error: Failed to send command

Hyper-V specific discussions

Client Error: Failed to send command

Veeam Logoby vbr4pppsgv » Tue Dec 10, 2013 5:45 pm

Case #: 00485039

3 Hyper-V Hosts getting errors with multiple VMs (Client error: Failed to send command)
Jobs all worked for about a month and then suddenly they started dropping.
Veeam B&R 7.0 Patch 2
Host 1: Hyper-V Server 2012 w/ local storage (2 of 9 VMs fail)
Host 2: Windows Server 2008 R2 in Microsoft Cluster w/ ISCSI SAN CSV's. (7 of 8 VMs fail)
Host 3: Windows Server 2008 R2 w/ local storage (1 of 2 VMs fail) This server had no problems until it rebooted yesterday.
Repository: Windows Server 2008 R2 with 7TB local storage (eSATA RAID 10 enclosure)
VEEAM server: Physical machine, Windows Server 2012

Tried numerous fixes:
- Rebooted all hosts, repository & veeam servers
- Changing from Incremental to Reverse Incremental
- Deleting jobs and recreating
- Splitting entire server jobs into individual VM jobs
- Convert VHDs from dynamic to fixed
- Converting VHDs to VHDXs
- Syncing NUMA nodes on 2012 hosts & disabling old processor migration
- Disabling Parallel Processing
- Exporting VMs and re-importing

The only thing I haven't done is reformat the repository.

Any help would be most appreciated.
Thanks.
-Chris
vbr4pppsgv
Influencer
 
Posts: 10
Liked: 1 time
Joined: Thu Sep 26, 2013 9:00 pm
Full Name: PPPSGV IT Admin

Re: Client Error: Failed to send command

Veeam Logoby foggy » Wed Dec 11, 2013 11:05 am

Chris, can you check whether the VeeamVssSupport service is running on the affected guests? And if yes, try to stop and delete it.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Client Error: Failed to send command

Veeam Logoby vbr4pppsgv » Wed Dec 11, 2013 5:09 pm

Hi Foggy,

It was running on 2 of my VMs. One on Host 1 and One on Host 2.
I stopped and disabled the service. Will try a VeeamZip to test.

Thanks,
Chris
vbr4pppsgv
Influencer
 
Posts: 10
Liked: 1 time
Joined: Thu Sep 26, 2013 9:00 pm
Full Name: PPPSGV IT Admin

Re: Client Error: Failed to send command

Veeam Logoby vbr4pppsgv » Wed Dec 11, 2013 7:48 pm

Nope. Jobs still failing.
I have a second site with a single Hyper-V host (Windows Server 2012) and a similar repository. The Veeam server is actually the repository for this site. This site has no issues at all. I even exported all VMs from an older host (Windows Server 2008 R2) to the new one and didn't upgrade the Hyper-V integration components, and it still completed all jobs successfully.

As another test, I connected a cheapo WD MyCloud single hard drive "NAS" and added it as a CIFS repository. VMs that fail normally, still fail. VMs that succeed, succeed here too.

Does anyone know what the actual error means? Could it be a networking issue, independent of Veeam? If so, why aren't all jobs failing? I can reliably backup 2 VMs with no problems, but all the others are failing.

Frustrating. . .
vbr4pppsgv
Influencer
 
Posts: 10
Liked: 1 time
Joined: Thu Sep 26, 2013 9:00 pm
Full Name: PPPSGV IT Admin

Re: Client Error: Failed to send command

Veeam Logoby foggy » Thu Dec 12, 2013 11:15 am

What causes this error should be identified from the job debug logs, so please continue working with your support engineer.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Client Error: Failed to send command

Veeam Logoby vbr4pppsgv » Sat Dec 14, 2013 1:00 am 1 person likes this post

Just wanted to post a follow-up:

The problem appears to have been related to a pair of SonicWALL WXA WAN Accelerator appliances causing the final copying of job info from the Hyper-V host at Site 1 to the Veeam server at Site 2 to fail. TCP connections were being "accelerated" and causing connection drops. So even after all the data blocks had been successfully transferred, in some cases 1TB+ worth, it would still report as a failure and the restore point wouldn't be available.

Disabling the WXAs has allowed all my jobs to complete successfully.
vbr4pppsgv
Influencer
 
Posts: 10
Liked: 1 time
Joined: Thu Sep 26, 2013 9:00 pm
Full Name: PPPSGV IT Admin

Re: Client Error: Failed to send command

Veeam Logoby v.Eremin » Sat Dec 14, 2013 7:14 am

Thanks for coming back and updating the topic with the found solution; much appreciated.
v.Eremin
Veeam Software
 
Posts: 13255
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 7 guests