-
- Influencer
- Posts: 10
- Liked: 1 time
- Joined: Sep 26, 2013 9:00 pm
- Full Name: PPPSGV IT Admin
- Contact:
Client Error: Failed to send command
Case #: 00485039
3 Hyper-V Hosts getting errors with multiple VMs (Client error: Failed to send command)
Jobs all worked for about a month and then suddenly they started dropping.
Veeam B&R 7.0 Patch 2
Host 1: Hyper-V Server 2012 w/ local storage (2 of 9 VMs fail)
Host 2: Windows Server 2008 R2 in Microsoft Cluster w/ ISCSI SAN CSV's. (7 of 8 VMs fail)
Host 3: Windows Server 2008 R2 w/ local storage (1 of 2 VMs fail) This server had no problems until it rebooted yesterday.
Repository: Windows Server 2008 R2 with 7TB local storage (eSATA RAID 10 enclosure)
VEEAM server: Physical machine, Windows Server 2012
Tried numerous fixes:
- Rebooted all hosts, repository & veeam servers
- Changing from Incremental to Reverse Incremental
- Deleting jobs and recreating
- Splitting entire server jobs into individual VM jobs
- Convert VHDs from dynamic to fixed
- Converting VHDs to VHDXs
- Syncing NUMA nodes on 2012 hosts & disabling old processor migration
- Disabling Parallel Processing
- Exporting VMs and re-importing
The only thing I haven't done is reformat the repository.
Any help would be most appreciated.
Thanks.
-Chris
3 Hyper-V Hosts getting errors with multiple VMs (Client error: Failed to send command)
Jobs all worked for about a month and then suddenly they started dropping.
Veeam B&R 7.0 Patch 2
Host 1: Hyper-V Server 2012 w/ local storage (2 of 9 VMs fail)
Host 2: Windows Server 2008 R2 in Microsoft Cluster w/ ISCSI SAN CSV's. (7 of 8 VMs fail)
Host 3: Windows Server 2008 R2 w/ local storage (1 of 2 VMs fail) This server had no problems until it rebooted yesterday.
Repository: Windows Server 2008 R2 with 7TB local storage (eSATA RAID 10 enclosure)
VEEAM server: Physical machine, Windows Server 2012
Tried numerous fixes:
- Rebooted all hosts, repository & veeam servers
- Changing from Incremental to Reverse Incremental
- Deleting jobs and recreating
- Splitting entire server jobs into individual VM jobs
- Convert VHDs from dynamic to fixed
- Converting VHDs to VHDXs
- Syncing NUMA nodes on 2012 hosts & disabling old processor migration
- Disabling Parallel Processing
- Exporting VMs and re-importing
The only thing I haven't done is reformat the repository.
Any help would be most appreciated.
Thanks.
-Chris
-
- Veeam Software
- Posts: 21072
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Client Error: Failed to send command
Chris, can you check whether the VeeamVssSupport service is running on the affected guests? And if yes, try to stop and delete it.
-
- Influencer
- Posts: 10
- Liked: 1 time
- Joined: Sep 26, 2013 9:00 pm
- Full Name: PPPSGV IT Admin
- Contact:
Re: Client Error: Failed to send command
Hi Foggy,
It was running on 2 of my VMs. One on Host 1 and One on Host 2.
I stopped and disabled the service. Will try a VeeamZip to test.
Thanks,
Chris
It was running on 2 of my VMs. One on Host 1 and One on Host 2.
I stopped and disabled the service. Will try a VeeamZip to test.
Thanks,
Chris
-
- Influencer
- Posts: 10
- Liked: 1 time
- Joined: Sep 26, 2013 9:00 pm
- Full Name: PPPSGV IT Admin
- Contact:
Re: Client Error: Failed to send command
Nope. Jobs still failing.
I have a second site with a single Hyper-V host (Windows Server 2012) and a similar repository. The Veeam server is actually the repository for this site. This site has no issues at all. I even exported all VMs from an older host (Windows Server 2008 R2) to the new one and didn't upgrade the Hyper-V integration components, and it still completed all jobs successfully.
As another test, I connected a cheapo WD MyCloud single hard drive "NAS" and added it as a CIFS repository. VMs that fail normally, still fail. VMs that succeed, succeed here too.
Does anyone know what the actual error means? Could it be a networking issue, independent of Veeam? If so, why aren't all jobs failing? I can reliably backup 2 VMs with no problems, but all the others are failing.
Frustrating. . .
I have a second site with a single Hyper-V host (Windows Server 2012) and a similar repository. The Veeam server is actually the repository for this site. This site has no issues at all. I even exported all VMs from an older host (Windows Server 2008 R2) to the new one and didn't upgrade the Hyper-V integration components, and it still completed all jobs successfully.
As another test, I connected a cheapo WD MyCloud single hard drive "NAS" and added it as a CIFS repository. VMs that fail normally, still fail. VMs that succeed, succeed here too.
Does anyone know what the actual error means? Could it be a networking issue, independent of Veeam? If so, why aren't all jobs failing? I can reliably backup 2 VMs with no problems, but all the others are failing.
Frustrating. . .
-
- Veeam Software
- Posts: 21072
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Client Error: Failed to send command
What causes this error should be identified from the job debug logs, so please continue working with your support engineer.
-
- Influencer
- Posts: 10
- Liked: 1 time
- Joined: Sep 26, 2013 9:00 pm
- Full Name: PPPSGV IT Admin
- Contact:
Re: Client Error: Failed to send command
Just wanted to post a follow-up:
The problem appears to have been related to a pair of SonicWALL WXA WAN Accelerator appliances causing the final copying of job info from the Hyper-V host at Site 1 to the Veeam server at Site 2 to fail. TCP connections were being "accelerated" and causing connection drops. So even after all the data blocks had been successfully transferred, in some cases 1TB+ worth, it would still report as a failure and the restore point wouldn't be available.
Disabling the WXAs has allowed all my jobs to complete successfully.
The problem appears to have been related to a pair of SonicWALL WXA WAN Accelerator appliances causing the final copying of job info from the Hyper-V host at Site 1 to the Veeam server at Site 2 to fail. TCP connections were being "accelerated" and causing connection drops. So even after all the data blocks had been successfully transferred, in some cases 1TB+ worth, it would still report as a failure and the restore point wouldn't be available.
Disabling the WXAs has allowed all my jobs to complete successfully.
-
- Product Manager
- Posts: 20284
- Liked: 2258 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Client Error: Failed to send command
Thanks for coming back and updating the topic with the found solution; much appreciated.
Who is online
Users browsing this forum: No registered users and 28 guests