Host-based backup of Microsoft Hyper-V VMs.
Post Reply
ccone@vascorltd.com
Influencer
Posts: 19
Liked: 2 times
Joined: Sep 08, 2017 1:47 pm
Full Name: Clint Cone
Contact:

Veeam 9.5 with Hyper-V 2016 intermittent issues

Post by ccone@vascorltd.com »

I am seeing two issues backing up VMs on our new Hyper-V 2016 cluster. They are intermittent in that they do not consistently affect the same VM(s) or physical servers. We have 4 identical servers in the Hyper-V cluster, and a total of about 130 VMs.

Hyper-V cluster info:
Cisco UCS C460 M4 servers, iSCSI booting from PureStorage. Hyper-V 2016, all updates applied. Firewalls disabled temporarily for job troubleshooting.

Veeam configuration:
Physical server - 2 X Xeon E5-2960 v2 (20 total cores @ 3.0 GHz) - 192 GB RAM - 10 Gb/s fiber LAN connection (Emulex).


The only Veeam backup job that encounters the errors is our main nightly backup (it processes 64 VMs). We never saw these errors against the Hyper-V 2012 R2 clusters we are replacing.

1 - Error: Only one usage of each socket address (protocol/network address/port) is normally permitted [IP Address of physical Hyper-V node ]:6202
- I changed the port range used to 6200 - 7500 as I noticed that our new cluster was using most of the 2500 - 5000 range for cluster communications. I'm still trying to figure out why. NETSTAT reported nothing using the port range I chose to use.
- Typically only see this error once. during the job.
- Subsequent job runs the error would occur on a different VM and/or different physical host.

2 - Error: The RPC server is unavailable. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP Address of physical Hyper-V node ].
- Again, only typically see this error once during the run of the job. All other VMs on the same host back up just fine.
- Subsequent runs of the job (or even just a retry), the error will not be encountered. Or if starting the entire job over, could be thrown processing a different VM, on a different physical server.

3 - Error: Hyper-V Integration Service is not accessible through the network on source host [Physical Hyper-V host name].
- Disabling Windows Firewall seems to have corrected this, so I need to know which rule this depends on.
- What is strange is that it is encountered only on 1 or two VMs located on a particular host, but the rest of the VMs on that host work fine. And on re-runs of the job, it would be on different VMs, but usually the same physical host.
- The firewalls on all hosts are set up the same, so it makes no sense to only see it on one physical host.


I have disabled the firewall completely, to rule it out as a cause. This seems to have helped with the last one only.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam 9.5 with Hyper-V 2016 intermittent issues

Post by PTide »

Hi,
main nightly backup (it processes 64 VMs).
Does any other job include that many (or more) VMs? How many disks do the VMs have in average?
Error: Only one usage of each socket address (protocol/network address/port) is normally permitted [IP Address of physical Hyper-V node ]:6202
Error: The RPC server is unavailable. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP Address of physical Hyper-V node ].
These two might be related to each other and have something to do with the workload during the backup job session. I'd recommend to contact support team directly. Once you do please post your case ID.

Thank you
ccone@vascorltd.com
Influencer
Posts: 19
Liked: 2 times
Joined: Sep 08, 2017 1:47 pm
Full Name: Clint Cone
Contact:

Re: Veeam 9.5 with Hyper-V 2016 intermittent issues

Post by ccone@vascorltd.com »

I think we got this sorted ourselves. Our new Hyper-V machines were having network issues related to SR-IOV. The errors observed seemed to go away after we did a rolling restart of the cluster. Then they would reappear a few days later when the cluster needed another restart. The underlying issues were not VEEAM related. We had a driver/firmware incompatibility issue. We may still have the underlying issue and are working with the hardware provider on it. We have not seen any of these errors since applying matching firmware/drivers to the hosts.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 22 guests