VMs protected by CDP Policy will power off if you try to do a vMotion

[filipsmeets]

Case #05144209

As mentioned in the subject, VMs protected by CDP Policy will power off if you try to do a vMotion. vMotion does not occur and the VMs gets restarted by vSphere HA.

I see the following events in vCenter:

Task Name
Migrate virtual machine
Status
Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout. 2021-11-22T11:00:13.91759Z Remote connection failure Failed to establish transport connection. Disconnected from virtual machine. 2021-11-22T11:00:13.751014Z vMotion migration [184164886:7352562474509221539] the remote host closed the connection unexpectedly and migration has stopped. The closed connection probably results from a migration failure detected on the remote host.

Error stack:
Remote connection failure
Failed to establish transport connection.
Disconnected from virtual machine.
2021-11-22T11:00:13.751014Z
vMotion migration [184164886:7352562474509221539] the remote host closed the connection unexpectedly and migration has stopped. The closed connection probably results from a migration failure detected on the remote host.
Failed waiting for data. Error 195887167. Connection closed by remote host, possibly due to timeout.

The virtual machine was restarted automatically by vSphere HA on this host. This response may be triggered by a failure of the host the virtual machine was originally running on or by an unclean power-off of the virtual machine (eg. if the vmx process was killed).

[Andreas Neufert]

I woudl speak with VMware why the vMotion failed. Error 195887167 is a VMware error.
There are many how to check and solve this error when you google. A good one that gives you examples what to test are here:
https://www.jordansphere.co.uk/vmotion- ... figuration.
As well VMware KB:
https://kb.vmware.com/s/article/2105786

[filipsmeets]

There are no issues with vMotion. As soon as I disable the CDP Job, I can vMotion the VMs without issues.
I also checked vMotion network between the host by using vmkping.

[Andreas Neufert]

Are you sure that the Veeam IO Filters (and potentially other filters that you use) are on the target host installed?
Test it by creating a empty VM and replicate it with Veeam CDP to check that this is possible and the filters are in place.

[filipsmeets]

Hi Andreas
I have checked the IO filters on the source and destination cluster and they are all up to date.
Replication actually is working.

I have 2 test VMs running in Datacenter DCR1 under the Cluster Workloads. This Cluster contains 5 ESXi Hosts and has DRS and HA enabled with the default settings. VMs are running on a shared storage (FC SAN).

I then created a CDP Policy to replicate these two VMs to Datacenter DCR2 under the Cluster Workloads. This Cluster also contains 5 ESXi Hosts with DRS and HA enabled with the default settings.

While these VMs were protected by the CDP policy, I tried to vMotion one of the original VMs in DCR1 to another host in the same cluster. I then noticed the VM got powered off and after a while got restarted by vCenter HA.
While looking at the CDP Job in Veeam, I could see it switching over to CBT.

I hope this makes some more sense.

[Andreas Neufert]

Do you use multiple NICs within the VMKernel interface that is responsible for vMotion?

[filipsmeets]

Each host has 3 vmkernels. One for management, a second for vMotion and a third for Veeam.
The vmkernels are connected to 3 different port groups with different subnets/vlans on a distributed switch with two active uplinks. No LACP. Load balance policy is set to physical NIC usage.

[Andreas Neufert]

I see that the support team has scheduled a remote session with you. Let´s see what they can find.

[Andreas Neufert]

Hi support logs show that there is an issue on the VMware side with a crashing application.
"An application (/bin/vmx) running on ESXi host has crashed "

Here is an example on how to solve it.
https://communities.vmware.com/t5/VMwar ... -p/1863755

[werner.vergauwen]

small feedback

the issue is now resolved. There was a bug registered at Veeam (Bug 368345)
A private fix was delivered for 11.0.1.1261, which needed to be installed on the VBR.

We have upgraded our Veeam environment to the proper level, applied the hotfix, and lo and behold, no more VM shutdown during vmotion.

[Andreas Neufert]

Thanks for coming back here and updating the entry.