During this weekend I created a new job with vm's isolated on their own cluster node and a separate CSV.
Those vm's still stops ungraceful and are restarted on other hosts, but the others seem to be fine during and after backup.
After the job had been running for about half an hour the following events started
Event ID 1230
A component on the server did not respond in a timely fashion. This caused the cluster resource 'Virtual Machine vm-name' (resource type 'Virtual Machine', DLL 'vmclusres.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource. Verify that the underlying infrastructure (such as storage, networking, or services) that are associated with the resource are functioning correctly.
Event ID 1146
The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which resource and resource DLL is causing the issue.
Event ID 1069
Cluster resource 'Virtual Machine vm-name' of type 'Virtual Machine' in clustered role 'vm-name' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.
Event ID 21502
'Virtual Machine Configuration vm-name' failed to unregister the virtual machine configuration during the initialization of the resource: The wait operation timed out. (0x00000102).
After a lot of these events all virtual machines in the job ends up being restarted on other hosts.
Those errors are only seen during backup, when I initiate migration in VMM everything runs fast and smooth.