Struggling with CSV failover during Backup Job (Hyper-V, ISCSI HP SAN) Case ID 03691396

[hmcit]

Case ID 03691396

Hi All,
I have recently set up a 2 node Hyper-V Cluster with the following specs
2 nodes running Windows Server 2019 Datacenter Edition (with GUI)
Each node had 4 network port set up as follows
192.168.1.X VM/Host LAN
192.168.120.X Cluster Network (on a vlan, HP managed switch)
192.168.121.X Live Migration Network (on a vlan, HP managed switch)
10.0.0.X ISCSI Connection to HP SAN (on a vlan, HP managed switch)
Veeam is installed on one of the host nodes and is running a backup job each night to an attached USB3 RDX drive. This works great for the most part and the cluster is always been stable EXCEPT every couple days during the backup I start to get errors and for a fraction of a second the node loses contact with the ISCSI connection causing a CSV failover. The errors are related to the CSV going into a paused state. I have done a lot of trouble shooting to try and resolve this, for one I have disabled ODX on the host nodes as the HP SAN apparently does not support this. I have also set up the back up job to only do one Snapshot at a time, I am currently backing up 7 VMs in total consisting of 4 Windows Server 2016, 1 Windows 10 pro and 2 ubuntu servers. Other than this problem with backup everything has been great but this is holding us up from being able to migrate or server roles to the cluster (we have always been a physical machines only environment). Also the backup it self is always successful.
Any thoughts or suggestions would be a great, sorry if I left out any helpful info.

[HannesK]

Hello,
please understand that the forum does not replace support.

Veeam is installed on one of the host nodes

that's the only suggestion I have: fix the setup. Applications should not be installed in the Hyper-V root partition and I'm not even sure whether Microsoft supports that.

Keep in mind that the GUI you see at a Hyper-V host is a VM...



Best regards,
Hannes

[hmcit]

Thanks for the reply Hannes, I thought installing B&R on the host was OK? This seems to say its alright https://helpcenter.veeam.com/docs/backu ... l?ver=95u4 and I would have considered this a "tape backup" type program and that seems to be OK with Microsoft. Actually originally I was going to use LTO tapes to do the backup but then we got the RDX drive. Do you have any source that counters that? I can set up a new VM on the cluster to do a test install and try that if needed. I'm not sure if I will be able to pass though the RDX drive to the VM though, I guess I will just have to try. thanks

[HannesK]

I thought installing B&R on the host was OK?

would you install a SQL server directly on the ESXi console if it was possible theoretically?

the user guide says

on a physical or virtual Windows-based machine

it does not say "in the root partition of the Hyper-V host"

[hmcit]

I must be Misinterpreting it then, I saw where it says "If you decide to use a simple deployment scenario, you can install Veeam Backup & Replication right on the Hyper-V host where VMs you want to work with reside." But you make a great point about the SQL server bit, I had not really even thought about the fact that it installs SQL express. I might install it on to a desktop machine with the rdx drive to testing and if that resolves the problem and then try to figure out the bit about making the RDX drive a cluster resource and assigning to a new VM as the backup server. Thanks for your help.

[HannesK]

good point, I will ask the documentation team to clarify.

[SSolutions337]

Hi hmcit,

We are having a similar issue with CSV Failover during backup jobs after the most recent B&R update. Were you able to find a solution to your issue?

[hmcit]

We were. In our case I decided to create a VM just to run B&R and disabled the job that was running on the host. Since we were unable to easily pass the RDX drive to the VM and I had an extra empty drive on the host I ended up sharing that empty drive out as a network-share and running the backup to that drive then I set up a 2ed Copy Job to copy it to the DRX drive so that we can take it off site. Since then we haven't had any problems with the cluster pausing drive access or failing over (so far). I realized afterward that I could have just sent the 1st Backup job directly to the RDX drive over the network but keeping it going to the drive first helps us achieve the 3-2-1 Backup setup.