Veeam Backups failing - Dell Compellent VSS Hardware Providers

[mhugradar]

Hi
so here is my scenario:


2 Hyper-V hosts in a cluster – W-SRV-HYPERV3 & W-SRV-HYPERV4. Both running Windows Server 2012 R2.

Both attached via 100GB iSCSI to the SC5020.

Mixture of VMs ranging from 2008 to 2019.

Veeam installed on W-SRV-HYPERV4.

The following applications installed on both Hyper-V hosts
• Dell Storage Replay Manager
• Dell Storage Replay Manager Management Tools
• Dell Storage Replay Manager Verification Services

If a VM is running on W-SRV-HYPERV3, Veeam is able to backup the VM using the Compellent Replay Manager VSS Provider 100% of the time.

If a VM is running on W-SRV-HYPERV4, Veeam will always give the following message:

Failed to create snapshot (Compellent Replay Manager VSS Provider) (mode: Veeam application-aware processing) Details: Unknown status of async operation.
The shadow copy provider had an error. Check the System and Application event logs for more information.
--tr:Failed to create VSS snapshot.
--tr:Failed to perform pre-backup tasks.

If I move a VM that failed on W-SRV-HYPERV4 to W-SRV-HYPERV3, the backup then works fine.

I have a ticket open with Veeam and they suggested contacting Dell because of the VSS provider that is being used. Its also worth mentioning, the reason we are using the Compellent Replay VSS provider is because the volume which hosts the VMs is larger than 64TB which I believe is the limit for Windows VSS provider.
Both DELL and VEEAM are sending me to each other and we are not getting any closer to a resolution.

following on from this, I was advised to upgrade both hosts to Server 2019, this has now been completed yet the same issue still remains.

Does anyone have any ideas?

I could install VEEAM on a VM instead and attempt this, but i assumed this would be the fastest way to get backups.

[Andreas Neufert]

Regarding the VM usage for the Off-Host Proxy.
It is documented here:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110


On the same page, you will find this sentence:
"If you plan to perform off-host backup for a Microsoft Hyper-V cluster with CSV, you must deploy an off-host backup proxy on a host that is not a part of the cluster. If the off-host backup proxy is deployed on a node of the cluster, the cluster will fail during VM data processing."
This is not a Veeam limitation, but it is by design of the VSS Framework from Microsoft.

Can you please share for the reference your support ticket number?

[Andreas Neufert]

Just an idea, you could make the other Hyper-V server an offhost proxy as well and select the opposit host as proxy for the VMs.
It would have some management overhead but would maybe work.

[mhugradar]

The VEEAM Reference is: 04930509



Whilst VEEAM is installed on the HV Host, if i select Off Host i still get the error, but also i noticed it wont display the backup proxy servers in the list. (Within the backup job)

I only have 2 hosts though, and both are part of the cluster, so does that rule out Of Host Backup whilst Veeam is on a VM?

[Andreas Neufert]

Off-host backup of CSV volumes need a third hardware server for off-host prox processing.
In the link you find a statement related to the VM usage as off-host proxy.

[mhugradar]

ok, i dont have a 3rd host to be able to use that method then unfortunately.

[mhugradar]

Update: the names were changed on the Hyperv Hosts

W-HV01
W-HV02 - VEEAM Installed

some weird combinations going on now

it seems any new Virtual Machine running Windows Server 2019 on Configuration Version 9.0 - backs up fine.

If the server is a prev OS version with Config 5.0 then it fails with that error

I found this out because, i was able to back new servers initially that were not in the CLUSTER yet, and i was wondering why non cluster machines are working but not VMs in the Cluster.

The problem i have now is, i can upgrade the Config on Servers running 2019, but what do i do about servers running 2012 R2 on 5.0!!

[Andreas Neufert]

We are not performing anything special for the VMs. It is all about the VSS framework and how the hardware vendor has implemented their own hardware VSS provider.
We know that it is not working in any case and as I shared above, it is not officially supported.

[mhugradar]

i tested a Windows Server 2012 running config file 5.0 and the backup failed

I upgraded the config file to 9.0 and backup succeeded ..

[Andreas Neufert]

Thanks for sharing this. Please remember that in the end this configuration is unsupported by definition within our documentation (see above link).

[eengland09]

Can someone confirm that the DellStorageReplayManager is the only .msi file needed on the Hyper-V nodes for the Dell EMC SC Series integration in Veeam to work together to do the snapshot? Trying to make sure I have everything understood before installing this on our nodes. This is all in attempt to get around the 64TB VSS limitation.

[Andreas Neufert]

I would check with Dell on this one.

[eengland09]

8/23/2024 3:54:35 PM :: Retrying snapshot creation attempt (Failed to add volume [\\?\Volume{3fae08e5-c5c1-4868-be7d-30760e0d5a20}\] to the VSS snapshot set
The shadow copy provider had an error. Check the System and Application event logs for more information.
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.)


maybe I need to upgrade os from 2012 to 2019 to get to 9.0 from 5?