Failed to create snapshot HPE Storage VSS Provider

[pedroohenry]

Hello,

My open case with Veeam is 07774408.

For a little over 30 days, I have been trying to resolve a warning message and two jobs that I have. One backs up Linux VMs and the other backs up Windows VMs, both from a cluster consisting of two Hyper-Vs. While the backup is running, some errors referring to the HPE Storage VSS Provider appear:

Failed to create snapshot HPE Storage VSS Provider on XXX.com.br (mode: Veeam application-aware processing) Details: Failed to add volume [\\?\Volume{8e05d311-b8d4-4447-ad0e-7e3b267be9e2}\] to the VSS snapshot set. The shadow copy provider had an error. Check the System and Application event logs for more information.

I have read some other forum posts and also have an open case with Veeam, but it seems we are stuck and going around in circles.

The XXX server that appears in the Veeam logs is my off-host proxy server. It runs Windows Server 2022 Standard, and Hyper-V and VBR run Microsoft Windows Server 2022 Datacenter.

I list the provider with ‘vssadmin list providers’ and everything is OK.

My storage is an HPE 3PAR 8200 with OS version 3.3.2.159 (MU1)+P08,P10,P1. WSAPI is enabled and version 1.6.15.

The volumes are configured normally for snapshots, but what seems to be happening and causing problems is competition between snapshots on the same volume, generating these warning messages.

Changing volumes 1 and 2 of HV1 and HV2 to Microsoft CSV Shadow Copy Provider and changing the cluster volume to use Microsoft Software Shadow Copy Provider 1.0 on both Hyper-V did not work, and the following error was generated:

• Unable to allocate processing resources. Error: No hardware provider for off-host backup detected. Snapshot failover is disabled.

Support said that if the VMs being copied are stored on CSV, you will need to use the HPE VSS provider, as the files may be spread across multiple disks on the LUN, so the off-host proxy may not know which files are on which disks on the LUN. It essentially acts as a field guide to say, “Here's X, work with it.” If you choose a provider other than the storage volume, using an off-host proxy will not work, as the requirement for the storage provider to interact with CSV storage is crucial to this procedure.

Will using Microsoft's Shadow Copy Provider software prevent backups from using the SAN to read data (which happens when using the HPE Storage VSS Provider) and instead only use the Ethernet network?

I ask this because I know that using Microsoft's Shadow Copy Provider on-host, snapshots are generated successfully, but what I really wanted was to fix and use the HPE Storage VSS Provider to better leverage the proxy and remove the overhead on Hyper-V.

The other tests I performed were:

• I accessed the volume management settings for each Hyper-V in the cluster via Veeam and changed the “maximum simultaneous snapshots” field from 4 to 1 on all volumes.
• I removed and re-added VMs with warning messages to the task and ran the entire task wizard.
• I changed the order of the VMs to be processed in the task.

None of this solved the problem.

When reading the HPE Storage VSS Provider 2.7.x for Microsoft Windows User Guide, I found the following:

• The same set of snapshots containing VVs from different systems is not supported.
• HPE Storage VSS Provider cannot support volumes from different systems being processed within the same set of snapshots.

The HPE Storage VSS Provider installed on the servers is of the Hardware type, version 2.7.5.0.

The storage is as follows:

VV 1: VV_HV_CLUSTER_SITE_A_01
Exported to: HV1 and HV2

VV 2: VV_HV_CLUSTER_SITE_B_01
Exported to: HV1 and HV2

Both are part of the VVSet_HV_CLUSTER volume set.

Seeing this, I separated the tasks into Site A and Site B and, along with that, limited the maximum simultaneous tasks field in the proxy, setting it to 1. This worked, but I saw it as unfeasible because the backup will take longer and use the proxy very little.

Another suggestion given by Veeam support was KB 2196, disabling the simultaneous creation of shadow copy snapshots in the cluster for all providers by adding a ‘HyperVRestrictConcurrentSnapshotCreation’ registry key in VBR, but this seems to have only worked for the Linux job, as Windows still has warnings.

The Event Viewer logs do not suggest anything unusual:

• 3PARVSS5073: ERROR: Could not detect HPE 3PAR InForm CLI installation.
• 3PARVSS5065: ERROR: Target LUN HP is not a 3PAR Virtual Volume.
• Disk X has been unexpectedly removed

The last action I researched and used was the MaxVmCountOnHvHardSnapshot registry key, setting it to 14 and increasing the proxy's max concurrent tasks to 14 as well, but I still see warning messages.

Could anyone offer any further suggestions and help me resolve this issue?

I would be extremely grateful.

[david.domask]

Hi Pedro, welcome to the forums.

Thank you for the detailed post and for sharing the case number. Will answer your questions here first, but as the case appears to be in the process of escalation to the Advanced Support Team, will ask you please continue with Support.

> Will using Microsoft's Shadow Copy Provider software prevent backups from using the SAN to read data (which happens when using the HPE Storage VSS Provider) and instead only use the Ethernet network?

For Off-Host backup proxies, you must use the Hardware VSS Provider your storage vendor provides; the built-in Microsoft VSS providers will not work with off-host backups. Please see the requirements here

As for further suggestions, at this time best option is to continue with Support to review the warnings/errors, as we won't be able to troubleshoot over the forums.

Similarly, it's worth noting that our best practice is to use On-Host proxies as opposed to off-host; the performance difference is typically negligible and off-host proxies can be rather complex as you're seeing here. Is using on-host proxies an option in your environment?