We have already opened a Case # 07050740, just thought I'd start a thread here too to see if anyone else has encountered similar issues.
The job fail error message for specific VMs is:
Code: Select all
5:04:04 PM Failed to create VM recovery checkpoint (mode: Hyper-V child partition snapshot) Details: Unknown status of async operation The shadow copy provider had an unexpected error while trying to process the specified operation. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks. 00:20
5:04:24 PM Retrying snapshot creation attempt (Unknown status of async operation The shadow copy provider had an unexpected error while trying to process the specified operation. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks.)
Veeam version: 12.1 running on Windows Server 2022
Hyper-V version: Windows Server 2019
All hosts in the Hyper-V cluster store all VMs in one of the two SMB3 paths:
\\NETAPP1\...
\\NETAPP2\...
The above two SMB3 storage paths are not backed by Windows Servers, but are backed by two NetApp hardware NAS storage.
Both NetApp NAS are running ONTAP 9.8.
To the best of my knowledge, both NetApps are configured identically when it comes to SMB3/CIFS related configs.
Problem:
- Any VMs still using Configuration Version 5 AND stored on \\NETAPP1 will get backup error.
- Any VMs using Configuration Version 9 AND stored on \\NETAPP1 will backcup successfully.
- Both ver5 and ver9 VMs will backup successfully if they are stored on \\NETAPP2.
- Even ver5 VMs that got backup errors when they were on \\NETAPP1, if storage-migrated to \\NETAPP2, will backup successfully.
- If ver5 VMs storage-migrated back to \\NETAPP1, will get backup error again.
- If a ver5 VM in \\NETAPP1 is in-place upgraded to ver9, then the backup will now complete successfully.
We have already tested and isolated out the following:
- Enabling or disabling CBT in the Job settings will not affect the behavior.
- Enabling or disabling "Allow processing of multiple VMs with a single volume snapshot" in the Job settings will not affect the behavior.
- Enabled or disabling Hyper-V guest quiescenc with/without crash consistent in the Job settings will not affect the behavior.
- The VM Generation version does not affect the behavior. Does not matter if the VM is Gen1 or Gen2 version.
- Whether the VM is Windows or Linux does not affect the behavior.
- Whether the VM is powered-on or shutdown does not affect the behavior.
- Whether the VM is using .VHD or .VHDX virtual disks does not affect the behavior.
- Does not matter which Hyper-V host in the cluster the VM is running on, does not affect the behavior.
- Does not matter if the VM is a really old ver5 VM, or if it is a newly created ver5 VM, does not affect the behavior.
- All VMs can successfully create production-checkpoints or standard-checkpoints when done so manually in Hyper-V's management interface.
veeantest1 - ver5 VM running on \\NETAPP1 (backup will always fail, unless storage-mirated to \\NETAPP2)
veeantest2 - ver9 VM running on \\NETAPP1 (backup will always succeed)
veeantest3 - ver5 VM running on \\NETAPP2 (backup will always succeed, unless storage-migrated to \\NETAPP1)
veeantest4 - ver9 VM running on \\NETAPP2 (backup will always succeed)
Currently I'm in the process of collecting guest OS logs for the support... Which I thought was strange because this issue happens whether the guest OS is powered-on or powered-off... Anyhow, let's see where this leads us.