Backup Error : Failed to convert snapshot to referencePoint

[Fabian C.]

Hello,

I'm currently facing a problem with a backup job that keep failing with this error : "[WMI] Empty result. Failed to convert snapshot to referencePoint snapshot"
Open Ticket : #02171150

The environment :
- Brand new Windows server 2016 with Hyper-v role.
- 2 VM installed on it :
* Our principal called "Main" with : File server, Exchange, DC
* Another one called "DMZ" : Internet Fronted server (IIS web services mainly).

Lost Monday I manually restarted a Full backup, before this everything was working perfectly.
The problem : The backup to disk job of the "Main" VM fall in error because of missing disk place AND the VM "Main" itself fall in pause statement (Both VM and backup to disk share the same disk).
I cleaned up previous backup from disk to free same place.
When I tried to resume the VM, it failed. I had to restart the VM server.
After reboot, both VM (Main & DMZ) did restart and everything was back on track (From network and services point of view).

Except that from that moment any single time I tried to start a backup or replication job of the "Main" server it fall in error, while the other VM - DMZ (on the same server) is backing up perfectly.

The backup start correctly, the snapshot is created, the HDD are copied (This can take some time), but after the HDD process the error is raised.
After the backup my snapshot is merged back to the VM, and everything is fine for the VM point of view.

Here is a typical Veeam log :

Code: Select all

Queued for processing at 6/6/2017 11:01:18 PM 
Required backup infrastructure resources have been assigned 
VM processing started at 6/6/2017 11:01:23 PM 
VM size: 85.0 GB 
Preparing to create snapshot 
Creating VM recovery checkpoint (mode: Hyper-V child partition snapshot) 0:00:14
Using source proxy hyper-v1 (onhost) 
Saving config.wmi 
VMCX configuration file (163.8 KB) 163.8 KB read at 164 KB/s
VMRS configuration file (88.1 KB) 88.1 KB read at 88 KB/s
Hard disk 1 (85.0 GB) 65.5 GB read at 78 MB/s [CBT]0:14:26
Error: [WMI] Empty result. Failed to convert snapshot to referencePoint snapshot path '\\HYPER-V1\root\virtualization\v2:Msvm_VirtualSystemSettingData.InstanceID="Microsoft:B70A2C57-54A6-405F-8CBC-463673F9C852"'. 
Busy: Source 98% > Proxy 2% > Network 0% > Target 0% 
Primary bottleneck: Source 
Network traffic verification detected no corrupted blocks 
Processing finished with errors at 6/6/2017 11:16:32 PM 

The error trace from the log :

Code: Select all

[01.06.2017 13:56:05] <14> Error    Processing MAIN
[01.06.2017 13:56:05] <14> Error    [WMI] Empty result.
[01.06.2017 13:56:05] <14> Error    Failed to convert snapshot to referencePoint snapshot path '\\HYPER-V1\root\virtualization\v2:Msvm_VirtualSystemSettingData.InstanceID="Microsoft:24C334C4-D1DD-4DBD-88A0-74351C0067D6"'. (Veeam.Backup.ProxyProvider.CHvWmiProxyErrorException)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.ProxyProvider.CHvWmiReconnectableRemoteCommand.InvokeInThread(Delegate dlg, Object[] args)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.ProxyProvider.CHvWmiReconnectableRemoteCommand.DoInvoke(CHvWmiProxyRequestContextNdw context, Int32 reconnectsCount, Delegate dlg, Object[] args)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.ProxyProvider.CHvWmiReconnectableRemoteCommand.Invoke[Ret](CHvWmiProxyRequestContextNdw context, Func`1 dlg)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.ProxyProvider.CHvWmiSnapshotRemoteManager2015.ConvertToReferencePoint(CHvVmSnapshotInfo snapshot)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointHelper.ConvertToReferencePoint(CHvVmID vmId, Boolean succeeded)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.ResourceScheduler.CHvRecoveryCheckpointSnapshot.ConvertCheckPointToReferencePoint(Guid vmId, Boolean isBinaryVmConfig, Boolean taskSucceeded)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.CHvVmSource2015.AfterDisksTransfer(Boolean succeeded)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.CBackupVmTransferAlg.AfterDisksTransfer(CBackupSet backupSet, Boolean succeeded)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.CTaskProcessor.ProcessTask(IVmBackupTask task, ITasksScheduler tasksScheduler, LogStorage logStorage, IStopSessionSync stopSessionControl, Boolean disableParallelProcessing)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.CHvBackupJobPerformer.ExecuteTask(CHvVmTask task, ITaskSource source, CVmBackupTarget target, LogStorage logStorage)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.CHvBackupJobPerformer.ExecuteTask(CHvVmTask task)
[01.06.2017 13:56:05] <14> Error       at Veeam.Backup.Core.CHvBackupJobPerformer.ExecuteTaskAsync(CHvVmTask task)

The internal windows backup run perfectly on this VM.

Do you have idea where the problem is coming from ?
Test I could do ? (VSS writers are fine)
I have already spend some hours with phone support.

Thank you !

Fabian Céréssia.

[Fabian C.]

Quick answer concerning my latest test :

- I have install Windows server 2016 on an old server with HyperV role.
- Install Veeam on it.
- Restore the latest VM backup realized with Windows Server Backup.

Start a Veeam backup, that finished without error !

Could it be a veeam installation corruption ? Or the VM itself (not the content) ?

[foggy]

Hi Fabian, please continue investigating with the help of support and ask for case escalation if required.

[Fabian C.]

Hello,

While my case is still under investigation (And prevent any backup from my main server directly from Veeam Backup & Replication 9.5).
I decided to install "Veeam Agent for Microsoft Windows FREE" inside this server.

I configured it to push the backups inside the Veeam Backup & Replication repository.
So far so good, everything is running perfectly.

Some questions :
- I'm safe to keep on running this type of backup from my server, and give me the possibility to rebuild easily a working VM ?
- As I don't have a licence for this product, I'm using the free version. What are the downside of using this free version compared to the full one in my case ?

Thank you for your help

Fabian Céréssia

[Mike Resseler]

With an entire backup of your MAIN server you can do a few restore options:

1. bare metal restore. So what you will need to do is to create the recovery media (as an ISO) as fast as you can. Store it somewhere and then boot a fresh VM on that ISO. Then do a bare metal restore and restore the entire VM.
2. You can do instant restore to Hyper-V (https://helpcenter.veeam.com/docs/agent ... tml?ver=20)
3. You can export the backup as VM disks. Create a new VM and attach the disk.

The downside of the free version is the way application awareness is handled and support. Free version gives you basic VSS quiescence. Paid versions will give you advanced functionality. And then obviously support. Free is on a best effort basis, paid versions give you the support you probably want and need as a business. And then there are other differences also (File indexing, search, better scheduling...) See here for more info: https://www.veeam.com/veeam_agent_windo ... _en_ds.pdf

From the log that you posted, I notice that there is something wrong with the reference point which probably means your RCT (Resilient change tracking, which is the MSFT version of change block tracking) has an issue. Hence the reason why Windows Server Backup worked (it doesn't use that system as we do) So keep working with support but also, are your Hyper-V hosts up-to-date with the latest monthly updates? During the last 6 months some fixes came out for RCT and the new Backup architecture in server 2016

Thanks
Mike

[Fabian C.]

Hello,

Tx your for the information, I'm just trying to find a temporary backup solution.
The concerned servers are all up to date.

I ran an instant restore to Hyper-V to a Test environment today that went fine.
By the way the restored VM can be backup successfully directly from Veeam Backup & Replication repository on the test env.

I keep in touch with support, last action was to send all windows logs.
BTW here are some logs that keep on being generated and that concern the VM that can't be backup (even when no backup job is running) :
From Microsoft-Windows-Hyper-V-VMMS/Analytic

Code: Select all

Level,Date and Time,Source,Event ID,Task Category
Error,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1202,None,throw VmGeneralSystemException((DWORD) 2) : The system cannot find the file specified.
Error,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1102,None,Failed to initialize reference points of virtual machine 03C0773A-AB66-417E-9F74-4508E2A210EA. HRESULT = 0x80070002
Error,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1202,None,throw VmGeneralSystemException((DWORD) 2) : The system cannot find the file specified.
Error,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1102,None,[VmmsRealizedVirtualMachine::QueryReferencePointsInternal()@16631] Caught class Vml::VmGeneralSystemException: The system cannot find the file specified. (0x80070002)
Error,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1202,None,throw VmGeneralSystemException((DWORD) 2) : The system cannot find the file specified.
Error,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1102,None,[FailoverReplicationTracker::DeleteOrphanedContinuousDataPoints()@8083] Caught class Vml::VmGeneralSystemException: The system cannot find the file specified. (0x80070002)
Warning,6/12/2017 11:10:27 PM,Microsoft-Windows-Hyper-V-VMMS,1101,None,[FailoverReplicationTracker::DeleteContinuousDataPointsCallbackInternal()@8134] MAIN::03C0773A-AB66-417E-9F74-4508E2A210EA DeleteOrphanedContinuousDataPoints failed on retry. Will retry again.

[Fabian C.]

With an entire backup of your MAIN server you can do a few restore options:
...
From the log that you posted, I notice that there is something wrong with the reference point which probably means your RCT (Resilient change tracking, which is the MSFT version of change block tracking) has an issue. Hence the reason why Windows Server Backup worked (it doesn't use that system as we do) So keep working with support but also, are your Hyper-V hosts up-to-date with the latest monthly updates? During the last 6 months some fixes came out for RCT and the new Backup architecture in server 2016
...

By the way I have already tried to run a Veeam backup with CBT, guest quiescence, and guest processing disabled. It doesn't fix the problem.

[Fabian C.]

Hello again,

Still investigating the original problem caused by a "crash" of the VM because of a disk full during backup operation (VM and backup were running on same disk).

As exporting the VM (Windows backup) and mounting it to a new server is working, one thing that is not exported are the MRT/RCT file generated with the HDD while a backup is started.
Is there a way to regenerate them in the case of corruption (Or they can be deleted simply ?) ?

Regard.

[Mike Resseler]

Fabian,

change block tracking in Hyper-V happens at 3 levels.
1. Inside a memory bitmap (which is lost with a crash or VM migration or similar)
2. MRT file (bitmap on the same volume as the VHD(X)
3. RCT file (bitmap on the same volume as the VHD(X)

Theoretically, if a VM crashed, the memory bitmap is lost, and the RCT file might be corrupted, but the idea around the MRT file is that it still should continue to work. Unfortunately this doesn't seem to be the case. Maybe something that we need to pass to MSFT.

If my theory servers me well, what will happen with the scenario that you did, is that a new full backup will be done, and at that point in time I expect a new RCT/MRT file to appear in the same directory as the VHD{X) and from there on it will be incremental again.

To be save, I personally would not delete those files just yet, take a full backup, and then see if new RCT/MRT files appear. If they do, you can remove the old ones. If they don't, then I think something else is still wrong

Mike