The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem..

[mooseracing]

We are getting this error on two of the five VM's on a Hyper V 2012 host. One guest is Server 08R2 and the other Centos7.

I've made sure Application aware is unchecked, restarted the Hyper V Mgmt Service, VM's have enough free space (plenty but not sure on a specific number required).

We are trialing the software so I will admit limited knowledge here. What would be the next places to look at for errors? I'm going to try and kick off a Windows server backup and see if it pukes as well.

[nmdange]

Check the event log inside the VM. This is usually an issue with VSS within the guest OS. You can try using Windows Server Backup within the VM as a test. Also make sure you've updated the Hyper-V Integration Services to the latest version.

For Linux VMs, to do "VSS" backups you need either the Linux Integration Services or the Hyper-V backup daemon. This page has some info on that: https://docs.microsoft.com/en-us/window ... on-hyper-v

Also, if you are on Hyper-V 2012 R1, and not Hyper-V 2012 R2, you really should upgrade your host as the backups are more reliable on 2012 R2. 2016 also brings some significant improvements as well.

[mooseracing]

The host server is 2012R2, will be 2016 with some time as we have moved some of the other hosts to 2016.

The Windows VM is running the latest integration services. I restarted the services for the timed out VSS writers, they get stuck timing out when running a windows backup as well. Looking further into that.

[mooseracing]

Turning off Backup IS in the VM options on Hyper V allows the backup to complete. Turning them back on fails again.

[foggy]

Please contact technical support and share your case ID here for reference.

[jazzoberoi]

Hi Foggy,

We need urgent help with this issue.

My Case # 02418428.

Hyper-V Cluster Hosts and the Guest FileServer VM both are Win 2012 R2 with latest updates.

Code: Select all

14/12/2017 8:09:37 AM :: Queued for processing at 14/12/2017 8:09:37 AM  
14/12/2017 8:09:38 AM :: Required backup infrastructure resources have been assigned  
14/12/2017 8:09:42 AM :: VM processing started at 14/12/2017 8:09:42 AM  
14/12/2017 8:09:42 AM :: VM size: 10.4 TB (2.7 TB used)  
14/12/2017 8:09:44 AM :: Preparing to create snapshot  
14/12/2017 8:09:46 AM :: Failed to create snapshot (HP 3PAR VSS Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
14/12/2017 8:22:00 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
14/12/2017 8:22:02 AM :: Task has been rescheduled  
14/12/2017 8:22:14 AM :: Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
14/12/2017 8:32:56 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
14/12/2017 8:33:01 AM :: Queued for processing at 14/12/2017 8:33:01 AM  
14/12/2017 8:34:03 AM :: Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
14/12/2017 8:34:03 AM :: Network traffic verification detected no corrupted blocks  
14/12/2017 8:34:03 AM :: Processing finished with errors at 14/12/2017 8:34:03 AM  

Manually running the VSS creation command from within the Guest VM seems to be working.



Please help !

[Mike Resseler]

Hi Jazz,

First: It is good that you have a support case. Keep working with our engineers as they will be able to find much more valuable information in the logs then we can see here. From what I can see in the log, it is not a problem inside the VM (as you have shown also) but there is a problem with the CSV VSS writer (which will be on the host).

Cheers
Mike

[jazzoberoi]

Hi Mike,

I managed to resolve the issue by creating a new Windows 2016 Server VM for the fileserver and migrating all the data to it.

This new VM runs from the same host and the same 3par LUN but yet it backs-up perfectly fine.

This indicates that the issue has always been within the VM itself and not the host or the 3par storage array.

I've attached the logs for the backup job for the last 7 days to my case for your reference as well

[AlexLeadingEdge]

We are having the same issue, seeing this on about 30% of all VMs. I opened case # 02378199 and the Veeam technician said it was a Microsoft issue and couldn't help "as it is outside the scope of Veeam technical support"

[Mike Resseler]

Alex,

30 percent is a lot, really a lot. Are these all the same OS? And are these on the same host? Can you verify what the writers on the host are saying? (VSSadmin list writers)

[Rumple]

In order to get my machines to backup with this error, I had to shutdown the VM and disable the Backup (volume snapshot) integration in the hardware settings.
Even though I was not using application consistent in the backup job, it still seemed to be trying to call the vss provider or something and failing.

once I disabled that setting, then I was able to run the backup with only the warning (mode:crash consistent) being present.

[PTide]

Even though I was not using application consistent in the backup job, it still seemed to be trying to call the vss provider

Correct. During a Hyper-V VM backup, Hyper-V VSS writer calls the Hyper-V Volume Shadow Copy Requestor inside the guest OS. The requestor triggers all the VSS writers inside the guest to quiesce the system for the Hyper-V snapshot.

[AlexLeadingEdge]

Alex, 30 percent is a lot, really a lot. Are these all the same OS? And are these on the same host? Can you verify what the writers on the host are saying? (VSSadmin list writers)

Different Hyper-V Hosts (but same OS version), three different Guest VM OS's. Both the Host and the Guests have the Microsoft Hyper-V VSS Writer in a failed state. Getting the services on the Guest to restart 15 minutes before the backup starts allows the backups to work correctly (I have a batch file running in Task Scheduler), but the cause of the issue is still unknown to me. I haven't bothered to reboot the two Hyper-V Host's Microsoft Hyper-V VSS Writer service, mainly because I'm not sure what that might do to all the running VMs.

J Host 1 = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)
C Server = Microsoft Windows Server 2008 R2 Standard Edition, 64-bit Service Pack 1 (build 7601)
P Server = Microsoft Windows Server 2012 Standard Edition, 64-bit (build 9200)
Exchange Server = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)

S Host 2 = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)
M2 Server = Microsoft Windows Server 2008 R2 Standard Edition, 64-bit Service Pack 1 (build 7601)


I have upgraded all of the Guest VMs on J Host 1 to the latest Hyper-V Integration Services but I can't do S Host 2's M2 Server as it has to be planned days or even weeks in advance as the install requires a reboot.

The batch file contains:

net stop VSS
net stop Winmgmt
net stop SQLWriter
net start VSS
net start Winmgmt
net start SQLWriter

The Exchange Server also has this in the batch file, and has Winmgmt removed:

net stop MSExchangeIS
net start MSExchangeIS

The VSS Writer error on the Hosts and Guest VMs is:

Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {07434c75-70ef-4057-b4b1-3f1a7ffc486b}
State: [8] Failed
Last error: Non-retryable error

The error message on Backup and Replication is:

9/01/2018 7:23:19 p.m. :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing with failover) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

9/01/2018 7:26:04 p.m. :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)

9/01/2018 7:26:18 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)

[Mike Resseler]

Alex,

Did you look at this article: http://www.insidetechnologies.eu/en/blo ... ter-error/

This might be your problem
M/

[AlexLeadingEdge]

Did you look at this article: http://www.insidetechnologies.eu/en/blo ... ter-error/

This might be your problem

Hi Mike,

Interesting article, it seems to imply it is a Host issue, not a VM issue. It talks about a "Hold State" and in Figure 2 it says "Backing up...", which I have never seen on our Hosts.

I have restarted the Hyper-V Virtual Machine Management service on the J Host and will test.

[Mike Resseler]

Interested to learn what the result is Alex.

Yes, I agree it seems strange, but the fact that you have so many VMs (30%) immediately makes me think in the direction of the host.

M/

[AlexLeadingEdge]

Restarting the VSS writers on both the Host and the Guest 15 minutes before the backup has allowed the VSS writers to be picked up correctly. The problem is that it still errors, then at the retry it says the VSS writers are offline again

Code: Select all

23/01/2018 7:00:31 p.m. :: Queued for processing at 23/01/2018 7:00:31 p.m.  
23/01/2018 7:14:41 p.m. :: Preparing to create snapshot  
23/01/2018 7:15:53 p.m. :: Using guest interaction proxy BACKUPSVR (Same subnet)  
23/01/2018 7:15:56 p.m. :: Inventorying guest system  
23/01/2018 7:16:32 p.m. :: Subscribing to guest processing components  
23/01/2018 7:16:35 p.m. :: Creating snapshot Microsoft Software Shadow Copy provider 1.0 (mode: Veeam application-aware processing with failover)  
23/01/2018 7:19:51 p.m. :: Required backup infrastructure resources have been assigned  
23/01/2018 7:19:54 p.m. :: VM processing started at 23/01/2018 7:19:54 p.m.  
23/01/2018 7:19:54 p.m. :: VM size: 61.0 GB (52.8 GB used)  
23/01/2018 7:19:55 p.m. :: Truncating SQL server transaction logs  
[color=#FF0000]23/01/2018 7:19:59 p.m. :: Retrying snapshot creation attempt (Failed to create snapshot for vm b0bf5146-1e0c-4ba9-8e88-e606c2280ced)  [/color]
23/01/2018 7:19:59 p.m. :: Required backup infrastructure resources have been assigned  
23/01/2018 7:19:59 p.m. :: Required backup infrastructure resources have been assigned  
23/01/2018 7:25:28 p.m. :: Closing snapshot  
23/01/2018 7:47:17 p.m. :: VM processing started at 23/01/2018 7:47:17 p.m.  
23/01/2018 7:47:17 p.m. :: VM size: 61.0 GB (52.8 GB used)  
23/01/2018 7:47:24 p.m. :: Using guest interaction proxy BACKUPSVR (Same subnet)  
[color=#FF0000]23/01/2018 7:47:32 p.m. :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing with failover) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
23/01/2018 7:50:07 p.m. :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
23/01/2018 7:50:21 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)[/color]  
23/01/2018 7:50:26 p.m. :: Creating snapshot System Provider (mode: Crash consistent)  
23/01/2018 7:50:49 p.m. :: Using source proxy Hyper1.domain.local (onhost)  
23/01/2018 7:50:50 p.m. :: Getting list of guest file system local users  
23/01/2018 7:50:51 p.m. :: Saving B0BF5146-1E0C-4BA9-8E88-E606C2280CED.xml  
23/01/2018 7:50:51 p.m. :: Updating auxiliary data backup_doc  
23/01/2018 7:50:52 p.m. :: Updating auxiliary data writer_doc  
23/01/2018 7:50:54 p.m. :: Hard disk 1 (61.0 GB) 11.4 GB read at 445 MB/s [CBT] 
23/01/2018 7:51:37 p.m. :: Closing snapshot  
23/01/2018 7:51:41 p.m. :: Saving GuestMembers.xml  
23/01/2018 7:51:45 p.m. :: Finalizing  
23/01/2018 7:51:52 p.m. :: Swap file blocks skipped: 370.0 MB  
23/01/2018 7:51:52 p.m. :: Busy: Source 86% > Proxy 12% > Network 90% > Target 1%  
23/01/2018 7:51:52 p.m. :: Primary bottleneck: Network  
23/01/2018 7:51:52 p.m. :: Network traffic verification detected no corrupted blocks  
23/01/2018 7:51:52 p.m. :: Processing finished with warnings at 23/01/2018 7:51:52 p.m.

[Mike Resseler]

Alex,

So restarting the writers gives you a successful backup, but the second backup fails again. Did I understand that correctly?

What I propose is that you restart the VSS writers again, do a backup and then do a backup again (let it fail). Then you can go to the host, open command prompt as administrator and run vssadmin list writers. In that output you should be able to find out what writer is in error. I kind of fear that it is the WMI writer which would mean that you have a WMI corrupted database.

Let me know

[andersgustaf]

I have seen the same problem and this is (for us atleast) usualy the VM Guest that causes the error. The Guest OS often got multiple applications that registers with VSS.. but somehow it doesnt report any errors when listing all VSS inside the Guest OS.

My workaround is to just uncheck "Backup" in SCVMM and check it again... but with powershell

Code: Select all

# Veeam console and SCVMM Console needs to be installed to run the cmdlets
# Change SERVERNAME to Veeam Backupserver
Add-PSSnapin VeeamPSSnapin
Connect-VBRServer -Server SERVERNAME

foreach ($Job in $jobs){

    #Get latest job for every job that failed
    $FailedLastSession = $Job.FindLastSession() | Where-Object {$_.Result -eq "Failed"}


    #If failed jobs
    if($FailedLastSession){

            #Extact all vms from last failed job
            $vms = $FailedLastSession.GetTaskSessions() | Where-Object {$_.Status -eq "Failed"}
            Write-Host = "Failed VMs are:" $vms

                
                foreach ($vm in $vms){
                    # You need to disable backup and then enable backup on a failed VM in order for backup to work
                        Set-SCVirtualMachine $vm.Name -EnableBackup $false -ErrorAction Continue | Out-Null
                        Write-Host $VM.Name "- Deactivate Backup IC"
                        start-sleep -s 2
                        Set-SCVirtualMachine $vm.Name -EnableBackup $true -ErrorAction Continue | Out-Null
                        Write-Host $VM.Name "- Activate Backup IC"
                }
            
            #Start the job that will perform a backup on all failed VMs
            {
                Start-VBRJob -Job $FailedLastSession.JobName -RetryBackup -RunAsync -ErrorAction Continue
                Write-Host = "Restarting job" $FailedLastSession.JobName
            }
            Clear-Variable FailedLastSession
        }
}

Best regards,
Anders

[AlexLeadingEdge]

Alex,

So restarting the writers gives you a successful backup, but the second backup fails again. Did I understand that correctly?

The backup is successful, but I'm getting the same error as before. Going by the log above (if I am reading it correctly) it says the VSS writers are fine, but then says it failed to use them successfully, and when it retries it says the VSS writers are no longer available?

[Mike Resseler]

Go for the vssadmin list writers on host and guest so we can see which one fails.

[AlexLeadingEdge]

Go for the vssadmin list writers on host and guest so we can see which one fails.

No writers display on the Guest VM:

Code: Select all

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

[b]C:\Windows\system32>vssadmin list writers[/b]
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

C:\Windows\system32>

Microsoft Hyper-V VSS Writer failed on Host:

Code: Select all

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {e652c672-a61b-4d60-8664-57889edad8ff}
   State: [1] Stable
   Last error: No error

[b]Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {8db22e25-984f-4378-8a01-7a962046da31}
   State: [8] Failed
   Last error: Non-retryable error[/b]

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {0c375489-2460-496c-9d6b-583757881b1b}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {f9c51e6b-a5a8-43e3-9344-b2980d5b65ff}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {f5c93f9f-4524-407b-b392-42e91b948d71}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {93ad0fa8-a025-4e4a-9ee4-feaa7a6cd1d8}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {979e655f-bd30-4c8c-a7ee-e89b850c4ed2}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {5ee246dd-f950-4161-ac01-d5b044bc7aab}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {773b01f8-f000-44f1-9a0d-375d1f63b8c1}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {fe6ff903-5dfc-42ff-942f-4c1e6a3190b7}
   State: [1] Stable
   Last error: No error

[Mike Resseler]

So my guess was correct. The problem is with the host. Best you can do now is search in the event viewer. Search in Windows Logs, Application and System.  Look for Disk, VSS, and VolSnap entries around the time that you take a backup. And since it is about Hyper-V, also look at Applications and Services Logs -> Microsoft -> Windows -> Hyper-V-.... for specific errors around the time of backup.

[Bennon Maina]

I remember spending about 5 months with both Microsoft and Veeam Support with backup failures to due 'prepare snapshot' failures to no avail.

The issue (as described in this article https://blog.workinghardinit.work/2015/ ... l-machine/) turned out to be orphaned snapshot files (.AVHDX) in the VM disk storage location (a two node 2K12 R2 Hyper-V Cluster). I was able to delete some but others had been locked by 'System'. I stopped the Hyper-V VMM Service and Started after a couple of minutes (note not restarting). This will start a merge process for the orphaned snapshot files (files locked by 'System').

Once merged (Check Hyper-V Manager) for the progress. (You may need to stop and start the Hyper-V VMM Service severally until the merge is complete to 100%).
If they are still a few AVHDX files still left in the storage location, you can manually delete them. After re-running the backup, it completed successfully.

It might also be worth noting backing up WK28 R2 Exchange 2010 DAG ( 1 HT, 2 DB [Active - Passive]) with AAIP enabled) all running from one job, fails on occasion but do manage to rerun the job in case it fails (and do occasionally check for these orphaned snapshot files). I Am in the process of upgrading these VMs to 2012 - 2016.

Hope this helps your situation .

[AlexLeadingEdge]

Bennon, I think you may have just solved the mystery! The three VMs that we keep having issues with each have a VM_NAME-AutoRecovery.avhdx file in the VM folder dated from 2016!

[AlexLeadingEdge]

Bennon, I think you may have just solved the mystery! The three VMs that we keep having issues with each have a VM_NAME-AutoRecovery.avhdx file in the VM folder dated from 2016!

I have deleted these three orphaned files and will see how the backups go tonight.

[Mike Resseler]

Cool. I am hoping that it is solved. One thing I am interested to know about. (Both Alex and Bennon), were those avhdx files not visible in the hyper-v manager? And what was the location of those? Same directory as the VHDX files?

[Bennon Maina]

Hi Alex, it would be interesting to know if it works.
Mike, no they are not visible in Hyper-V Manager. Unless of course you manually create a snapshot/checkpoint. The snapshot/checkpoint file will have .AVHDX file extension in the VM disk storage folder/location.

Another way around this, although wouldn't recommend it on backups jobs for VMs with high I/O that are failing to create at preparing a snapshot/checkpoint; create one manually from Hyper-V Manager or SCVMM.
Once completed, delete the snapshot. All AVHDX files associated with the VMs will be merged but do check the storage location in case any are left. You should be able to delete them manually and run a backup job.

I haven't had a backup failure since.

[Mike Resseler]

Yeah, I don't think it is a good workaround to be honest but if it works now, great. Doesn't mean we should look at it for a future view. By the way, do you have a case number?

[AlexLeadingEdge]

Great news! Backups are working perfectly now!

Cool. I am hoping that it is solved. One thing I am interested to know about. (Both Alex and Bennon), were those avhdx files not visible in the hyper-v manager?

No, the Hyper-V Manager doesn't show them

And what was the location of those? Same directory as the VHDX files?

Yes, in the same directory as the VHDX files. I never would have suspected looking in there! See image. I had to blank out details for security reasons, this image will delete itself in 31 days time.