Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

[hbilke]

Hi fellas,

what we have:
"migrated" 2-Node cluster from Windows Server 2016 Std. rto Windows Server 2019 DC (rolling cluster upgrade)
Fujitsu Eternus hosting CSV
no errors in cluster check
weird: two identical quora (share on hardware)

B&R 9.5 first installed 3a (2016, working)
B&R 9.5 Upd 4 : first working - problems began with one of the instances, then all of the backups stalling
--> enough space on repo
--> snapshots during backup are created, gone within GUI, get-vmsnapshot won't find any snapshot BUT
*.CRT, *.MRT (as expected I Think) and one (or more) *AVHDX are left during the full backup

all following incrementals fail due to no rights/not accessible files
merge won't work (neither during shutdown nor restarting the VM)
--> the VM-machine ID of the VM to which the files belong is listed within the error details is blocking access
some of the files IDs match when checked via the GUI
some of the VMs won't restart without reestablishing the connection between the diskfiles (VHDX, AVHDX) AND ignoring the ID (as they still won't match)
The avhdx files have a weird SID on their respective security tab.

One tried fix was to move the instances from one host to the other: differencing disks gone, first backup running but not the incrementals afterwards
No change after updating to 4a

All jobs are now disabled (not deleted yet)

Any ideas / hints welcome

Traveling on site now

[Mike Resseler]

Hey hbilke,

First: a CRT and MRT file have to be there. That is the change block tracking implementation of Microsoft. The AVHDX files that are left is another case. They should disappear (unless it is a shared disk).

Unfortunately it will be impossible to figure out what is going wrong without the logs, so since you created a support call, you should be able to let them review everything. It looks like an issue that used to be a problem with older versions of Windows but certainly not with server 2019, so I kind of hope that the issue from Microsoft has not returned...

[hbilke]

Hi Mike,

yes, the CRT and MRT have to stay. Up to now (test backups are running since yesterday) the only way to remedy seems:

1. Export the VM (not funny with 1 to 3 TB data per VM)
2. NO Import - building a new VM-instance
3. Start, check and configure networking etc.
4. Build a NEW backup job

Sounds familiar with the older MS problem?

[hbilke]

This morning I started two of the jobs that worked yesterday on solo today in the morning - together
AVHDX stays!!!
a third job that ran yesterday and today solo was okay - after the two that errored out: no snapshots could be created.

Mike: What was the old error "complex" you mentioned and how to check it?

Harry

[hbilke]

Looks like a "Login as service" problem.

[Mike Resseler]

The older problem was indeed that there was some sort of access denied to the snapshots causing it to stay there. Some issue with the referencepoint value from Microsoft itself.

Is the investigation going well?

[hbilke]

All resolved now. Some of the instances had to be joined "manuallly" some did it during shutdown/restart.

The customer had fiddled with the "Login as a service rights".

Case closed

[Mike Resseler]

Good to know. I do wonder why they started to play with Login as a service rights as it is used by many applications so I am assuming they might have had other issues also. Also kind of wondering how you figured that one out, nice researching