Host-based backup of Microsoft Hyper-V VMs.
Post Reply
hbilke
Expert
Posts: 113
Liked: 4 times
Joined: Jan 14, 2017 9:05 pm
Contact:

Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by hbilke »

Hi fellas,

what we have:
"migrated" 2-Node cluster from Windows Server 2016 Std. rto Windows Server 2019 DC (rolling cluster upgrade)
Fujitsu Eternus hosting CSV
no errors in cluster check
weird: two identical quora (share on hardware)

B&R 9.5 first installed 3a (2016, working)
B&R 9.5 Upd 4 : first working - problems began with one of the instances, then all of the backups stalling
--> enough space on repo
--> snapshots during backup are created, gone within GUI, get-vmsnapshot won't find any snapshot BUT
*.CRT, *.MRT (as expected I Think) and one (or more) *AVHDX are left during the full backup

all following incrementals fail due to no rights/not accessible files
merge won't work (neither during shutdown nor restarting the VM)
--> the VM-machine ID of the VM to which the files belong is listed within the error details is blocking access
some of the files IDs match when checked via the GUI
some of the VMs won't restart without reestablishing the connection between the diskfiles (VHDX, AVHDX) AND ignoring the ID (as they still won't match)
The avhdx files have a weird SID on their respective security tab.

One tried fix was to move the instances from one host to the other: differencing disks gone, first backup running but not the incrementals afterwards
No change after updating to 4a

All jobs are now disabled (not deleted yet)

Any ideas / hints welcome

Traveling on site now
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by Mike Resseler »

Hey hbilke,

First: a CRT and MRT file have to be there. That is the change block tracking implementation of Microsoft. The AVHDX files that are left is another case. They should disappear (unless it is a shared disk).

Unfortunately it will be impossible to figure out what is going wrong without the logs, so since you created a support call, you should be able to let them review everything. It looks like an issue that used to be a problem with older versions of Windows but certainly not with server 2019, so I kind of hope that the issue from Microsoft has not returned...
hbilke
Expert
Posts: 113
Liked: 4 times
Joined: Jan 14, 2017 9:05 pm
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by hbilke »

Hi Mike,

yes, the CRT and MRT have to stay. Up to now (test backups are running since yesterday) the only way to remedy seems:

1. Export the VM (not funny with 1 to 3 TB data per VM)
2. NO Import - building a new VM-instance
3. Start, check and configure networking etc.
4. Build a NEW backup job

Sounds familiar with the older MS problem?
hbilke
Expert
Posts: 113
Liked: 4 times
Joined: Jan 14, 2017 9:05 pm
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by hbilke »

This morning I started two of the jobs that worked yesterday on solo today in the morning - together
AVHDX stays!!!
a third job that ran yesterday and today solo was okay - after the two that errored out: no snapshots could be created.

Mike: What was the old error "complex" you mentioned and how to check it?

Harry
hbilke
Expert
Posts: 113
Liked: 4 times
Joined: Jan 14, 2017 9:05 pm
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by hbilke »

Looks like a "Login as service" problem.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by Mike Resseler »

The older problem was indeed that there was some sort of access denied to the snapshots causing it to stay there. Some issue with the referencepoint value from Microsoft itself.

Is the investigation going well?
hbilke
Expert
Posts: 113
Liked: 4 times
Joined: Jan 14, 2017 9:05 pm
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by hbilke »

All resolved now. Some of the instances had to be joined "manuallly" some did it during shutdown/restart.

The customer had fiddled with the "Login as a service rights".

Case closed
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Hyper-V Cluster not "releasing" snapshots (Case ID 03542343)

Post by Mike Resseler »

Good to know. I do wonder why they started to play with Login as a service rights as it is used by many applications so I am assuming they might have had other issues also. Also kind of wondering how you figured that one out, nice researching ;-)
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 26 guests