Planned Failover fails on Win2016 with Win2012R2 backend

[ptoro]

We have recently made two big changes

1 Updated our front end hyperv hosts to Windows 2016
2 Installed Update 3

We have two hyperv clusters which each use their own scale out file server (SOFS) for SMB3 shares.

for simplicity let's call them
hpvclus1 -> hyperv cluster in site 1
hpvclus2 -> hyperv cluster in site 2
sofsclus1 -> scale out file server cluster in site 1
sofsclus2 -> scale out file server cluster in site 1
hpvclus1 uses sofsclus1
hpvclus2 uses sofsclus2

Things that still work:
1. backups jobs complete successfully and can restore
2. backup copy jobs complete successfully
3. replication jobs complete successfully and update the replicas (replica job uses the backup copy as its source)
4. can issue a failover and VM starts as expected

Things that used to work:
1. Planned failovers

Now when I do a planned failover, the job fails with access denied to the actual source SMB share
"Failed to process replication task Error: Access is denied. Failed to open file [\\##redacted##.vhdx] in readonly mode. Agent failed to process method {VHDX.GetDiskInformation}."

I have made sure to add even more SMB delegation settings (which I never had before when it was working) just to prove that I could from a hpvclus2 server connect to sofsclus1 SMB share and create a VM with a vhdx without errors (I could).


Anyone else seeing this on planned failovers?

[Mike Resseler]

Hi ptoro,

I probably misunderstand it but are the source hosts 2016 and the target hosts 2012 R2?

[ptoro]

BOTH source and target hosts are 2016. It's the backend storage (SOFS with SMB shares) that still is 2012r2

[Mike Resseler]

OK. My first guess with this error message would lead me to think that something is locking the VHDX files. But what is difficult to say from that message. With a planned failover, we do replicate the latest changes to the replica before doing the actual failover. Maybe there is something there that is locking it. I would suggest creating a support call (post the support case ID here and the outcome after investigation with the engineers) and let them search through the logs why there is an access denied. The logs should have more information on it.

Cheers
Mike

[ptoro]

quick update:

I was working with support while I started this post. Our Backup proxies also reside on the Scale Out Files Servers (SMB backend for HyperV servers). It would appear that the backup proxy used has to be the owner of the CSV in question for it to work. We also tried to do a backup with a backup proxy that was not the owner and it also failed (my backup jobs are grouped be CSV and were set to use the CSV owner as the backup proxy so I never saw that issue).

Disabling the backup proxies forces an onhost backup or "final sync" during the planned failover and can successfully finish the job.

Will update on what support comes up with for as to why this is happening now (backend storage has not changed on my end).