Error: The system cannot find the path specified. Failed to

Hyper-V specific discussions

Error: The system cannot find the path specified. Failed to

Veeam Logoby skumflum » Thu Mar 02, 2017 6:45 pm

case: 02087617

We have an issue with backup. Most of our jobs fails with:
Failed to open file [\\?\GLOBALROOT\Device\CSV:
e.x:
Code: Select all
3/2/2017 3:33:42 PM :: Processing DLEBJ01 Error: The system cannot find the path specified.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] in readonly mode.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] for reading.
What we have tried:
Create new job with affected VM
Upgrade from 9.0 to 9.5 update 1
Restart VMM server
Restart VMMS, WMI, VEEAM, VMM Agent service on a host
Refresh Cluster from VMM
Live migrated VM to another host (sometimes this help but not always)

What happen before the issue began?
We removed some empty CSV from the Hyper-V Cluster.
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Thu Mar 02, 2017 7:17 pm

It seems to work fine when VMs on a host are placed on the CSV that are currently owned by the same hosts
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby mpark » Thu Mar 02, 2017 7:29 pm

Can you confirm that you have provided the correct credentials for SCVMM server under Backup Infrastructure?
Also, I'd check and test each hosts for credential verification and see if you encounter any errors. I recall having read-only error due to credential issue on the Veeam
mpark
Influencer
 
Posts: 23
Liked: 3 times
Joined: Wed Jan 20, 2016 4:43 am
Full Name: Min Park

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Thu Mar 02, 2017 7:45 pm

Credentials are ok. I can browse SCVMM from Veeam, rescan and so on
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Thu Mar 02, 2017 8:43 pm

I can now confirm the pattern.

The backup works only on VM's that are located on a host who is also the CSV owner of the VM's current storage location.
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby Mike Resseler » Fri Mar 03, 2017 7:41 am

Hi,

Could you please create a support call (and post the support case ID and follow up here)? More details are needed to investigate this and our engineers will be able to see what is happening in the logs.

It might be something really simple but it could be obviously something bad (let's hope not :-))

Keep us informed
Mike
Mike Resseler
Veeam Software
 
Posts: 3346
Liked: 380 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Sat Mar 04, 2017 8:38 am

Hi,
I have a case id: 02087617

There is something wrong with VSS. The issue manifest itself when doing shadow copy of a CSV that isn’t owned by the cluster node (host).
I can replicate the error by doing shadow copy manually:

diskshadow
set verbose on
set context persistent nowriters
add volume C:\ClusterStorage\VolumeName
create
expose %vss_shadow_1% x:

This will fail unless it’s done against a CSV that is owned by the node.

Any thoughts?
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby Mike Resseler » Sat Mar 04, 2017 1:39 pm

Hi.

Sorry missed that from your first post you already had a case ID. What OS is your Hyper-V host? 2012 or R2? From what I remember this used to be an issue but it was fixed with some hotfixes.

Did you check our KB on the needed hotfixes (note: not all of them come with windows update): https://www.veeam.com/kb1838

Mike
Mike Resseler
Veeam Software
 
Posts: 3346
Liked: 380 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Sat Mar 04, 2017 8:08 pm

Thank you :)

I have read release notes on all the recommended hotfixes but I do not think that any of them apply to my problem. The closest one about CSV is...
"Cluster service stops during the VSS backup in a Windows Server 2012 R2 or Windows Server 2012-based Hyper-V cluster" But that's not what going on.

I will try to reboot all hosts one-by-one and see how this goes.
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Sat Mar 04, 2017 10:44 pm

ok.. we tried to reboot a host - problem not resolved. Then we rebooted ALL the host and now the problem is gone!

Great, but we don't know what happened :(
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby Mike Resseler » Mon Mar 06, 2017 6:14 am

And you are sure that the backup now works on VM's running on a host are not the CSV owner?

Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?

(To be honest, I am confused also :-()
Mike Resseler
Veeam Software
 
Posts: 3346
Liked: 380 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Mon Mar 06, 2017 1:46 pm

Mike Resseler wrote:And you are sure that the backup now works on VM's running on a host are not the CSV owner?

Yes I have verified this 100%



Mike Resseler wrote:Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?

No updates installed.... well since august 2016 :(... hmm maybe I should suggest to patch more often

Mike Resseler wrote:(To be honest, I am confused also :-()
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby Mike Resseler » Tue Mar 07, 2017 7:29 am

Well,

I know very well it is a pain, but yes, patch, patch, patch but that is not enough. You need to install hotfixes also (as said by that KB I mentioned) because they don't come with Windows Update. And I know that normally the best practice is to install only hotfixes that are "applicable" to your problem but in the case of Hyper-V and CSV's I tend to install them as a part of my baseline setup.

Please keep us informed if the issues would arise again.
Mike Resseler
Veeam Software
 
Posts: 3346
Liked: 380 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby skumflum » Tue Mar 07, 2017 9:23 am

Thank you Mike :)

Okay, so you would recommend installing all the hotfixes recommended even them that is not obvious applicable? This goes against my "childhood" teaching :)

The KB you mentioned has not been updated since 2015-12-15.
Microsoft also maintain some lists (Hyper-V and Cluster)

https://support.microsoft.com/en-us/kb/3135020
https://support.microsoft.com/en-us/kb/2920151

Would you recommend installing all the listed patches/hotfixes? Some of them are rather old.
skumflum
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Mon Jun 13, 2016 6:51 am
Full Name: Søren Emig

Re: Error: The system cannot find the path specified. Failed

Veeam Logoby Mike Resseler » Tue Mar 07, 2017 9:29 am

I realize the KB is rather old but as far as I know most of those are still applicable. However, if your support case is still open, I would ask the support engineers about this. They are much more familiar with it then I am and they work daily on it so their advice is most certainly better than mine on this one ;-)
Mike Resseler
Veeam Software
 
Posts: 3346
Liked: 380 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 9 guests