Discussions specific to the Microsoft Hyper-V hypervisor
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Error: The system cannot find the path specified. Failed to

Post by skumflum » Mar 02, 2017 6:45 pm

case: 02087617

We have an issue with backup. Most of our jobs fails with:
Failed to open file [\\?\GLOBALROOT\Device\CSV:
e.x:

Code: Select all

3/2/2017 3:33:42 PM :: Processing DLEBJ01 Error: The system cannot find the path specified.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] in readonly mode.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] for reading.
What we have tried:
Create new job with affected VM
Upgrade from 9.0 to 9.5 update 1
Restart VMM server
Restart VMMS, WMI, VEEAM, VMM Agent service on a host
Refresh Cluster from VMM
Live migrated VM to another host (sometimes this help but not always)

What happen before the issue began?
We removed some empty CSV from the Hyper-V Cluster.

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 02, 2017 7:17 pm

It seems to work fine when VMs on a host are placed on the CSV that are currently owned by the same hosts

mpark
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jan 20, 2016 4:43 am
Full Name: Min Park
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by mpark » Mar 02, 2017 7:29 pm

Can you confirm that you have provided the correct credentials for SCVMM server under Backup Infrastructure?
Also, I'd check and test each hosts for credential verification and see if you encounter any errors. I recall having read-only error due to credential issue on the Veeam

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 02, 2017 7:45 pm

Credentials are ok. I can browse SCVMM from Veeam, rescan and so on

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 02, 2017 8:43 pm

I can now confirm the pattern.

The backup works only on VM's that are located on a host who is also the CSV owner of the VM's current storage location.

Mike Resseler
Product Manager
Posts: 5272
Liked: 556 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler » Mar 03, 2017 7:41 am

Hi,

Could you please create a support call (and post the support case ID and follow up here)? More details are needed to investigate this and our engineers will be able to see what is happening in the logs.

It might be something really simple but it could be obviously something bad (let's hope not :-))

Keep us informed
Mike

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 04, 2017 8:38 am

Hi,
I have a case id: 02087617

There is something wrong with VSS. The issue manifest itself when doing shadow copy of a CSV that isn’t owned by the cluster node (host).
I can replicate the error by doing shadow copy manually:

diskshadow
set verbose on
set context persistent nowriters
add volume C:\ClusterStorage\VolumeName
create
expose %vss_shadow_1% x:

This will fail unless it’s done against a CSV that is owned by the node.

Any thoughts?

Mike Resseler
Product Manager
Posts: 5272
Liked: 556 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler » Mar 04, 2017 1:39 pm

Hi.

Sorry missed that from your first post you already had a case ID. What OS is your Hyper-V host? 2012 or R2? From what I remember this used to be an issue but it was fixed with some hotfixes.

Did you check our KB on the needed hotfixes (note: not all of them come with windows update): https://www.veeam.com/kb1838

Mike

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 04, 2017 8:08 pm

Thank you :)

I have read release notes on all the recommended hotfixes but I do not think that any of them apply to my problem. The closest one about CSV is...
"Cluster service stops during the VSS backup in a Windows Server 2012 R2 or Windows Server 2012-based Hyper-V cluster" But that's not what going on.

I will try to reboot all hosts one-by-one and see how this goes.

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 04, 2017 10:44 pm

ok.. we tried to reboot a host - problem not resolved. Then we rebooted ALL the host and now the problem is gone!

Great, but we don't know what happened :(

Mike Resseler
Product Manager
Posts: 5272
Liked: 556 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler » Mar 06, 2017 6:14 am

And you are sure that the backup now works on VM's running on a host are not the CSV owner?

Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?

(To be honest, I am confused also :-()

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 06, 2017 1:46 pm

Mike Resseler wrote:And you are sure that the backup now works on VM's running on a host are not the CSV owner?
Yes I have verified this 100%


Mike Resseler wrote:Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?
No updates installed.... well since august 2016 :(... hmm maybe I should suggest to patch more often
Mike Resseler wrote:(To be honest, I am confused also :-()

Mike Resseler
Product Manager
Posts: 5272
Liked: 556 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler » Mar 07, 2017 7:29 am

Well,

I know very well it is a pain, but yes, patch, patch, patch but that is not enough. You need to install hotfixes also (as said by that KB I mentioned) because they don't come with Windows Update. And I know that normally the best practice is to install only hotfixes that are "applicable" to your problem but in the case of Hyper-V and CSV's I tend to install them as a part of my baseline setup.

Please keep us informed if the issues would arise again.

skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum » Mar 07, 2017 9:23 am

Thank you Mike :)

Okay, so you would recommend installing all the hotfixes recommended even them that is not obvious applicable? This goes against my "childhood" teaching :)

The KB you mentioned has not been updated since 2015-12-15.
Microsoft also maintain some lists (Hyper-V and Cluster)

https://support.microsoft.com/en-us/kb/3135020
https://support.microsoft.com/en-us/kb/2920151

Would you recommend installing all the listed patches/hotfixes? Some of them are rather old.

Mike Resseler
Product Manager
Posts: 5272
Liked: 556 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler » Mar 07, 2017 9:29 am

I realize the KB is rather old but as far as I know most of those are still applicable. However, if your support case is still open, I would ask the support engineers about this. They are much more familiar with it then I am and they work daily on it so their advice is most certainly better than mine on this one ;-)

Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests