Host-based backup of Microsoft Hyper-V VMs.
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Error: The system cannot find the path specified. Failed to

Post by skumflum »

case: 02087617

We have an issue with backup. Most of our jobs fails with:
Failed to open file [\\?\GLOBALROOT\Device\CSV:
e.x:

Code: Select all

3/2/2017 3:33:42 PM :: Processing DLEBJ01 Error: The system cannot find the path specified.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] in readonly mode.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] for reading.
What we have tried:
Create new job with affected VM
Upgrade from 9.0 to 9.5 update 1
Restart VMM server
Restart VMMS, WMI, VEEAM, VMM Agent service on a host
Refresh Cluster from VMM
Live migrated VM to another host (sometimes this help but not always)

What happen before the issue began?
We removed some empty CSV from the Hyper-V Cluster.
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

It seems to work fine when VMs on a host are placed on the CSV that are currently owned by the same hosts
mpark
Enthusiast
Posts: 32
Liked: 4 times
Joined: Jan 20, 2016 4:43 am
Full Name: Min Park
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by mpark »

Can you confirm that you have provided the correct credentials for SCVMM server under Backup Infrastructure?
Also, I'd check and test each hosts for credential verification and see if you encounter any errors. I recall having read-only error due to credential issue on the Veeam
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

Credentials are ok. I can browse SCVMM from Veeam, rescan and so on
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

I can now confirm the pattern.

The backup works only on VM's that are located on a host who is also the CSV owner of the VM's current storage location.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler »

Hi,

Could you please create a support call (and post the support case ID and follow up here)? More details are needed to investigate this and our engineers will be able to see what is happening in the logs.

It might be something really simple but it could be obviously something bad (let's hope not :-))

Keep us informed
Mike
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

Hi,
I have a case id: 02087617

There is something wrong with VSS. The issue manifest itself when doing shadow copy of a CSV that isn’t owned by the cluster node (host).
I can replicate the error by doing shadow copy manually:

diskshadow
set verbose on
set context persistent nowriters
add volume C:\ClusterStorage\VolumeName
create
expose %vss_shadow_1% x:

This will fail unless it’s done against a CSV that is owned by the node.

Any thoughts?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler »

Hi.

Sorry missed that from your first post you already had a case ID. What OS is your Hyper-V host? 2012 or R2? From what I remember this used to be an issue but it was fixed with some hotfixes.

Did you check our KB on the needed hotfixes (note: not all of them come with windows update): https://www.veeam.com/kb1838

Mike
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

Thank you :)

I have read release notes on all the recommended hotfixes but I do not think that any of them apply to my problem. The closest one about CSV is...
"Cluster service stops during the VSS backup in a Windows Server 2012 R2 or Windows Server 2012-based Hyper-V cluster" But that's not what going on.

I will try to reboot all hosts one-by-one and see how this goes.
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

ok.. we tried to reboot a host - problem not resolved. Then we rebooted ALL the host and now the problem is gone!

Great, but we don't know what happened :(
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler »

And you are sure that the backup now works on VM's running on a host are not the CSV owner?

Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?

(To be honest, I am confused also :-()
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

Mike Resseler wrote:And you are sure that the backup now works on VM's running on a host are not the CSV owner?
Yes I have verified this 100%


Mike Resseler wrote:Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?
No updates installed.... well since august 2016 :(... hmm maybe I should suggest to patch more often
Mike Resseler wrote:(To be honest, I am confused also :-()
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler »

Well,

I know very well it is a pain, but yes, patch, patch, patch but that is not enough. You need to install hotfixes also (as said by that KB I mentioned) because they don't come with Windows Update. And I know that normally the best practice is to install only hotfixes that are "applicable" to your problem but in the case of Hyper-V and CSV's I tend to install them as a part of my baseline setup.

Please keep us informed if the issues would arise again.
skumflum
Service Provider
Posts: 33
Liked: 1 time
Joined: Jun 13, 2016 6:51 am
Full Name: Søren Emig
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by skumflum »

Thank you Mike :)

Okay, so you would recommend installing all the hotfixes recommended even them that is not obvious applicable? This goes against my "childhood" teaching :)

The KB you mentioned has not been updated since 2015-12-15.
Microsoft also maintain some lists (Hyper-V and Cluster)

https://support.microsoft.com/en-us/kb/3135020
https://support.microsoft.com/en-us/kb/2920151

Would you recommend installing all the listed patches/hotfixes? Some of them are rather old.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed

Post by Mike Resseler »

I realize the KB is rather old but as far as I know most of those are still applicable. However, if your support case is still open, I would ask the support engineers about this. They are much more familiar with it then I am and they work daily on it so their advice is most certainly better than mine on this one ;-)
TheKaiser
Lurker
Posts: 2
Liked: 2 times
Joined: Nov 28, 2018 12:56 pm
Full Name: Achim Kaiser
Contact:

[MERGED] The system cannot find the file specified

Post by TheKaiser »

We have an issue with backup after Update 4 for B&R 9.5

Of the 57 VMs always the same 10 VMs fail

Environment: Windows Server 2016 with Veeam Backup & Replication 9.5.4.2615
Windows Server 2012 R2 Hyper-V Host

Code: Select all

The backup fails with: Failed to open file 
[\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy: 

Code: Select all 30/1/2019 3:33:42 PM :: Processing DEV04 Error: The system cannot find the path specified. 
Failed to open file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy12\Virtual Machines\Snapshots\6C8E1C7E-D871-4C02-AF36-143CA3F5C6E7.xml] in readonly mode. 
Failed to open file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy12\Virtual Machines\Snapshots\6C8E1C7E-D871-4C02-AF36-143CA3F5C6E7.xml] for reading. 
what we have done so far:

-Create new job with affected VM
-Restart the VMs
-Restart the Hyper-V 2012 Host
-Migrated the affected VM to another host

-Check the with vssadmin list shadows |more

Code: Select all

PS C:\Windows\system32> vssadmin list shadows |more
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

No items found that satisfy the query.
Veeam Support - Case # 03388708 - Error: The system cannot find the file specified

Thanks
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by Dima P. »

Hello and welcome Achim!

I've merged your post to the existing discussion. This problem is likely caused by issues with VSS components, so, kindly, keep working with our support team in order to find the root cause. Cheers!
nikods
Novice
Posts: 9
Liked: 2 times
Joined: May 25, 2015 11:46 am
Full Name: Dragan
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by nikods »

I have the same problem after update to 9.5 update 4. I have three servers in windows server 2012 r2 failover cluster, all of the servers backup jobs are failing, the same day I have updated to update 4 one time all of the jobs have completed and tomorrow they started to fail with error:
Processing SERVERNAME Error: The system cannot find the file specified.
Failed to open file [\\?\GLOBALROOT\Device\CSV.....xml] in readonly mode.
Failed to open file [\\?\GLOBALROOT\Device\CSV.....xml] in for reading.
I try to put VM on same host that holds CSV but same error. I can rescan hosts and check backup files but cant do backup jobs. I try to make new job, same error. I have restarted, reconnected iscsi to csv storage and no change.
Please help :-(
TheKaiser
Lurker
Posts: 2
Liked: 2 times
Joined: Nov 28, 2018 12:56 pm
Full Name: Achim Kaiser
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by TheKaiser » 2 people like this post

Hello at all.

Our problem was solved together with Veeam Support!

Here the solution of the Veeam Support:

Here's what I want you to do:
- open VM settings (e.g. busoDEV01) in Hyper-V Manager
- select "Checkpoint File Location" -> "Browse...". -> select the path "F:\Virtual Machines\Snapshots".
(currently the wrong path "F:\virtual machines" is selected there)

So we adjusted the path "Checkpoint File Location".

Then restart the Veeam Backup job.

And the problems are solved

Thanks
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by Mike Resseler » 1 person likes this post

Hi TheKaiser,

Thanks for letting us know. Did you change that path manually or was it this by default?
nikods
Novice
Posts: 9
Liked: 2 times
Joined: May 25, 2015 11:46 am
Full Name: Dragan
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by nikods » 1 person likes this post

The manual setting of the paths for Snapshots has resolved the problem form me also!

Thank you All!

Niko
JSi
Influencer
Posts: 17
Liked: 2 times
Joined: Oct 23, 2013 9:02 am
Full Name: Jan Simko
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by JSi »

Hi to all,

we have exactly the same problem (Win2012R2 Hyper-V cluster) after updating to U4, unfortunately our support contract expired.
Change of checkpoint path in Failover Cluster Manager unfortunately doesn`t help. Is there any other required condition (backup server reboot, hyper-v node reboot,...)?

Jan
JSi
Influencer
Posts: 17
Liked: 2 times
Joined: Oct 23, 2013 9:02 am
Full Name: Jan Simko
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by JSi »

Probably it takes some time. It looks ok now. :-)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by Mike Resseler »

Hi Jan,

Just to be clear: You did the change of the checkpoint path and it didn't work to start with? But now it works? I do believe you need to restart the VM to make the new checkpoint settings applicable. (But not so sure anymore)
JSi
Influencer
Posts: 17
Liked: 2 times
Joined: Oct 23, 2013 9:02 am
Full Name: Jan Simko
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by JSi » 1 person likes this post

Hi Mike,

many of impacted VMs were in offline state. It doesn`t help at first retry, but without any other change, next retry was ok. Thank you for reply.

Anyway Veeam thanx for your work on new update and your general attitude.

J.
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

[MERGED] Cannot find the file specified

Post by ejenner »

I've got a couple of Hyper-V VMs which won't backup.

Code: Select all

The following error: 04/02/2019 04:05:40 :: Processing SERVERNAME Error: The system cannot find the file specified.
Failed to open file [\\?\Volume{4561c16c-2825-11e9-80c5-2c768a5d7fc0}\SERVERNAME\Snapshots\424D66E2-309E-4ECC-8589-1C691E1E6B5C.xml] in readonly mode.
Failed to open file [\\?\Volume{4561c16c-2825-11e9-80c5-2c768a5d7fc0}\SERVERNAME\Snapshots\424D66E2-309E-4ECC-8589-1C691E1E6B5C.xml] for reading.
The thing both these servers have in common is that they're DMZ servers. The VMs are running from a HP 3PAR and I have Direct SAN backup configured.

I'm guessing this is a blocked ports issue due to the DMZ configuration but just putting it out there for comments / suggestions. Just in case someone has seen it before.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by PTide »

EJ,

Although your config might be a little different (no CSV apparently), this thread seems to be approppriate for your post. Kindly open your own support case and let us know case ID.

Thanks!
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by ejenner »

yes, I did see this. Going to have to think about this for a while as although the error message is the same... the symptoms are different.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by Mike Resseler »

Hey EJ,

Most likely it isn't this problem, but I heard yesterday from a friend that his checkpoints got into read-only mode after an AV signature update. Are those checkpoint locations excluded?
roger_ramjet
Novice
Posts: 4
Liked: 2 times
Joined: Jul 25, 2016 4:00 am
Contact:

Re: Error: The system cannot find the path specified. Failed to

Post by roger_ramjet » 1 person likes this post

TheKaiser wrote: Jan 31, 2019 9:44 am Hello at all.

Our problem was solved together with Veeam Support!

Here the solution of the Veeam Support:

Here's what I want you to do:
- open VM settings (e.g. busoDEV01) in Hyper-V Manager
- select "Checkpoint File Location" -> "Browse...". -> select the path "F:\Virtual Machines\Snapshots".
(currently the wrong path "F:\virtual machines" is selected there)

So we adjusted the path "Checkpoint File Location".

Then restart the Veeam Backup job.

I cannot believe this worked for me!! Mine was slightly different where there was no "Snapshots" folder in the save location. My issue was there were no capital letters where there should have been. Looks like you have to be very specific in regards to capital letters in the whole file name location with Veeam. :?:

EDIT: Also if you're using Hyper-V with Failover Clustering, you have to open "Settings" from the VM in Failover Cluster Manager to change the snapshot path.
Post Reply

Who is online

Users browsing this forum: chad.shutts, Semrush [Bot] and 26 guests