Error: The system cannot find the path specified. Failed to

[skumflum]

case: 02087617

We have an issue with backup. Most of our jobs fails with:
Failed to open file [\\?\GLOBALROOT\Device\CSV:
e.x:

Code: Select all

3/2/2017 3:33:42 PM :: Processing DLEBJ01 Error: The system cannot find the path specified.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] in readonly mode.
Failed to open file [\\?\GLOBALROOT\Device\CSV{bd58ab47-5258-46b3-a7ea-006b2df6fcff}\DLEBJ01\Virtual Machines\65795C4A-9E10-4102-B1E8-14825D82BE16.xml] for reading.

What we have tried:
Create new job with affected VM
Upgrade from 9.0 to 9.5 update 1
Restart VMM server
Restart VMMS, WMI, VEEAM, VMM Agent service on a host
Refresh Cluster from VMM
Live migrated VM to another host (sometimes this help but not always)

What happen before the issue began?
We removed some empty CSV from the Hyper-V Cluster.

[skumflum]

It seems to work fine when VMs on a host are placed on the CSV that are currently owned by the same hosts

[mpark]

Can you confirm that you have provided the correct credentials for SCVMM server under Backup Infrastructure?
Also, I'd check and test each hosts for credential verification and see if you encounter any errors. I recall having read-only error due to credential issue on the Veeam

[skumflum]

Credentials are ok. I can browse SCVMM from Veeam, rescan and so on

[skumflum]

I can now confirm the pattern.

The backup works only on VM's that are located on a host who is also the CSV owner of the VM's current storage location.

[Mike Resseler]

Hi,

Could you please create a support call (and post the support case ID and follow up here)? More details are needed to investigate this and our engineers will be able to see what is happening in the logs.

It might be something really simple but it could be obviously something bad (let's hope not )

Keep us informed
Mike

[skumflum]

Hi,
I have a case id: 02087617

There is something wrong with VSS. The issue manifest itself when doing shadow copy of a CSV that isn’t owned by the cluster node (host).
I can replicate the error by doing shadow copy manually:

diskshadow
set verbose on
set context persistent nowriters
add volume C:\ClusterStorage\VolumeName
create
expose %vss_shadow_1% x:

This will fail unless it’s done against a CSV that is owned by the node.

Any thoughts?

[Mike Resseler]

Hi.

Sorry missed that from your first post you already had a case ID. What OS is your Hyper-V host? 2012 or R2? From what I remember this used to be an issue but it was fixed with some hotfixes.

Did you check our KB on the needed hotfixes (note: not all of them come with windows update): https://www.veeam.com/kb1838

Mike

[skumflum]

Thank you

I have read release notes on all the recommended hotfixes but I do not think that any of them apply to my problem. The closest one about CSV is...
"Cluster service stops during the VSS backup in a Windows Server 2012 R2 or Windows Server 2012-based Hyper-V cluster" But that's not what going on.

I will try to reboot all hosts one-by-one and see how this goes.

[skumflum]

ok.. we tried to reboot a host - problem not resolved. Then we rebooted ALL the host and now the problem is gone!

Great, but we don't know what happened

[Mike Resseler]

And you are sure that the backup now works on VM's running on a host are not the CSV owner?

Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?

(To be honest, I am confused also )

[skumflum]

And you are sure that the backup now works on VM's running on a host are not the CSV owner?

Yes I have verified this 100%

Can you also check your hosts if something is installed the between the reboot and coming back up? Maybe a patch / hotfix that still needed to "apply"?

No updates installed.... well since august 2016 ... hmm maybe I should suggest to patch more often

(To be honest, I am confused also )

[Mike Resseler]

Well,

I know very well it is a pain, but yes, patch, patch, patch but that is not enough. You need to install hotfixes also (as said by that KB I mentioned) because they don't come with Windows Update. And I know that normally the best practice is to install only hotfixes that are "applicable" to your problem but in the case of Hyper-V and CSV's I tend to install them as a part of my baseline setup.

Please keep us informed if the issues would arise again.

[skumflum]

Thank you Mike

Okay, so you would recommend installing all the hotfixes recommended even them that is not obvious applicable? This goes against my "childhood" teaching

The KB you mentioned has not been updated since 2015-12-15.
Microsoft also maintain some lists (Hyper-V and Cluster)

https://support.microsoft.com/en-us/kb/3135020
https://support.microsoft.com/en-us/kb/2920151

Would you recommend installing all the listed patches/hotfixes? Some of them are rather old.

[Mike Resseler]

I realize the KB is rather old but as far as I know most of those are still applicable. However, if your support case is still open, I would ask the support engineers about this. They are much more familiar with it then I am and they work daily on it so their advice is most certainly better than mine on this one

[TheKaiser]

We have an issue with backup after Update 4 for B&R 9.5

Of the 57 VMs always the same 10 VMs fail

Environment: Windows Server 2016 with Veeam Backup & Replication 9.5.4.2615
Windows Server 2012 R2 Hyper-V Host

Code: Select all

The backup fails with: Failed to open file 
[\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy: 

Code: Select all 30/1/2019 3:33:42 PM :: Processing DEV04 Error: The system cannot find the path specified. 
Failed to open file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy12\Virtual Machines\Snapshots\6C8E1C7E-D871-4C02-AF36-143CA3F5C6E7.xml] in readonly mode. 
Failed to open file [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy12\Virtual Machines\Snapshots\6C8E1C7E-D871-4C02-AF36-143CA3F5C6E7.xml] for reading. 

what we have done so far:

-Create new job with affected VM
-Restart the VMs
-Restart the Hyper-V 2012 Host
-Migrated the affected VM to another host

-Check the with vssadmin list shadows |more

Code: Select all

PS C:\Windows\system32> vssadmin list shadows |more
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

No items found that satisfy the query.

Veeam Support - Case # 03388708 - Error: The system cannot find the file specified

Thanks

[Dima P.]

Hello and welcome Achim!

I've merged your post to the existing discussion. This problem is likely caused by issues with VSS components, so, kindly, keep working with our support team in order to find the root cause. Cheers!

[nikods]

I have the same problem after update to 9.5 update 4. I have three servers in windows server 2012 r2 failover cluster, all of the servers backup jobs are failing, the same day I have updated to update 4 one time all of the jobs have completed and tomorrow they started to fail with error:
Processing SERVERNAME Error: The system cannot find the file specified.
Failed to open file [\\?\GLOBALROOT\Device\CSV.....xml] in readonly mode.
Failed to open file [\\?\GLOBALROOT\Device\CSV.....xml] in for reading.
I try to put VM on same host that holds CSV but same error. I can rescan hosts and check backup files but cant do backup jobs. I try to make new job, same error. I have restarted, reconnected iscsi to csv storage and no change.
Please help

[TheKaiser]

Hello at all.

Our problem was solved together with Veeam Support!

Here the solution of the Veeam Support:

Here's what I want you to do:
- open VM settings (e.g. busoDEV01) in Hyper-V Manager
- select "Checkpoint File Location" -> "Browse...". -> select the path "F:\Virtual Machines\Snapshots".
(currently the wrong path "F:\virtual machines" is selected there)

So we adjusted the path "Checkpoint File Location".

Then restart the Veeam Backup job.

And the problems are solved

Thanks

[Mike Resseler]

Hi TheKaiser,

Thanks for letting us know. Did you change that path manually or was it this by default?

[nikods]

The manual setting of the paths for Snapshots has resolved the problem form me also!

Thank you All!

Niko

[JSi]

Hi to all,

we have exactly the same problem (Win2012R2 Hyper-V cluster) after updating to U4, unfortunately our support contract expired.
Change of checkpoint path in Failover Cluster Manager unfortunately doesn`t help. Is there any other required condition (backup server reboot, hyper-v node reboot,...)?

Jan

[JSi]

Probably it takes some time. It looks ok now.

[Mike Resseler]

Hi Jan,

Just to be clear: You did the change of the checkpoint path and it didn't work to start with? But now it works? I do believe you need to restart the VM to make the new checkpoint settings applicable. (But not so sure anymore)

[JSi]

Hi Mike,

many of impacted VMs were in offline state. It doesn`t help at first retry, but without any other change, next retry was ok. Thank you for reply.

Anyway Veeam thanx for your work on new update and your general attitude.

J.

[ejenner]

I've got a couple of Hyper-V VMs which won't backup.

Code: Select all

The following error: 04/02/2019 04:05:40 :: Processing SERVERNAME Error: The system cannot find the file specified.
Failed to open file [\\?\Volume{4561c16c-2825-11e9-80c5-2c768a5d7fc0}\SERVERNAME\Snapshots\424D66E2-309E-4ECC-8589-1C691E1E6B5C.xml] in readonly mode.
Failed to open file [\\?\Volume{4561c16c-2825-11e9-80c5-2c768a5d7fc0}\SERVERNAME\Snapshots\424D66E2-309E-4ECC-8589-1C691E1E6B5C.xml] for reading.

The thing both these servers have in common is that they're DMZ servers. The VMs are running from a HP 3PAR and I have Direct SAN backup configured.

I'm guessing this is a blocked ports issue due to the DMZ configuration but just putting it out there for comments / suggestions. Just in case someone has seen it before.

[PTide]

EJ,

Although your config might be a little different (no CSV apparently), this thread seems to be approppriate for your post. Kindly open your own support case and let us know case ID.

Thanks!

[ejenner]

yes, I did see this. Going to have to think about this for a while as although the error message is the same... the symptoms are different.

[Mike Resseler]

Hey EJ,

Most likely it isn't this problem, but I heard yesterday from a friend that his checkpoints got into read-only mode after an AV signature update. Are those checkpoint locations excluded?

[roger_ramjet]

Hello at all.

Our problem was solved together with Veeam Support!

Here the solution of the Veeam Support:

Here's what I want you to do:
- open VM settings (e.g. busoDEV01) in Hyper-V Manager
- select "Checkpoint File Location" -> "Browse...". -> select the path "F:\Virtual Machines\Snapshots".
(currently the wrong path "F:\virtual machines" is selected there)

So we adjusted the path "Checkpoint File Location".

Then restart the Veeam Backup job.

I cannot believe this worked for me!! Mine was slightly different where there was no "Snapshots" folder in the save location. My issue was there were no capital letters where there should have been. Looks like you have to be very specific in regards to capital letters in the whole file name location with Veeam.

EDIT: Also if you're using Hyper-V with Failover Clustering, you have to open "Settings" from the VM in Failover Cluster Manager to change the snapshot path.