File Level Restore errors with Zone.Identifier Alternate Data Stream

[bbuchanan]

Case #05992095 - I'm doing a full server restore as part of a migration and Veeam is having a tough time dealing with the Zone.Identifier alternate data stream.

Code: Select all

Warning Abort item: 'D:\verylongfilepath.txt'. Reason: 'WriteFilePart error: [Failed to obtain a handler for the file: D:\verylongfilepath.txt:Zone.Identifier.
Warning Failed to create new binary file [D:\verylongfilepath.txt:Zone.Identifier].
Warning Win32 error:The system cannot find the path specified.
Warning Code: 3]'

https://learn.microsoft.com/en-us/opens ... 16a9df36e8

Second question, are there restrictions on the supported characters for File-Level Restore? Maybe when creating a directory? (words here reduced to just the first letter because it's long)
Reason: 'CreateDir error: [cannot create directory: [D:\Share\d\H\C\C.A\S\N\H\I\R\M\o25TH%ILE reports(ne,removed)].

The directory certainly exists on the mount server in C:\VeeamFLR, and I was able to create it manually in the destination.

Now I still have Crowdstrike installed on the mount server as per company policy, but I removed it from the destination server for testing. I guess that's my next test?

[HannesK]

Hello,
as far as I understand the issue, it's a Microsoft limitation with path length.

I'm doing a full server restore

how do you do that? with the Veeam File level explorer? Windows Explorer? My alternative idea would be to mount the backup to your new server with instant disk recovery or data integration api. Then copy with windows native tools and see how that goes.

Security software sometimes does not like the low level operations we do. So all Veeam paths must be excluded from checking https://www.veeam.com/kb1999

Best regards,
Hannes

[bbuchanan]

Hi,

Do you think Veeam will support long file paths anytime soon? MAX_PATH limitations were removed with Windows 10 1607 (and Server 2016), if opting in (which I have via group policy)

Starting in Windows 10, version 1607, MAX_PATH limitations have been removed from common Win32 file and directory functions. However, you must opt-in to the new behavior.

https://learn.microsoft.com/en-us/windo ... limitation

Thanks.

[HannesK]

Hello,
Veeam supports long file names / paths in general (we are bound to the Microsoft limits).

If you enabled it everywhere, then support needs to continue investigating.

As a quick fix, my suggestions should work.

Best regards,
Hannes

[bbuchanan]

It looks like long paths are a problem as I can easily copy the file from the mount server to the destination. (e.g. copy C:\VeeamFLR\xxx \\server\d$\path)

I was also able to turn off long path support and the copy command fails, and then succeeds again when long path support is enabled. although I didn't try a "pull", running the copy command on the destination.

[bbuchanan]

I've fallen back to robocopy as it is shows differences between the source and the restored files. Using /ZB /MIR /COPYALL /DCOPY:DAT I expected robocopy to report all files as "same" but it actually reported "modified". Subsequent invocations of robocopy from both the FLR and the production source then report "same". It also copied the ~4,000 long path files that Veeam couldn't restore successfully. Support is baffled.

[bbuchanan]

I've been able to reproduce the problem in an isolate lab and updated Support. Veeam is unable to restore long paths, but manually copying the file works.

My testing methodology:

Start by creating an unsupported long path on a Windows Server 2012 R2 VM by using the subst command.

Code: Select all

set numbers=12345678901234567890123

set dir1=Dir_%numbers:~0,20%%numbers:~0,20%%numbers:~0,20%
set dir2=Dir_%numbers:~1,20%%numbers:~1,20%%numbers:~1,20%
set dir3=Dir_%numbers:~2,20%%numbers:~2,20%%numbers:~2,20%
set dir4=Dir_%numbers:~3,20%%numbers:~3,20%%numbers:~3,20%

mkdir C:\LongPath\%dir1%\%dir2%

subst F: C:\LongPath\%dir1%\%dir2%
mkdir F:\%dir3%\%dir4%

echo Test File > F:\%dir3%\%dir4%\TestFile.txt

subst F: /d

Backup that VM and attempt to restore it to another server (using a mount server and destination server that both support long paths)

Code: Select all

4/26/2023 10:36:00 AM          Failed to restore "C:\Restore\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123\TestFile.txt": WriteFilePart error: [Failed to obtain a handler for the file: C:\Restore\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123\TestFile.txt. Failed to create new binary file [C:\Restore\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123\TestFile.txt]. Win32 error:The system cannot find the path specified.  Code: 3]

While the FLR is still mounted, successfully copy the file from the mount server to the destination server, demonstrating that long paths are enabled on the mount server and destination server.

Code: Select all

net use X: \\server2019\restore
C:\>mkdir X:\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123

C:\>copy C:\VeeamFLR\Server2012R2_afa44c3f\Volume1\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123\TestFile.txt X:\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123
        1 file(s) copied.

C:\>dir X: /a-d /s
 Volume in drive X has no label.
 Volume Serial Number is 000A-7A07

 Directory of X:\LongPath\Dir_123456789012345678901234567890123456789012345678901234567890\Dir_234567890123456789012345678901234567890123456789012345678901\Dir_345678901234567890123456789012345678901234567890123456789012\Dir_456789012345678901234567890123456789012345678901234567890123

04/26/2023  10:28 AM                12 TestFile.txt
               1 File(s)             12 bytes

     Total Files Listed:
               1 File(s)             12 bytes
               0 Dir(s)  117,080,051,712 bytes free

I've never dug this deep into file-level restore errors before V12, but something isn't right.

[bbuchanan]

Update: Support has been able to reproduce the long path problem using the "Restore To" function and has escalated to RnD. The "Copy To" function, specifying \\server2019\c$\, etc. does work as it should.

[HannesK]

thanks for the update and the patience. As the problem is identified, it should be possible to fix it. "restore to" works different than "copy to". "restore to" is using the Veeam native data movers while "copy to" is plain "windows copy".