Veeam files to tape job can't find files from share with first try

[sasilik]

We have a files to tape job which files part is cleared and populated anew every monday. Full backup is scheduled to start 16:30 on Monday. For some reason job does not see files in some folders. The structure of folder is

<DD share>\DAY\devel\archlogs
<DD share>\DAY\devel\control
<DD share>\DAY\devel\db_backup

Its same for three environment, devel, test, prod. Veeam enumerates folders and finds files in first two folders but not in db_backup and same with all three environments. In log file there are rows for a scheduled job

[06.06.2022 16:30:24] <01> Info [CTapeFileSystemEnumerator2] Enumerating directory <DD share>\DAY\devel\db_backup recursively...
[06.06.2022 16:30:24] <01> Info [CTapeFileSystemEnumerator2] Starting recursive enumeration for: <DD share>\DAY\devel\db_backup
[06.06.2022 16:30:24] <12> Info Subscribe for cancel the operation: e92cb397-267b-4c4e-bc84-e6338ebdc477
[06.06.2022 16:30:29] <12> Info The operation completed: e92cb397-267b-4c4e-bc84-e6338ebdc477
[06.06.2022 16:30:29] <01> Info [CTapeFileSystemEnumerator2] Enumeration completed: 5,268sec, files: 0

If I stop the job and start again then it does incremental and job finds files

[06.06.2022 17:34:37] <01> Info [TapeFileBackupJobPerformer] Using incremental pass last write time filter
...
[06.06.2022 17:34:41] <01> Info [CTapeFileSystemEnumerator2] Enumerating directory <DD share>\DAY\devel\db_backup recursively...
[06.06.2022 17:34:41] <01> Info [CTapeFileSystemEnumerator2] Starting recursive enumeration for: <DD share>\DAY\devel\db_backup
[06.06.2022 17:34:41] <13> Info Subscribe for cancel the operation: a7102565-ac40-41f6-98ec-0187629f8467
[06.06.2022 17:34:42] <13> Info The operation completed: a7102565-ac40-41f6-98ec-0187629f8467
[06.06.2022 17:34:42] <01> Info [CTapeFileSystemEnumerator2] Enumeration completed: 1,141sec, files: 855

If I stop the job and do Active full then it also finds files

[06.06.2022 17:49:00] <01> Info [CTapeFileSystemEnumerator2] Enumerating directory <DD share>\DAY\devel\db_backup recursively...
[06.06.2022 17:49:00] <01> Info [CTapeFileSystemEnumerator2] Starting recursive enumeration for: <DD share>\DAY\devel\db_backup
[06.06.2022 17:49:00] <14> Info Subscribe for cancel the operation: 14742ab7-d6c9-49ba-a6bf-13c0aa11afa5
[06.06.2022 17:49:00] <14> Info The operation completed: 14742ab7-d6c9-49ba-a6bf-13c0aa11afa5
[06.06.2022 17:49:00] <01> Info [CTapeFileSystemEnumerator2] Enumeration completed: 0,122sec, files: 855

I am out of ideas what could be problem here that Veeam does not see any files in these folders. And just in these db_backup folders. It enumerates and finds files in all other folders. We only found it when we did restore from tapes as our yearly plan requires because there are no errors, Veeam just does not see files for some reason.

[HannesK]

Hello,
I guess a permission issue... what did Veeam support say (case number)?

Best regards,
Hannes

[sasilik]

I haven't contacted support yet, my vacation just started and I thought I ask here first. I doubt that it is a permission issue because it does not see only with first try and permissions do not change during the 5 minutes when I stop the job and start again. I also tried manually, cleared the files list and added these paths manually and Veeam found files. Also Veeam leaves info in logs when it can't access and enumerate files.

[HannesK]

Hello,

I thought I ask here first

Well, it's written in red actually... Please provide a support case ID for this issue, as requested when you click New Topic
Without case number, the topic will eventually be
deleted by moderators.

Best regards,
Hannes

[sasilik]

I opened the case, Case #05473465, but as tape jobs work until friday and sometimes even longer then my ability to do something is limited and it could take quite long time.

[sasilik]

I asked the case to be closed because after some discussion in our department we decided to switch from NFS to CIFS and it seems there is no problem with CIFS.

I also did a little more digging and at least for me it seems there may be something that can be improved on Veeam side. Gist of the problem was that tape server did not get file list from NFS share in timely manner. It mounted share and sent NFS command to get list of files in folder but then it got "The semaphore timeout period has expired." error after about 5 seconds and Veeam decided that there is no files in the folder and went on as usual. No errors or warnings in job itself. In the background the NFS server sent the files list after some more seconds but that did not get processed anymore because Veeam got its 0 files in folder already. I guess that kind of situation could be handled better.